diff --git a/docs/setup/emr.md b/docs/setup/emr.md index e39ac0ab8a5..40ea4196d3e 100644 --- a/docs/setup/emr.md +++ b/docs/setup/emr.md @@ -78,3 +78,54 @@ spark.sql("SELECT ST_Point(0, 0)").show() ``` Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that. + +## Use Sedona in R + +The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand. + +!!!note + The `apache.sedona` 1.9.1 release on CRAN supports Spark 3.x only. Every EMR 7.x release ships Spark 3.5, so any of them works. + +### Extend the initialization script + +Add the following to the bootstrap script above. Like the rest of the script, it runs on every node in the cluster. Spark SQL queries from R only need the R packages on the node you run R from; worker nodes need them only if you run R code on executors with `spark_apply()`. + +```bash +# Install R and the Sedona R interface. libcurl-devel is needed to build the R curl package that sparklyr depends on. +sudo yum install -y R libcurl-devel +sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org", Ncpus = parallel::detectCores()); stopifnot(all(c("sparklyr", "apache.sedona") %in% rownames(installed.packages())))' +``` + +`install.packages()` only warns when a package fails to build, so the `stopifnot()` call is what makes the `R` command fail instead of reporting success. Expect the R installation to add several minutes to the bootstrap of each node. + +### Connect to the cluster from R + +EMR installs Spark under `/usr/lib/spark`. When you connect from R, the Spark driver runs in YARN client mode on the node where R runs. The `spark.yarn.dist.jars` setting above only ships jars to the executors, so it does not put Sedona on the driver's classpath. `apache.sedona` adds the jars to the driver itself, either by downloading them from Maven Central or from the local files listed in `SEDONA_JAR_FILES`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars`: + +```r +library(sparklyr) +library(apache.sedona) + +Sys.setenv( + "SEDONA_JAR_FILES" = paste( + "/jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar", + "/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", + sep = ":" + ) +) + +sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark") +``` + +The Jupyter example above does not need this step because EMR runs Livy in cluster deploy mode, where the driver runs on a YARN container that receives the jars. + +!!!note + `SEDONA_JAR_FILES` holds a `:`-separated list of jars, and setting it only replaces the Sedona Maven coordinate. `apache.sedona` still requests `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}` through `--packages`, so the driver needs access to Maven Central, or that jar already in its Ivy cache, either way. Listing the GeoTools wrapper jar in `SEDONA_JAR_FILES` is harmless. If you leave `SEDONA_JAR_FILES` unset, `apache.sedona` also requests `org.apache.sedona:sedona-spark-shaded-_:{{ sedona.current_version }}`. Ivy caches the downloaded jars per user, so only the first connection downloads them, but on a slow network that first connection can exceed the default `sparklyr.connect.timeout`. + +### Verify the R installation + +```r +sdf_sql(sc, "SELECT ST_Point(0.0, 0.0) AS geom") %>% collect() +``` + +For more on what the R interface offers, see the [Sedona R documentation](https://sedona.apache.org/latest/api/rdocs/). diff --git a/docs/setup/emr.zh.md b/docs/setup/emr.zh.md index a9adea4d41c..50b3009f30d 100644 --- a/docs/setup/emr.zh.md +++ b/docs/setup/emr.zh.md @@ -78,3 +78,54 @@ spark.sql("SELECT ST_Point(0, 0)").show() ``` 注意:您不需要再调用 `SedonaRegistrator.registerAll(spark)` 或 `SedonaContext.create(spark)`,因为配置中的 `org.apache.sedona.sql.SedonaSqlExtensions` 已经为您完成了这些工作。 + +## 在 R 中使用 Sedona + +[`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R 包是一个 [`sparklyr`](https://spark.rstudio.com) 扩展。只要在 `spark_connect()` 之前加载它,Sedona 的序列化器、UDT 与 UDF 就会自动注册,因此 R 中不需要手动调用与 `SedonaContext.create()` 等价的方法。 + +!!!note + CRAN 上的 `apache.sedona` 1.9.1 版本仅支持 Spark 3.x。所有 EMR 7.x 版本都搭载 Spark 3.5,因此均可使用。 + +### 扩展初始化脚本 + +在上面的引导脚本中追加以下内容。与脚本的其余部分一样,它会在集群的每个节点上运行。从 R 执行 Spark SQL 查询时,只有运行 R 的节点需要这些 R 包;只有在使用 `spark_apply()` 于 executor 上运行 R 代码时,工作节点才需要它们。 + +```bash +# 安装 R 以及 Sedona 的 R 接口。sparklyr 依赖的 R curl 包需要 libcurl-devel 才能编译。 +sudo yum install -y R libcurl-devel +sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org", Ncpus = parallel::detectCores()); stopifnot(all(c("sparklyr", "apache.sedona") %in% rownames(installed.packages())))' +``` + +包编译失败时 `install.packages()` 只会给出警告,因此需要通过 `stopifnot()` 让 `R` 命令以失败退出,而不是报告成功。安装 R 预计会使每个节点的引导时间增加几分钟。 + +### 从 R 连接集群 + +EMR 将 Spark 安装在 `/usr/lib/spark` 下。从 R 连接时,Spark driver 以 YARN client 模式运行在运行 R 的节点上。上面的 `spark.yarn.dist.jars` 设置只会把 jar 分发给 executor,并不会把 Sedona 加入 driver 的 classpath。`apache.sedona` 会自行把 jar 加入 driver:要么从 Maven Central 下载,要么使用 `SEDONA_JAR_FILES` 中列出的本地文件。请将 `SEDONA_JAR_FILES` 指向引导脚本已经下载到 `/jars` 的 jar 包: + +```r +library(sparklyr) +library(apache.sedona) + +Sys.setenv( + "SEDONA_JAR_FILES" = paste( + "/jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar", + "/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", + sep = ":" + ) +) + +sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark") +``` + +上面的 Jupyter 示例不需要这一步,因为 EMR 以 cluster 部署模式运行 Livy,driver 运行在能收到这些 jar 的 YARN 容器中。 + +!!!note + `SEDONA_JAR_FILES` 是一个以 `:` 分隔的 jar 列表,设置它只会替换 Sedona 的 Maven 坐标。`apache.sedona` 仍会通过 `--packages` 请求 `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}`,因此无论如何 driver 都需要能访问 Maven Central,或者 Ivy 缓存中已有该 jar。在 `SEDONA_JAR_FILES` 中列出 GeoTools wrapper 的 jar 并无害处。如果不设置 `SEDONA_JAR_FILES`,`apache.sedona` 还会请求 `org.apache.sedona:sedona-spark-shaded-_:{{ sedona.current_version }}`。Ivy 会按用户缓存下载的 jar,因此只有首次连接需要下载,但在网络较慢时,首次连接可能超过 `sparklyr.connect.timeout` 的默认值。 + +### 验证 R 端安装 + +```r +sdf_sql(sc, "SELECT ST_Point(0.0, 0.0) AS geom") %>% collect() +``` + +关于 R 接口的更多功能,请参阅 [Sedona R 文档](https://sedona.apache.org/latest/api/rdocs/)。