From 7541db7779f27b7528419d3f4e54b085510a5a04 Mon Sep 17 00:00:00 2001 From: Minh Pham Date: Mon, 7 Sep 2026 19:04:59 -0400 Subject: [PATCH 1/2] [GH-1024] Add Sedona R instructions to the AWS EMR setup page The mkdocs site had no R deployment guidance anywhere: both "Install Sedona R" nav entries point at the generated pkgdown site, which documents the R API but not how to get Sedona onto a cluster. Adds a "Use Sedona in R" section to the EMR page (and its Chinese mirror) covering the bootstrap-script additions, connecting with sparklyr in YARN client mode, and pointing SEDONA_JAR_FILES at the jars the page's existing bootstrap script already downloads to /jars. --- docs/setup/emr.md | 47 ++++++++++++++++++++++++++++++++++++++++++++ docs/setup/emr.zh.md | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 94 insertions(+) diff --git a/docs/setup/emr.md b/docs/setup/emr.md index e39ac0ab8a5..e525db88df5 100644 --- a/docs/setup/emr.md +++ b/docs/setup/emr.md @@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show() ``` Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that. + +## Use Sedona in R + +The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand. + +!!!note + The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version. + +### Extend the initialization script + +Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from: + +```bash +# Install R and the Sedona R interface +sudo yum install -y R +sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")' +``` + +### Connect to the cluster from R + +EMR installs Spark under `/usr/lib/spark`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars` so that `sparklyr` uses them instead of resolving the Sedona coordinates from Maven Central every time you connect: + +```r +library(sparklyr) +library(apache.sedona) + +Sys.setenv( + "SEDONA_JAR_FILES" = paste( + "/jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar", + "/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", + sep = ":" + ) +) + +sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark") +``` + +!!!note + `SEDONA_JAR_FILES` holds a `:`-separated list and replaces *both* Maven coordinates that `apache.sedona` would otherwise request, which is why the GeoTools wrapper jar has to be listed next to the Sedona jar. If you leave `SEDONA_JAR_FILES` unset, every connection downloads `org.apache.sedona:sedona-spark-shaded-_:{{ sedona.current_version }}` and `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}`. That requires outbound internet access from the driver and can take long enough to exceed the default `sparklyr.connect.timeout`. + +### Verify the R installation + +```r +sdf_sql(sc, "SELECT ST_Point(0.0, 0.0) AS geom") %>% collect() +``` + +For more on what the R interface offers, see the [Sedona R documentation](https://sedona.apache.org/latest/api/rdocs/). diff --git a/docs/setup/emr.zh.md b/docs/setup/emr.zh.md index a9adea4d41c..12ae405d5db 100644 --- a/docs/setup/emr.zh.md +++ b/docs/setup/emr.zh.md @@ -78,3 +78,50 @@ spark.sql("SELECT ST_Point(0, 0)").show() ``` 注意：您不需要再调用 `SedonaRegistrator.registerAll(spark)` 或 `SedonaContext.create(spark)`，因为配置中的 `org.apache.sedona.sql.SedonaSqlExtensions` 已经为您完成了这些工作。 + +## 在 R 中使用 Sedona + +[`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R 包是一个 [`sparklyr`](https://spark.rstudio.com) 扩展。只要在 `spark_connect()` 之前加载它，Sedona 的序列化器、UDT 与 UDF 就会自动注册，因此 R 中不需要手动调用与 `SedonaContext.create()` 等价的方法。 + +!!!note + R 接口仅支持 Spark 3.x。请确认您选择的 EMR 版本自带的是 Spark 3 系列版本。 + +### 扩展初始化脚本 + +在上面的引导脚本中追加以下内容，以便在运行 R 的节点上安装 R 及这两个 R 包： + +```bash +# 安装 R 以及 Sedona 的 R 接口 +sudo yum install -y R +sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")' +``` + +### 从 R 连接集群 + +EMR 将 Spark 安装在 `/usr/lib/spark` 下。请将 `SEDONA_JAR_FILES` 指向引导脚本已经下载到 `/jars` 的 jar 包，这样 `sparklyr` 就会直接使用它们，而不必在每次连接时从 Maven Central 解析 Sedona 坐标： + +```r +library(sparklyr) +library(apache.sedona) + +Sys.setenv( + "SEDONA_JAR_FILES" = paste( + "/jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar", + "/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", + sep = ":" + ) +) + +sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark") +``` + +!!!note + `SEDONA_JAR_FILES` 是一个以 `:` 分隔的列表，并且会同时替换 `apache.sedona` 原本请求的*两个* Maven 坐标，因此必须把 GeoTools wrapper 的 jar 与 Sedona 的 jar 一起列出。如果不设置 `SEDONA_JAR_FILES`，每次连接都会下载 `org.apache.sedona:sedona-spark-shaded-_:{{ sedona.current_version }}` 与 `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}`。这要求 driver 能够访问外网，并且耗时可能超过 `sparklyr.connect.timeout` 的默认值。 + +### 验证 R 端安装 + +```r +sdf_sql(sc, "SELECT ST_Point(0.0, 0.0) AS geom") %>% collect() +``` + +关于 R 接口的更多功能，请参阅 [Sedona R 文档](https://sedona.apache.org/latest/api/rdocs/)。 From b7f1d05b24c6d48dc5fabfd15e79662fd780dc7e Mon Sep 17 00:00:00 2001 From: Minh Pham Date: Fri, 11 Sep 2026 10:17:19 -0400 Subject: [PATCH 2/2] Update EMR setup documentation for Sedona R interface and installation instructions --- docs/setup/emr.md | 20 ++++++++++++-------- docs/setup/emr.zh.md | 20 ++++++++++++-------- 2 files changed, 24 insertions(+), 16 deletions(-) diff --git a/docs/setup/emr.md b/docs/setup/emr.md index e525db88df5..40ea4196d3e 100644 --- a/docs/setup/emr.md +++ b/docs/setup/emr.md @@ -84,21 +84,23 @@ Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or ` The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand. !!!note - The R interface supports Spark 3.x only. Make sure the EMR release you pick ships a Spark 3 version. + The `apache.sedona` 1.9.1 release on CRAN supports Spark 3.x only. Every EMR 7.x release ships Spark 3.5, so any of them works. ### Extend the initialization script -Add the following to the bootstrap script above, so that R and the two R packages are available on the node you run R from: +Add the following to the bootstrap script above. Like the rest of the script, it runs on every node in the cluster. Spark SQL queries from R only need the R packages on the node you run R from; worker nodes need them only if you run R code on executors with `spark_apply()`. ```bash -# Install R and the Sedona R interface -sudo yum install -y R -sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")' +# Install R and the Sedona R interface. libcurl-devel is needed to build the R curl package that sparklyr depends on. +sudo yum install -y R libcurl-devel +sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org", Ncpus = parallel::detectCores()); stopifnot(all(c("sparklyr", "apache.sedona") %in% rownames(installed.packages())))' ``` +`install.packages()` only warns when a package fails to build, so the `stopifnot()` call is what makes the `R` command fail instead of reporting success. Expect the R installation to add several minutes to the bootstrap of each node. + ### Connect to the cluster from R -EMR installs Spark under `/usr/lib/spark`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars` so that `sparklyr` uses them instead of resolving the Sedona coordinates from Maven Central every time you connect: +EMR installs Spark under `/usr/lib/spark`. When you connect from R, the Spark driver runs in YARN client mode on the node where R runs. The `spark.yarn.dist.jars` setting above only ships jars to the executors, so it does not put Sedona on the driver's classpath. `apache.sedona` adds the jars to the driver itself, either by downloading them from Maven Central or from the local files listed in `SEDONA_JAR_FILES`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars`: ```r library(sparklyr) @@ -106,7 +108,7 @@ library(apache.sedona) Sys.setenv( "SEDONA_JAR_FILES" = paste( - "/jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar", + "/jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar", "/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", sep = ":" ) @@ -115,8 +117,10 @@ Sys.setenv( sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark") ``` +The Jupyter example above does not need this step because EMR runs Livy in cluster deploy mode, where the driver runs on a YARN container that receives the jars. + !!!note - `SEDONA_JAR_FILES` holds a `:`-separated list and replaces *both* Maven coordinates that `apache.sedona` would otherwise request, which is why the GeoTools wrapper jar has to be listed next to the Sedona jar. If you leave `SEDONA_JAR_FILES` unset, every connection downloads `org.apache.sedona:sedona-spark-shaded-_:{{ sedona.current_version }}` and `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}`. That requires outbound internet access from the driver and can take long enough to exceed the default `sparklyr.connect.timeout`. + `SEDONA_JAR_FILES` holds a `:`-separated list of jars, and setting it only replaces the Sedona Maven coordinate. `apache.sedona` still requests `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}` through `--packages`, so the driver needs access to Maven Central, or that jar already in its Ivy cache, either way. Listing the GeoTools wrapper jar in `SEDONA_JAR_FILES` is harmless. If you leave `SEDONA_JAR_FILES` unset, `apache.sedona` also requests `org.apache.sedona:sedona-spark-shaded-_:{{ sedona.current_version }}`. Ivy caches the downloaded jars per user, so only the first connection downloads them, but on a slow network that first connection can exceed the default `sparklyr.connect.timeout`. ### Verify the R installation diff --git a/docs/setup/emr.zh.md b/docs/setup/emr.zh.md index 12ae405d5db..50b3009f30d 100644 --- a/docs/setup/emr.zh.md +++ b/docs/setup/emr.zh.md @@ -84,21 +84,23 @@ spark.sql("SELECT ST_Point(0, 0)").show() [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R 包是一个 [`sparklyr`](https://spark.rstudio.com) 扩展。只要在 `spark_connect()` 之前加载它，Sedona 的序列化器、UDT 与 UDF 就会自动注册，因此 R 中不需要手动调用与 `SedonaContext.create()` 等价的方法。 !!!note - R 接口仅支持 Spark 3.x。请确认您选择的 EMR 版本自带的是 Spark 3 系列版本。 + CRAN 上的 `apache.sedona` 1.9.1 版本仅支持 Spark 3.x。所有 EMR 7.x 版本都搭载 Spark 3.5，因此均可使用。 ### 扩展初始化脚本 -在上面的引导脚本中追加以下内容，以便在运行 R 的节点上安装 R 及这两个 R 包： +在上面的引导脚本中追加以下内容。与脚本的其余部分一样，它会在集群的每个节点上运行。从 R 执行 Spark SQL 查询时，只有运行 R 的节点需要这些 R 包；只有在使用 `spark_apply()` 于 executor 上运行 R 代码时，工作节点才需要它们。 ```bash -# 安装 R 以及 Sedona 的 R 接口 -sudo yum install -y R -sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org")' +# 安装 R 以及 Sedona 的 R 接口。sparklyr 依赖的 R curl 包需要 libcurl-devel 才能编译。 +sudo yum install -y R libcurl-devel +sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org", Ncpus = parallel::detectCores()); stopifnot(all(c("sparklyr", "apache.sedona") %in% rownames(installed.packages())))' ``` +包编译失败时 `install.packages()` 只会给出警告，因此需要通过 `stopifnot()` 让 `R` 命令以失败退出，而不是报告成功。安装 R 预计会使每个节点的引导时间增加几分钟。 + ### 从 R 连接集群 -EMR 将 Spark 安装在 `/usr/lib/spark` 下。请将 `SEDONA_JAR_FILES` 指向引导脚本已经下载到 `/jars` 的 jar 包，这样 `sparklyr` 就会直接使用它们，而不必在每次连接时从 Maven Central 解析 Sedona 坐标： +EMR 将 Spark 安装在 `/usr/lib/spark` 下。从 R 连接时，Spark driver 以 YARN client 模式运行在运行 R 的节点上。上面的 `spark.yarn.dist.jars` 设置只会把 jar 分发给 executor，并不会把 Sedona 加入 driver 的 classpath。`apache.sedona` 会自行把 jar 加入 driver：要么从 Maven Central 下载，要么使用 `SEDONA_JAR_FILES` 中列出的本地文件。请将 `SEDONA_JAR_FILES` 指向引导脚本已经下载到 `/jars` 的 jar 包： ```r library(sparklyr) @@ -106,7 +108,7 @@ library(apache.sedona) Sys.setenv( "SEDONA_JAR_FILES" = paste( - "/jars/sedona-spark-shaded-3.3_2.12-{{ sedona.current_version }}.jar", + "/jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar", "/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", sep = ":" ) @@ -115,8 +117,10 @@ Sys.setenv( sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark") ``` +上面的 Jupyter 示例不需要这一步，因为 EMR 以 cluster 部署模式运行 Livy，driver 运行在能收到这些 jar 的 YARN 容器中。 + !!!note - `SEDONA_JAR_FILES` 是一个以 `:` 分隔的列表，并且会同时替换 `apache.sedona` 原本请求的*两个* Maven 坐标，因此必须把 GeoTools wrapper 的 jar 与 Sedona 的 jar 一起列出。如果不设置 `SEDONA_JAR_FILES`，每次连接都会下载 `org.apache.sedona:sedona-spark-shaded-_:{{ sedona.current_version }}` 与 `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}`。这要求 driver 能够访问外网，并且耗时可能超过 `sparklyr.connect.timeout` 的默认值。 + `SEDONA_JAR_FILES` 是一个以 `:` 分隔的 jar 列表，设置它只会替换 Sedona 的 Maven 坐标。`apache.sedona` 仍会通过 `--packages` 请求 `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}`，因此无论如何 driver 都需要能访问 Maven Central，或者 Ivy 缓存中已有该 jar。在 `SEDONA_JAR_FILES` 中列出 GeoTools wrapper 的 jar 并无害处。如果不设置 `SEDONA_JAR_FILES`，`apache.sedona` 还会请求 `org.apache.sedona:sedona-spark-shaded-_:{{ sedona.current_version }}`。Ivy 会按用户缓存下载的 jar，因此只有首次连接需要下载，但在网络较慢时，首次连接可能超过 `sparklyr.connect.timeout` 的默认值。 ### 验证 R 端安装