Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions docs/setup/emr.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,54 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```

Note that: you don't need to run the `SedonaRegistrator.registerAll(spark)` or `SedonaContext.create(spark)` because `org.apache.sedona.sql.SedonaSqlExtensions` in the config will take care of that.

## Use Sedona in R

The [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R package is a [`sparklyr`](https://spark.rstudio.com) extension. Attaching it before `spark_connect()` is enough to register Sedona's serializers, UDTs and UDFs, so there is no R equivalent of `SedonaContext.create()` to call by hand.

@jiayuasu jiayuasu Sep 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, no change needed. One side effect worth knowing: attaching the package sets spark.kryo.registrator to SedonaVizKryoRegistrator through an R option, which sparklyr passes as --conf, overriding the SedonaKryoRegistrator from spark-defaults. Harmless, it's a superset.


!!!note
The `apache.sedona` 1.9.1 release on CRAN supports Spark 3.x only. Every EMR 7.x release ships Spark 3.5, so any of them works.

### Extend the initialization script

Add the following to the bootstrap script above. Like the rest of the script, it runs on every node in the cluster. Spark SQL queries from R only need the R packages on the node you run R from; worker nodes need them only if you run R code on executors with `spark_apply()`.

```bash
# Install R and the Sedona R interface. libcurl-devel is needed to build the R curl package that sparklyr depends on.
sudo yum install -y R libcurl-devel
sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org", Ncpus = parallel::detectCores()); stopifnot(all(c("sparklyr", "apache.sedona") %in% rownames(installed.packages())))'
```

`install.packages()` only warns when a package fails to build, so the `stopifnot()` call is what makes the `R` command fail instead of reporting success. Expect the R installation to add several minutes to the bootstrap of each node.

### Connect to the cluster from R

EMR installs Spark under `/usr/lib/spark`. When you connect from R, the Spark driver runs in YARN client mode on the node where R runs. The `spark.yarn.dist.jars` setting above only ships jars to the executors, so it does not put Sedona on the driver's classpath. `apache.sedona` adds the jars to the driver itself, either by downloading them from Maven Central or from the local files listed in `SEDONA_JAR_FILES`. Point `SEDONA_JAR_FILES` at the jars the bootstrap script already downloaded into `/jars`:

```r
library(sparklyr)
library(apache.sedona)

Sys.setenv(
"SEDONA_JAR_FILES" = paste(
"/jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar",
"/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar",
sep = ":"
)
)

sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark")
```

The Jupyter example above does not need this step because EMR runs Livy in cluster deploy mode, where the driver runs on a YARN container that receives the jars.

!!!note
`SEDONA_JAR_FILES` holds a `:`-separated list of jars, and setting it only replaces the Sedona Maven coordinate. `apache.sedona` still requests `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}` through `--packages`, so the driver needs access to Maven Central, or that jar already in its Ivy cache, either way. Listing the GeoTools wrapper jar in `SEDONA_JAR_FILES` is harmless. If you leave `SEDONA_JAR_FILES` unset, `apache.sedona` also requests `org.apache.sedona:sedona-spark-shaded-<spark version>_<scala version>:{{ sedona.current_version }}`. Ivy caches the downloaded jars per user, so only the first connection downloads them, but on a slow network that first connection can exceed the default `sparklyr.connect.timeout`.

### Verify the R installation

```r
sdf_sql(sc, "SELECT ST_Point(0.0, 0.0) AS geom") %>% collect()
```

For more on what the R interface offers, see the [Sedona R documentation](https://sedona.apache.org/latest/api/rdocs/).
51 changes: 51 additions & 0 deletions docs/setup/emr.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,54 @@ spark.sql("SELECT ST_Point(0, 0)").show()
```

注意:您不需要再调用 `SedonaRegistrator.registerAll(spark)` 或 `SedonaContext.create(spark)`,因为配置中的 `org.apache.sedona.sql.SedonaSqlExtensions` 已经为您完成了这些工作。

## 在 R 中使用 Sedona

@jiayuasu jiayuasu Sep 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same edits needed here: libcurl-devel, 3.5_2.12, and the two reworded notes. The translation itself reads fine.


[`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R 包是一个 [`sparklyr`](https://spark.rstudio.com) 扩展。只要在 `spark_connect()` 之前加载它,Sedona 的序列化器、UDT 与 UDF 就会自动注册,因此 R 中不需要手动调用与 `SedonaContext.create()` 等价的方法。

!!!note
CRAN 上的 `apache.sedona` 1.9.1 版本仅支持 Spark 3.x。所有 EMR 7.x 版本都搭载 Spark 3.5,因此均可使用。

### 扩展初始化脚本

在上面的引导脚本中追加以下内容。与脚本的其余部分一样,它会在集群的每个节点上运行。从 R 执行 Spark SQL 查询时,只有运行 R 的节点需要这些 R 包;只有在使用 `spark_apply()` 于 executor 上运行 R 代码时,工作节点才需要它们。

```bash
# 安装 R 以及 Sedona 的 R 接口。sparklyr 依赖的 R curl 包需要 libcurl-devel 才能编译。
sudo yum install -y R libcurl-devel
sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org", Ncpus = parallel::detectCores()); stopifnot(all(c("sparklyr", "apache.sedona") %in% rownames(installed.packages())))'
```

包编译失败时 `install.packages()` 只会给出警告,因此需要通过 `stopifnot()` 让 `R` 命令以失败退出,而不是报告成功。安装 R 预计会使每个节点的引导时间增加几分钟。

### 从 R 连接集群

EMR 将 Spark 安装在 `/usr/lib/spark` 下。从 R 连接时,Spark driver 以 YARN client 模式运行在运行 R 的节点上。上面的 `spark.yarn.dist.jars` 设置只会把 jar 分发给 executor,并不会把 Sedona 加入 driver 的 classpath。`apache.sedona` 会自行把 jar 加入 driver:要么从 Maven Central 下载,要么使用 `SEDONA_JAR_FILES` 中列出的本地文件。请将 `SEDONA_JAR_FILES` 指向引导脚本已经下载到 `/jars` 的 jar 包:

```r
library(sparklyr)
library(apache.sedona)

Sys.setenv(
"SEDONA_JAR_FILES" = paste(
"/jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar",
"/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar",
sep = ":"
)
)

sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark")
```

上面的 Jupyter 示例不需要这一步,因为 EMR 以 cluster 部署模式运行 Livy,driver 运行在能收到这些 jar 的 YARN 容器中。

!!!note
`SEDONA_JAR_FILES` 是一个以 `:` 分隔的 jar 列表,设置它只会替换 Sedona 的 Maven 坐标。`apache.sedona` 仍会通过 `--packages` 请求 `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}`,因此无论如何 driver 都需要能访问 Maven Central,或者 Ivy 缓存中已有该 jar。在 `SEDONA_JAR_FILES` 中列出 GeoTools wrapper 的 jar 并无害处。如果不设置 `SEDONA_JAR_FILES`,`apache.sedona` 还会请求 `org.apache.sedona:sedona-spark-shaded-<spark 版本>_<scala 版本>:{{ sedona.current_version }}`。Ivy 会按用户缓存下载的 jar,因此只有首次连接需要下载,但在网络较慢时,首次连接可能超过 `sparklyr.connect.timeout` 的默认值。

### 验证 R 端安装

```r
sdf_sql(sc, "SELECT ST_Point(0.0, 0.0) AS geom") %>% collect()
```

关于 R 接口的更多功能,请参阅 [Sedona R 文档](https://sedona.apache.org/latest/api/rdocs/)。
Loading