-
Notifications
You must be signed in to change notification settings - Fork 784
[GH-1024] Add Sedona R instructions to the AWS EMR setup page #3334
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+102
−0
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -78,3 +78,54 @@ spark.sql("SELECT ST_Point(0, 0)").show() | |
| ``` | ||
|
|
||
| 注意:您不需要再调用 `SedonaRegistrator.registerAll(spark)` 或 `SedonaContext.create(spark)`,因为配置中的 `org.apache.sedona.sql.SedonaSqlExtensions` 已经为您完成了这些工作。 | ||
|
|
||
| ## 在 R 中使用 Sedona | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same edits needed here: libcurl-devel, |
||
|
|
||
| [`apache.sedona`](https://cran.r-project.org/package=apache.sedona) R 包是一个 [`sparklyr`](https://spark.rstudio.com) 扩展。只要在 `spark_connect()` 之前加载它,Sedona 的序列化器、UDT 与 UDF 就会自动注册,因此 R 中不需要手动调用与 `SedonaContext.create()` 等价的方法。 | ||
|
|
||
| !!!note | ||
| CRAN 上的 `apache.sedona` 1.9.1 版本仅支持 Spark 3.x。所有 EMR 7.x 版本都搭载 Spark 3.5,因此均可使用。 | ||
|
|
||
| ### 扩展初始化脚本 | ||
|
|
||
| 在上面的引导脚本中追加以下内容。与脚本的其余部分一样,它会在集群的每个节点上运行。从 R 执行 Spark SQL 查询时,只有运行 R 的节点需要这些 R 包;只有在使用 `spark_apply()` 于 executor 上运行 R 代码时,工作节点才需要它们。 | ||
|
|
||
| ```bash | ||
| # 安装 R 以及 Sedona 的 R 接口。sparklyr 依赖的 R curl 包需要 libcurl-devel 才能编译。 | ||
| sudo yum install -y R libcurl-devel | ||
| sudo R -e 'install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org", Ncpus = parallel::detectCores()); stopifnot(all(c("sparklyr", "apache.sedona") %in% rownames(installed.packages())))' | ||
| ``` | ||
|
|
||
| 包编译失败时 `install.packages()` 只会给出警告,因此需要通过 `stopifnot()` 让 `R` 命令以失败退出,而不是报告成功。安装 R 预计会使每个节点的引导时间增加几分钟。 | ||
|
|
||
| ### 从 R 连接集群 | ||
|
|
||
| EMR 将 Spark 安装在 `/usr/lib/spark` 下。从 R 连接时,Spark driver 以 YARN client 模式运行在运行 R 的节点上。上面的 `spark.yarn.dist.jars` 设置只会把 jar 分发给 executor,并不会把 Sedona 加入 driver 的 classpath。`apache.sedona` 会自行把 jar 加入 driver:要么从 Maven Central 下载,要么使用 `SEDONA_JAR_FILES` 中列出的本地文件。请将 `SEDONA_JAR_FILES` 指向引导脚本已经下载到 `/jars` 的 jar 包: | ||
|
|
||
| ```r | ||
| library(sparklyr) | ||
| library(apache.sedona) | ||
|
|
||
| Sys.setenv( | ||
| "SEDONA_JAR_FILES" = paste( | ||
| "/jars/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar", | ||
| "/jars/geotools-wrapper-{{ sedona.current_geotools }}.jar", | ||
| sep = ":" | ||
| ) | ||
| ) | ||
|
|
||
| sc <- spark_connect(master = "yarn", spark_home = "/usr/lib/spark") | ||
| ``` | ||
|
|
||
| 上面的 Jupyter 示例不需要这一步,因为 EMR 以 cluster 部署模式运行 Livy,driver 运行在能收到这些 jar 的 YARN 容器中。 | ||
|
|
||
| !!!note | ||
| `SEDONA_JAR_FILES` 是一个以 `:` 分隔的 jar 列表,设置它只会替换 Sedona 的 Maven 坐标。`apache.sedona` 仍会通过 `--packages` 请求 `org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}`,因此无论如何 driver 都需要能访问 Maven Central,或者 Ivy 缓存中已有该 jar。在 `SEDONA_JAR_FILES` 中列出 GeoTools wrapper 的 jar 并无害处。如果不设置 `SEDONA_JAR_FILES`,`apache.sedona` 还会请求 `org.apache.sedona:sedona-spark-shaded-<spark 版本>_<scala 版本>:{{ sedona.current_version }}`。Ivy 会按用户缓存下载的 jar,因此只有首次连接需要下载,但在网络较慢时,首次连接可能超过 `sparklyr.connect.timeout` 的默认值。 | ||
|
|
||
| ### 验证 R 端安装 | ||
|
|
||
| ```r | ||
| sdf_sql(sc, "SELECT ST_Point(0.0, 0.0) AS geom") %>% collect() | ||
| ``` | ||
|
|
||
| 关于 R 接口的更多功能,请参阅 [Sedona R 文档](https://sedona.apache.org/latest/api/rdocs/)。 | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confirmed, no change needed. One side effect worth knowing: attaching the package sets
spark.kryo.registratortoSedonaVizKryoRegistratorthrough an R option, which sparklyr passes as--conf, overriding theSedonaKryoRegistratorfrom spark-defaults. Harmless, it's a superset.