Repository navigation
Conversation
Add a new configuration `spark.kubernetes.driver.service.publishNotReadyAddresses` (boolean, default `false`). When enabled: - `DriverServiceFeatureStep` sets `spec.publishNotReadyAddresses: true` on the headless driver service, so DNS records for the driver pod are published even while the pod is not Ready. - The driver pod readiness wait before executor allocation (introduced in SPARK-32975) is skipped in `ExecutorPodsAllocator`, `StatefulSetPodsAllocator`, and `DeploymentPodsAllocator`, since the sole purpose of that wait — ensuring the headless service is resolvable by DNS before executors start — no longer depends on pod readiness. Kubernetes publishes DNS records for a headless service only for Ready pods. Users commonly attach a readiness probe to the driver pod through `spark.kubernetes.driver.podTemplateFile` — a typical setup for long-running Spark Connect servers, where pod readiness gates traffic routing. When such a probe gates on a port that binds after `SparkContext` initialization completes (e.g. the Spark Connect gRPC port), executors launched during initialization cannot resolve the driver service and die with `UnknownHostException`. With static allocation, `maxNumExecutorFailures = max(3, 2 * spark.executor.instances)` is reached before initialization can complete, and the driver exits with code 11 (`EXCEED_MAX_EXECUTOR_FAILURES`) on every attempt — the application can never start. The existing mitigation from SPARK-32975 (`waitUntilReady` in the allocators, tunable via `spark.kubernetes.allocation.driver.readinessTimeout` per SPARK-49079) cannot help here: the wait runs inside `SparkContext` initialization, while such a probe can only pass after initialization completes, so the wait always burns the full timeout and the race resumes. The driver service is a single-pod discovery endpoint rather than a load-balancing service, and `publishNotReadyAddresses` is the standard Kubernetes pattern for headless discovery services. There is currently no first-class way to set it — the service spec is not customizable via pod templates, and the only workarounds are re-implementing the service in a custom feature step (via `spark.kubernetes.driver.pod.excludedFeatureSteps`) or cluster-level admission webhooks. This follows the precedent of exposing driver service spec fields as configurations (SPARK-39490: `ipFamilyPolicy`/`ipFamilies`). The configuration is opt-in (default `false`) because publishing not-ready addresses is not desirable for deployments that rely on endpoint readiness to gate traffic to the driver service (e.g. through a service mesh); flipping the default could be discussed separately. No change by default. A new opt-in configuration is added and documented in `running-on-kubernetes.md`. New unit tests: - `DriverServiceFeatureStepSuite`: the default leaves the field `false`; enabling the configuration sets it to `true`. - `ExecutorPodsAllocatorSuite`, `StatefulSetAllocatorSuite`, `DeploymentAllocatorSuite`: the driver pod readiness wait still runs by default, and enabling the configuration skips it (`waitUntilReady` is not invoked). Also verified the behavior end-to-end on a real Kubernetes cluster by applying `publishNotReadyAddresses: true` to the driver service through an admission webhook (equivalent to what this configuration does): executors resolved the driver service and registered while the driver pod was still NotReady behind a Spark Connect port readiness probe, and a static-allocation application that previously failed on every attempt with exit code 11 started successfully. Yes. Co-authored with Claude Opus. Closes apache#57242 from j1wonpark/SPARK-58113-publish-not-ready. Authored-by: Jiwon Park <jpark92@outlook.kr> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 01b3c25)
…r service ports The driver service declares a fixed set of ports: the driver RPC, the block manager and the UI. A server the driver runs besides them - a Kyuubi engine's Thrift port, a debugger - has no port on the service, so whatever discovers the driver through its service, by label or annotation, has no port to find it by. spark.kubernetes.driver.service.port.<name>=<number> adds a port with that name and number, and the same target port, after the built-in ones. The ports are checked when the step is built, before the driver pod is created: a name Kubernetes would refuse (IANA_SVC_NAME), a number out of 1-65535, a name or number the driver service already declares, or two extra ports with one number fail the submission with the offending key. Only the service changes; the driver container's ports are left to the pod template, which may already declare them.
…inio It seems `minio/minio:RELEASE.2025-09-07T16-13-09Z` has gone. I just searched on Docker Hub and pick one that looks reliable as the replacement https://hub.docker.com/r/alpine/minio Recover K8s IT. No. Wait for CI. No. Closes apache#58764 from pan3793/minio-image. Authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit 1f1c57e)
giggsoff
marked this pull request as ready for review
October 9, 2026 07:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #71