feat(k8s): Driver service ports

[giggsoff]

What changes were proposed in this pull request?

Two commits:

1. Backport of SPARK-58113: spark.kubernetes.driver.service.publishNotReadyAddresses (default false). When true, the headless driver service publishes the driver pod's address before the pod is Ready, and the executor allocators skip the driver readiness wait. Cherry-picked with -x from apache/spark 01b3c2534fb. The only change from upstream: .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE) is dropped, as ConfigBindingPolicy is not imported in this branch's Config.scala.

2. New spark.kubernetes.driver.service.port.[PortName]=<number>: adds a port with that name and number (targetPort = port) to the driver service, after the built-in driver-rpc-port, blockmanager, spark-ui and spark-connect. Checked when DriverServiceFeatureStep is built, before the driver pod is created. Submission fails with the offending key if:

   • the name is not a valid Kubernetes port name (IANA_SVC_NAME: at most 15 of [a-z0-9-], at least one letter, no leading, trailing or repeated -);
   • the number is outside 1-65535;
   • the name or number is one the driver service already declares (including the Spark Connect port);
   • two extra ports share a number.

   Only the service changes. The driver container's ports are left to the pod template, which may already declare them; adding them here could duplicate a name and get the pod rejected.

Why are the changes needed?

Kyuubi (the stateless Kyuubi work, PR 40 in arenadata/kyuubi) discovers pre-deployed Spark SQL engines through Kubernetes Services: a Service labelled kyuubi/enabled=true with a port named kyuubi. The driver service is the natural one to use, since its lifetime is the driver's and its labels and annotations are already configurable. But its ports are fixed. The only workarounds:

• a hand-made Service per engine;
• or putting the engine on a port Spark already declares, which costs the Spark UI and metrics (spark-ui) or Spark Connect (spark-connect).

Separately, a readiness probe on a port that opens after SparkContext start (the engine's Thrift port, or the Spark Connect port) deadlocks startup. Executors resolve the driver through the headless service, which lists only Ready pods. That is exactly what SPARK-58113 fixes upstream (fixed in 4.3.0).

Upstream hardcoded one more port for Spark Connect (SPARK-52193). The new setting is the general form, so a server the driver runs besides Spark's own does not need another hardcoded port.

Does this PR introduce any user-facing change?

Yes, two new optional settings; nothing changes when they are unset.

# Kyuubi engine: Spark's own driver service gets a port Kyuubi discovers it by
spark.kubernetes.driver.service.port.kyuubi=10009
spark.kubernetes.driver.service.label.kyuubi/enabled=true
spark.kubernetes.driver.service.annotation.kyuubi/type=SPARK_SQL
spark.kubernetes.driver.service.annotation.kyuubi/profile=spark4
# with a readiness probe on 10009 in the driver pod spec
spark.kubernetes.driver.service.publishNotReadyAddresses=true