[feat][client] PIP-478: asynchronous v5 client authentication and TLS factory integration (core migration)

[lhotari]

PIP: #25890 (pip/pip-478.md)

Motivation

This is the third implementation PR for PIP-478, following #26222 (the pulsar-tls-factory-api / pulsar-http-client-api SPI modules) and #26271 (the default FileBasedTlsFactory implementation).

The two PRs so far were additive: the new SPI and its default implementation landed with no consumers, and PIP-337's PulsarSslFactory / SecurityUtility stack remained the TLS path actually in use. This PR is the migration itself — it wires the asynchronous v5 authentication interface and the PulsarTlsFactory SPI into the client, admin, broker, proxy, WebSocket proxy and functions worker, so the new SPI becomes the path that runs.

That is also what turns the engine and provider selection added in #26271 into something an operator can configure end to end, and what makes PIP-478's secure-by-default posture (hostname verification on, SAN-only matching) real.

Modifications

Asynchronous v5 client authentication.

• pulsar-client-api-v5 gains the org.apache.pulsar.client.api.v5.auth SPI: Authentication, AuthenticationData, AuthChallenge / ChallengeResponse, the binary and HTTP challenge handlers, and the init/call contexts. Every operation that can perform I/O returns a CompletableFuture, so an authentication plugin never blocks an event-loop thread — the core motivation of the PIP.
• pulsar-client implements it in org.apache.pulsar.client.impl.auth.v5: BinaryAuthenticationExchange (the binary-protocol challenge/response engine), HttpAuthenticationDriver + FrameworkHttpClient (the HTTP side, on the PulsarHttpClient SPI), the built-in token/basic plugins, and AuthMetrics.
• v4 compatibility runs both ways: LegacyV4AuthenticationAdapter drives an existing v4 plugin from the v5 engine, and V5ToV4AuthenticationAdapter lets a v5 plugin serve the v4 client. Existing third-party v4 plugins keep working unchanged.

Server-side and client-side TLS factory integration.

• Broker: PulsarService, BrokerService, PulsarChannelInitializer and WebService acquire TLS through DefaultBrokerTlsFactory / JettyTlsFactory instead of PulsarSslFactory.
• Proxy: ProxyService, DirectProxyHandler and AdminProxyHandler (Jetty SslContextFactory.Client), plus ProxyTlsFactories for the proxy's own purposes.
• WebSocket proxy and functions worker acquire their web-listener and broker-client material the same way.
• Client and admin: ClientCnx, ConnectionPool, PulsarClientImpl, HttpClient and the admin HTTP connector.

Configuration surface.

• The PIP-337 sslFactoryPlugin / sslFactoryPluginParams / brokerClientSslFactoryPlugin / brokerClientSslFactoryPluginParams config keys are removed, together with the ClientBuilder / PulsarAdminBuilder sslFactoryPlugin(...) / sslFactoryPluginParams(...) methods and the pulsar-admin clusters --tls-factory-plugin / --tls-factory-plugin-params CLI options. A stale non-default value in a config file is rejected at startup with a migration pointer rather than silently ignored. The ClusterData.brokerClientSslFactoryPlugin / ...Params accessors are the one exception: they stay on the metadata schema for wire compatibility, now @Deprecated, and a value read from the store is ignored with a WARN rather than making a cluster unloadable.
• Their successors arrive: tlsFactoryClassName / tlsFactoryConfig (and brokerClientTlsFactoryClassName / brokerClientTlsFactoryConfig) select a PulsarTlsFactory by name, on the server configs and on the v4 client and admin builders.
• A new jsseProvider key (client, broker, proxy, and the brokerClient* variants) names the JSSE (SSLContext) java.security.Provider. tlsProvider stays overloaded for v4 parity: a value that is not a Netty engine literal is routed to jsseProvider, which wins when both are set.
• ClusterData gains brokerClientTlsFactoryClassName / brokerClientTlsFactoryConfig, so a deployment can drive connections to one remote cluster through its own factory. A cluster entry already carries that cluster's TLS material; these select the factory that consumes it, for both broker-client legs the entry drives — the binary replication client and the cross-cluster admin (HTTPS) client. A blank value inherits the broker-level setting rather than reverting to the default factory: the factory selects the mechanism rather than the material, so a deployment with a custom broker-client factory must not silently downgrade because a cluster entry did not repeat it. pulsar-admin clusters create|update expose them as --tls-factory-class-name / --tls-factory-config.

Default value changes (all called out in the PIP's Upgrade section):

• Hostname verification is on by default — tlsHostnameVerificationEnabled (broker/proxy/websocket), tlsEnableHostnameVerification (client), tlsEnableHostnameVerification (functions worker) — and CN-based matching is gone, so a server certificate must carry a matching SubjectAltName. This is the secure-by-default hardening PIP-478 exists to deliver; the remediation (reissue certificates with a SAN, or opt out per component) is documented in the PIP.
• webServiceTlsProvider (broker and proxy) and the WebSocket proxy's tlsProvider no longer default to Conscrypt. Under PIP-337 that default only reached Jetty's SslContextFactory.setProvider(...), which is inert on a factory that overrides getSslContext() with a pre-built context — so it never actually selected a provider. This PR makes those keys authoritative on the JSSE axis, and conscrypt-openjdk-uber ships native libraries for x86_64 only, so keeping the default would fail the web listener at startup on aarch64 (Apple silicon, ARM servers) and s390x. Unset selects the JVM default, which is what deployments have effectively been running; an operator who wants Conscrypt configures it explicitly and still gets a loud failure where it cannot be loaded.

Test fixtures. tests/certificate-authority/ec/server.cert.pem and jks/broker.keystore.jks are regenerated with a localhost SubjectAltName — they previously identified the host only through the CN, which SAN-only verification rejects. generate_keystore.sh records why only the broker keystore needs a SAN (the client and proxy keystores are client identities and are never hostname-verified).

Not in this PR. The PIP-337 classes themselves (PulsarSslFactory, DefaultPulsarSslFactory, SecurityUtility, KeyStoreSSLContext) still exist, now unused by production code; deleting them, along with migrating the Athenz and SASL plugins to the v5 SPI, are the remaining PRs in the series.

Review follow-ups (since the first review)

The review on this PR found three functional regressions and a set of consistency items. All confirmed ones are fixed here, each mutation-verified:

• Per-cluster TLS factory selection never took effect. BrokerService resolved the cluster's choice onto the configuration, but both PulsarService apply methods read ServiceConfiguration directly — so a broker-level factory silently discarded the cluster override, and a blank broker-level key left the custom factory unwrapped, making a compliant BROKER_CLIENT-only factory answer CLIENT_DEFAULT with empty and fail the connection. The class name and its config now also resolve atomically.
• PulsarAdmin dropped an OAuth2 plugin's IdP TLS material (regression vs v4, [feat][client] oauth2 trustcerts file and timeouts #24944). The framework HTTP client factory reads conf.getTlsFactory(), which the admin path never sets, and binding it disables the StandaloneOAuth2HttpClientFactory fallback that honours idpTlsPolicy(). Such a plugin is now left unbound — which also covers a plaintext http:// admin URL.
• The proxy bound its listeners before building its broker-client TLS, so a connection accepted in that window saw a null SslContext/factory. Master is unaffected: it held no broker-client TLS state and built per remote host lazily.
• A 4.x-serialized AuthenticationToken stayed deserializable (the supplier move renamed classes, which Java serialization resolves by name).
• LegacyV4TlsAdapter now starts and closes the v4 plugin it wraps.
• The promised-but-never-emitted WARN for a stale per-cluster brokerClientSslFactoryPlugin; five pip-478.md passages that still claimed the ClusterData fields were removed; and jsseProvider documented in conf/functions_worker.yml.

One reported item is not a defect: HttpAuthenticationDriver / AsyncHttpAuthenticationProvider look unused because their implementor is the next PR in the series — AuthenticationSasl implements the provider and SaslAuthenticationV5 implements HttpAuthChallengeHandler, which the driver consumes.

Verifying this change

This change added tests and can be verified as follows:

• New unit coverage for the v5 authentication engine (challenge/response state machines, the v4↔v5 bridges, the HTTP driver) and for the TLS wiring on each component (ProxyTlsFactoriesTest, ProxyTlsFactoryMetricsTest, DefaultBrokerTlsPolicyTest, TlsFactorySupportTest).
• DefaultBrokerTlsPolicyTest#theDefaultConfigurationPinsNoJsseProviderOnAnyPurpose locks down the provider-default change; PerfClientUtilsTest#hostnameVerificationAloneDoesNotEnableTls locks down that hostname verification being on by default does not imply TLS intent. Both were verified to fail against the unfixed code.
• TestCmdClusters#testTlsFactoryOptions and ReplicatorTlsFactoryTest#perClusterTlsFactorySelectsTheFactoryWhenTheBrokerLevelSettingIsUnset cover the per-cluster factory: CLI parsing through to the ClusterData sent to the admin API, and the resolution rule across cluster-only, broker-only and both-set. Both were verified to fail against the unfixed code — the both-set case is what catches inverted precedence.
• Existing TLS/auth suites now exercise the new stack end to end, including TlsWithECCertificateFileTest, the keystore-TLS tests, TlsProducerConsumerTest, AdminApiTlsAuthTest, ReplicatorTlsTest and the proxy authentication tests.
• Local: :pulsar-common:test :pulsar-broker-common:test :pulsar-client-original:test :pulsar-client-admin-original:test :pulsar-client-tools:test :pulsar-testclient:test :pulsar-websocket:test :pulsar-client-v5:test :pulsar-functions:pulsar-functions-worker:test — 2726 tests, 0 failures. assemble rat spotlessCheck checkstyleMain checkstyleTest checkBinaryLicense passes.
• Full CI (Personal CI, see below) is green across all unit, integration and system suites.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API — adds the org.apache.pulsar.client.api.v5.auth SPI, and adds brokerClientTlsFactoryClassName / brokerClientTlsFactoryConfig to ClusterData (accessors plus builder methods); removes the v4 ClientBuilder.sslFactoryPlugin(...) / sslFactoryPluginParams(...) and the PulsarAdminBuilder equivalents (a source-compatibility break on upgrade, inventoried in the PIP). The ClusterData factory-plugin accessors are deprecated and ignored rather than removed, so no existing metadata field changes meaning.
• The schema
• The default values of configurations — hostname verification now defaults to on (client, broker, proxy, websocket, functions worker), and webServiceTlsProvider / the websocket tlsProvider no longer default to Conscrypt. Both are detailed under Modifications above and in the PIP's Upgrade section.
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options — pulsar-admin clusters create|update lose the PIP-337 --tls-factory-plugin / --tls-factory-plugin-params and gain --tls-factory-class-name / --tls-factory-config, which write the new per-cluster ClusterData fields.
• The metrics
• Anything that affects deployment — a deployment whose server certificates carry the hostname only in the CN will fail TLS after upgrade until the certificates are reissued with a SubjectAltName, and configuration that sets a removed PIP-337 key is rejected at startup.

Documentation

• doc-required — the configuration surface changes (new tlsFactoryClassName / tlsFactoryConfig / jsseProvider keys, removed sslFactoryPlugin* keys) and the secure-by-default hostname-verification behaviour need to be reflected in the security documentation on the website.
• doc-not-needed
• doc
• doc-complete

Matching PR in forked repository

PR in forked repository: lhotari#249 (Personal CI — full matrix green)

_{This PR was prepared with the assistance of Claude Code (Opus 5); the change was reviewed and is submitted by a human contributor who takes responsibility for it, per the ASF Generative Tooling guidance.}

[david-streamlio]

Reviewed the full diff against bab5c77631 (215 files) in a worktree at d4ce308dc9; ./gradlew quickCheck passes. This is an impressively disciplined migration — the offload discipline, the ctor-throw cleanup on the TLS initializers, the use-after-free pinning around newHandler/newEngine, and the generation/liveness guards in the ClientCnx state machine all read as carefully thought through. The notes below are what I found; the first three look like genuine functional regressions, the rest are consistency/robustness/doc items.

Per-cluster TLS factory selection does not reach the cross-cluster admin client

The PR describes ClusterData.brokerClientTlsFactoryClassName as selecting the factory "for both broker-client legs the entry drives — the binary replication client and the cross-cluster admin (HTTPS) client." The admin leg does not honour it.

In BrokerService.getClusterPulsarAdmin, configAdminTlsSettings(...) writes the resolved (cluster-preferred) value onto the builder's conf via adminBuilder.tlsFactoryClassName(...) / .tlsFactoryConfig(...). Then BrokerService.java:1918 calls pulsar.applyBrokerClientTlsFactoryToAdmin(builder), which at PulsarService.java:1915 is gated on — and parameterised by — the broker-level getConfiguration().getBrokerClientTlsFactoryClassName():

• broker-level set, cluster overrides it → conf.setTlsFactory(ClientTlsFactorySupport.brokerClientTlsFactory(conf, /* broker-level */ factoryClassName)) and conf.setTlsFactoryParams(parseFactoryConfig(/* broker-level */ ...)). resolveClientTlsFactory short-circuits on a non-null conf.getTlsFactory(), so the per-cluster name and config that configAdminTlsSettings just wrote are never read. The cluster override is silently dropped.
• broker-level blank, cluster names a factory → the method returns early, and AsyncHttpConnector later resolves by name through resolveClientTlsFactory(conf, ..., /* brokerClientPurpose */ false). So the custom factory is not wrapped in BrokerClientPurposeFactory. By the reasoning in ClientTlsFactorySupport#wrapBrokerClientPurpose, a compliant custom factory serving only BROKER_CLIENT then returns empty() for the transport's CLIENT_DEFAULT request — a hard connect failure.

ReplicatorTlsFactoryTest#perClusterTlsFactorySelectsTheFactoryWhenTheBrokerLevelSettingIsUnset is a good test, but it only exercises getReplicationClient; nothing covers getClusterPulsarAdmin, which is why this is invisible in CI.

Admin-only OAuth2 loses its IdP TLS material

PulsarAdminImpl.java:466 builds the FrameworkHttpClientFactory with conf::getTlsFactory as the TLS supplier. Nothing ever sets a factory on that conf on the admin path: AsyncHttpConnector.resolveNewTlsFactory keeps its factory in its own field rather than on the conf, and it runs after auth.start() anyway. So hasTlsFactory() is false, FrameworkHttpClientFactory.configureTls takes the legacy branch, and because the purpose is CLIENT_OAUTH2 rather than CLIENT_DEFAULT it applies nothing — the IdP connection falls back to the platform default trust store.

Binding a factory at all is also what disables the fallback that would have handled this: FlowBase.resolveHttpClientFactory() only reaches StandaloneOAuth2HttpClientFactory (the one place idpTlsPolicy() is consumed) when httpClientFactory == null.

Net effect: a PulsarAdmin using OAuth2 with trustCertsFilePath / tlsCertFile / tlsKeyFile in its authParams silently stops honouring them. v4 honoured them via FlowBase.defaultHttpClient, and PulsarClientImpl explicitly preserves the behaviour through hasOAuth2IdpTlsMaterial() / foldOAuth2IdpPolicy — the admin path has no equivalent. AdminOnlyOAuth2AuthTest uses a plaintext mock IdP, and none of OAuth2IdpTlsFoldTest / OAuth2IdpTlsFrameworkClientTest / OAuth2IdpTlsPlaintextBrokerTest involves PulsarAdmin.

tlsCertFile / tlsKeyFile pairing is no longer validated

The removed FlowBase.defaultHttpClient threw on a half-configured pair:

if (hasCertFile != hasKeyFile) {
    throw new IllegalArgumentException("Invalid TLS client certificate configuration: "
        + CONFIG_PARAM_CERT_FILE + " and " + CONFIG_PARAM_TLS_KEY_FILE + " must be provided together");
}

FlowBase.idpTlsPolicy() (line 154) replaces it with if (isNotBlank(certFile) && isNotBlank(keyFile)), and hasOwnTlsMaterial() returns true when either is set. A config with only one of the two now yields a policy with no client identity and no error — the mTLS handshake to the IdP silently degrades instead of failing at startup. Nothing else in the tree validates the pair (TlsClientAuthFlow.fromParameters doesn't either, and TlsClientAuthFlowTest lost its factory-mocking wrapper).

tlsCertRefreshCheckDurationSec=0 now means two different things

DefaultBrokerTlsFactory.refreshIntervalSeconds maps <= 0 to 0, with a deliberate comment: "an operator who set 0 still gets no poll." The other three components do the opposite:

• ProxyTlsFactories.java:144 → DEFAULT_REFRESH_INTERVAL_SECONDS (60)
• ProxyServer.buildDefaultWebTlsFactory (websocket) → same
• WorkerServer.java:392 → same

Under PIP-337 all three gated the refresh task on > 0, so 0 disabled polling everywhere. This is both a behaviour change and an inconsistency with the broker's documented handling of the same key.

Proxy startup race: listeners bind before the broker-client TLS is built

ProxyService.start() binds listenChannel and listenChannelTls and only then, at line 344, builds brokerClientTlsFactory / brokerClientSslContext / lookupClientTlsFactory. A connection accepted in that window hits either TlsContextAcquisition.withPinnedContext(service::getBrokerClientSslContext, ...) in DirectProxyHandler with a null context, or a null conf.getTlsFactory() from ProxyConnection.createClientConfiguration — the javadoc on getLookupClientTlsFactory() already acknowledges it is "transiently null while it is being built at startup". Under PIP-337 DirectProxyHandler built its factory lazily per remote host, so no such window existed. Moving the isTlsEnabledWithBroker() block above the binds closes it.

Per-cluster factory class and its config fall back independently

BrokerService.java:1756 resolveBrokerClientTlsFactory is applied separately to ...ClassName and ...Config. A ClusterData that sets only the class name therefore inherits the broker-level brokerClientTlsFactoryConfig — factory A's init params handed to factory B. Given the reasoning in the method's own javadoc (don't silently change the TLS mechanism), the pair should probably resolve atomically: if the cluster supplies a class name, its config wins even when blank.

The documented WARN is never emitted

ClusterData#getBrokerClientSslFactoryPlugin, the ClusterDataImpl @Schema descriptions, the PR body, and pip-478.md §1192 all say a stale value read from the store is "ignored with a WARN". Grepping every non-test main source for sslFactoryPlugin turns up only field declarations and comments — nothing logs. Since this is explicitly called out as the one place a stale PIP-337 value cannot fail loud, the WARN is the whole remediation signal.

pip-478.md contradicts itself and this PR

§1238 and §1241 still state that the ClusterData factory fields are removed from the metadata model along with their accessors and builder methods, that the pulsar-admin clusters options are removed, and that "per-cluster factory selection no longer exists — set the factory broker-level." §1192 and this PR do the opposite: deprecate-and-retain the old fields and add per-cluster selection plus --tls-factory-class-name / --tls-factory-config. The PIP is the normative spec and this PR already touches it, so those two sections should be updated in the same change.

Smaller items

• ProxyServer.createTlsFactoryWebServer (line 239) leaks on partial init failure. WebService.createTlsFactoryWebServer, ServiceChannelInitializer.initializeTlsFactory and the broker's PulsarChannelInitializer.initializeTlsFactory all wrap the post-createFactory steps in a try/catch that disposes the subscription and closes the factory. ProxyServer doesn't, and its caller rethrows as PulsarServerException without ever reaching close(), so the cert watchers survive a failed startup.
• The v4 auth shims rebuild the driver per connection attempt. AuthenticationToken, AuthenticationBasic and AuthenticationOAuth2 each do new V5BinaryAuthenticationDriver(new XxxV5(...), authServices).newAuthenticationExchange(host) inside newAuthenticationExchange. That constructs a fresh AuthMetrics — two OpenTelemetry instrument builds — and re-runs ensureInitialized()'s initializeAsync().join() on every connection attempt, including reconnect storms. HttpAuthenticationDriver's own javadoc states the intended shape ("One instance is created per plugin and reused across requests"); caching the driver in a field would match it.
• ensureInitialized() joins on the calling thread in both V5BinaryAuthenticationDriver and HttpAuthenticationDriver. The comments justify it by "the built-in bodies complete immediately", which holds for the built-ins, but nothing in the Authentication SPI javadoc makes that a contract — a third-party v5 plugin doing I/O in initializeAsync would block whatever thread first opens an exchange, which is the hazard the PIP exists to remove. Worth either stating it as a hard contract on initializeAsync or hoisting init off the call path.
• LegacyV4AuthenticationAdapter.wrap routes by auth-method name. Only the literal "sasl" gets LegacyV4ChallengeResponseAdapter; every other name falls to LegacyV4CredentialAdapter, which does not expose BinaryAuthChallengeHandler. A third-party v4 multi-round plugin under a different method name will fail the first CommandAuthChallenge with "does not expose BinaryAuthChallengeHandler". Separately, LegacyV4TlsAdapter forwards configure(...) but never v4.start() or v4.close(), so a custom plugin reporting method name "tls" is neither started nor closed.
• LegacyV4CredentialAdapter.getHttpHeadersAsync calls the deprecated no-arg v4.getAuthData() even though the HttpAuthCallContext carries requestUri(), dropping per-host credential selection.
• Per-host client TLS material. The old client keyed a PulsarSslFactory per SNI host and fed it getAuthData(host); DirectProxyHandler did the same per remote host. Both now share one factory/context with only a TlsEndpoint hint. Is dropping per-host client identity intentional? If so it's probably worth a line in the PIP's upgrade notes.
• newConnectCommand() → buildConnectCommand(AuthData) silently breaks out-of-tree ClientCnx subclasses (all in-tree callers are updated). A deliberate 5.0 break, but it isn't in the PR's public-API inventory.
• PulsarClientProvider.authenticationTls(String, String) → authenticationTls() is a v5 public-API signature change not called out under Modifications.
• jsseProvider is missing from conf/functions_worker.yml even though WorkerConfig gained the field and broker/proxy/standalone/client all document it.
• Deleted tests for still-present classes. KeyStoreTlsTest, JettySslContextFactoryTest and JettySslContextFactoryWithKeyStoreTest are removed here, but the PIP-337 classes they cover (KeyStoreSSLContext, and JettySslContextFactory — that one is deleted, so its tests going is fine) remain in the tree until a later PR. Was there a reason not to keep KeyStoreTlsTest until KeyStoreSSLContext itself goes?