Repository navigation
feat: DT support - #8
Merged
Asmoday merged 28 commits intoAug 31, 2026
Merged
Conversation
…oding Writable layout matches AbstractDelegationTokenIdentifier (Text owner, renewer, realUser; vlong issueDate, maxDate; vint sequenceNumber, masterKeyId) so external tooling can wrap it into a Hadoop Token without re-encoding. Golden byte vectors in the test are generated with real Hadoop 3.4.3 WritableUtils/Text.
Token password is derived as HMAC-SHA256(masterKey, identifierBytes) and never stored. Key comes from the file named by zookeeper.tokenAuth.secretFile when zookeeper.tokenAuth.enabled is set; a misconfigured server fails at startup. Lifetime validation covers maxDate only until the token store lands.
…acket Split SecurityUtils.createSaslServer into createGssSaslServer and createDigestSaslServer; the old entry point keeps its behavior for quorum auth. ZooKeeperSaslServer now defers SaslServer creation to the first evaluateResponse: an empty first token means DIGEST-MD5, a non-empty one GSSAPI. A Kerberized server thus accepts both mechanisms with no wire protocol or client change; a server without a Kerberos principal still always negotiates DIGEST-MD5.
A token client sends base64(identifier) as the DIGEST-MD5 username and
base64(HMAC-SHA256(masterKey, identifier)) as the password. The server
callback handler strictly parses the username as a token identifier,
validates its lifetime and derives the password from the master key;
anything that does not parse falls back to the static user_* map.
AuthorizeCallback maps the token to its owner principal, so tokens land
in the existing Id("sasl", ...) scheme and current ACLs keep working.
zkTokenTool.sh mints a delegation token against the ensemble master key file and prints the identifier/password pair plus a ready JAAS client section. The e2e test drives the full path: tool-minted token, client auth through DigestLoginModule, session authorized as the token owner (sasl ACL id), with a static DIGEST user coexisting on the same server.
…t API New quorum opcodes getDelegationToken (120), renewDelegationToken (121) and cancelDelegationToken (122) with jute request/response records and the matching ZooKeeper client methods. Server-side handling follows.
Token state lives under /zookeeper/token: each token node stores the current expiry plus the identifier and replicates as ordinary txns. PrepRequestProcessor turns getDelegationToken into a multi of plain create sub-txns (creating or ACL-healing the parent on the way), renew into setData and cancel into delete, so any ensemble member can apply them. FinalRequestProcessor derives the password from the master key when answering issuance. Issuance requires a SASL session and is denied to token-authenticated sessions (marked with a 'token' auth id) to forbid token chains; renew requires the renewer principal, cancel the owner or renewer, and super may do both. A ContainerManager-style cleanup on the leader cancels expired tokens.
The HMAC password alone stays valid until maxDate, so authentication now also checks the replicated store: the token node must exist, match the presented identifier and not be past its current expiry. This is what makes cancellation and renew-based expiry effective; tokens minted offline with only the master key are rejected.
The primary mode now connects to the ensemble as a SASL client and requests a token through getDelegationToken, so the token lands in the replicated store. The offline --secret-file mode stays for diagnostics and warns that such tokens are rejected by servers.
Hadoop 3 AbstractDelegationTokenIdentifier writes a leading version byte (0) before the fields; the stage-1 golden vectors were generated against bare Text/WritableUtils and missed it. Caught by the interop test that round-trips through the real Hadoop implementation.
New module with the Hadoop-side glue for ZOOKEEPER_DELEGATION_TOKEN: ZooKeeperDelegationTokenIdentifier and ZooKeeperTokenRenewer registered via ServiceLoader (drop the jar on the YARN RM classpath to get renew and cancel), plus ZooKeeperDelegationTokens for obtaining live tokens, selecting them from Credentials and installing a JAAS configuration that lets a plain ZooKeeper client authenticate with a token. Hadoop dependencies are provided-scope. The integration test drives the whole path against a real server through the Hadoop Token API.
ZooKeeperDelegationTokenProvider (ServiceLoader-registered) obtains a ZooKeeper delegation token at submit time when spark.zookeeper.quorum is set and files it into the job credentials under the configured service alias, so it reaches executors through the standard credential distribution on both YARN and Kubernetes. Spark is a provided-scope dependency.
selectTokenFromUgi/installTokenFromUgi pick the ZooKeeper token for a service from the current UGI credentials — the one-liner a YARN container or Spark executor needs to authenticate to ZooKeeper with the distributed job token.
Follower/Observer request processors did not forward the new opcodes to the leader and CommitProcessor did not wait for their commit, so on a follower a token request fell through to FinalRequestProcessor as a read with no txn. Caught by the new quorum test, which issues through a follower, authenticates the token against every ensemble member and cancels through a follower.
The restart test proves the token store survives a snapshot/txn log reload: an issued token keeps authenticating, renewing and cancelling after the server restarts. The MiniKdc test runs a Kerberized server and shows GSSAPI and delegation token clients coexisting through the lazy mechanism selection, with the token session authorized as the Kerberos principal that minted it.
New section covering how tokens work (DIGEST-MD5 over an HMAC-derived password, replicated store under /zookeeper/token), the issue/renew/ cancel API and zkTokenTool, the Hadoop/Spark integration jar, the tokenAuth.* configuration options and the security caveats (TLS recommended, FIPS incompatibility, rolling upgrade order, follower replication window).
ReadOnlyRequestProcessor did not list the token opcodes as writes, so a read-only server would prep and apply the token txn locally without a quorum. Reply with NOTREADONLY like any other write.
Key rotation mode: leader-generated keys replicated in the token store, referenced from identifiers by masterKeyId. Key id 1 stays reserved for the static secret file, so pre-rotation tokens keep validating. Key entry: version byte + created + expiry + key bytes; key znodes carry a no-permission ACL. With rotation enabled the secret file is optional.
Internal-only opcode rollDelegationTokenKey (sessionId 0 + super only): prep turns it into a multi of plain create/delete sub-txns that creates the next key znode and prunes keys past their expiry, so rolls replicate like any other write. The leader-side cleanup posts a roll when the newest valid key is missing or older than the roll interval. Issuance signs with the newest valid key and bootstraps the first key inside its own txn; the response password is derived with the key the identifier references.
The callback handler resolves the signing key from the identifier's masterKeyId: the static file key for id 1, otherwise the key entry from the replicated store (rejected when unknown or past its expiry).
Bootstrap on first issuance, roll through the write path with old tokens still authenticating, restricted key znode ACL, and pruning of expired keys on roll.
FIPS-friendly mechanism (SHA-256/HMAC/PBKDF2 only): the server salts the callback-provided password per session, so the existing DIGEST callback handler — including delegation token validation and the token session marker — serves SCRAM unchanged. No channel binding, no security layer, no JCA provider registration. Verified against the RFC 7677 example exchange.
The lazy mechanism selection now recognizes a SCRAM client-first packet by its GS2 header byte (n/y/p); GSSAPI tokens always start with 0x60 and DIGEST-MD5 clients with an empty packet, so the three coexist with no wire change. Clients opt in with zookeeper.sasl.client.mechanism= SCRAM-SHA-256 (quorum SASL stays DIGEST-MD5); the SCRAM path works with FIPS mode on, which is the point. End-to-end test runs static-user and delegation token auth with fips-mode enabled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.