fix(cluster): consume the join token only on successful join

[hmlanigan]

Fixing a bug seen in sunbeam: clusterd followers fail admission, then retries fail with CoreTokenRecord not found

Root cause

The cluster join is not retryable because the one-shot join token is consumed
too early and the rollback path consumes it again.

In microcluster, admission happens in clusterPost
(internal/rest/resources/cluster.go), which runs on the leader. Its
transaction:

1. looks up the token (GetCoreTokenRecord),
2. creates the CoreClusterMember record (role pending),
3. deletes the CoreTokenRecord — all before the follower has actually
   joined the dqlite cluster.

Control then returns to the follower's controlPost
(internal/rest/resources/control.go), which still has to run
setupLocalMember and StartAPI → d.db.Join (the dqlite join and
Upgrade: dqlite handshake to peers on port 7000). When that fails — as in
this report, "database-upgrade request … port 7000: connection refused" — the
deferred reverter tears the member back out via
DeleteClusterMember(..., force=true) (which lands in clusterMemberDelete)
and re-execs the daemon, leaving it "uninitialized", and POST /core/control
returns HTTP 500.

The key detail: clusterMemberDelete also runs on that rollback path. If it
deletes the join token (as an earlier iteration of the fix did), then after the
rollback both the member record and the token are gone. The charm's
join_node_to_cluster retries with the same token, GetCoreTokenRecord
returns HTTP 404 "CoreTokenRecord not found", and the node can never rejoin.
The charm does not regenerate the token (add_node_to_cluster skips generation
whenever the {unit}.join_token peer-data key still exists), so the units
block permanently.

Investigation: consume only on success, not "tolerate consumed"

Two failure windows exist:

• Pre-dqlite-join (this report). d.db.Join fails before the member is
  ever registered in dqlite, so its role never leaves pending. The token must
  survive so the follower can retry.
• Post-dqlite-join. d.db.Join succeeds, the member gets a dqlite role and
  the heartbeat confirms it; only then does the token get consumed. If the
  follower fails after that point it has already joined, so a retry should be
  rejected, not servo'd a fresh join.

Treating a missing token as "re-issue for the existing member" (the earlier
approach) is wrong:

• In the normal rollback case it can never fire — rollback removes the member
  too, so there is no existing member to re-issue against.
• In the one case it could fire (member already confirmed, token gone), it
  would incorrectly authorise a duplicate join of an already-joined member.

Conclusion: the token is the sole credential authorising a join. It should be
consumed exactly once — when the join succeeds — and a missing token should
always be rejected.

Fix

1. Admission (clusterPost / admitClusterMember) never deletes the token.
   A missing token is rejected outright — no fallback to an existing member.
   (Member creation remains idempotent so a replayed admission whose member
   record survived reuses it, and mismatched certificates/addresses are still
   rejected.)
2. Confirmation (beginHeartbeat) deletes the token when the member's dqlite
   role transitions out of pending — the single consumption point, i.e.
   "consume only on successful join".
3. Removal/rollback (clusterMemberDelete) no longer deletes the token, so
   a failed join's rollback preserves the token and the follower can retry.

This is covered by unit tests in
internal/rest/resources/cluster_test.go:

• createsMemberAndKeepsToken
• retryAfterRollback (the reported scenario)
• consumedTokenWithExistingMember (consumed token is rejected)
• consumedTokenWithoutMember (unknown token is rejected)
• existingMemberMismatch (mismatched cert / address is rejected)
• expiredToken, sanMismatch

Tradeoff

A member that is aborted while still pending (e.g. operator scales down
mid-join and it never confirms or retries) leaves its token record behind
(no expiry). This is harmless: the secret is no longer known to anyone and the
token is bound to a member name, and a confirmed-then-removed member has
already had its token deleted at confirmation.

The same defect exists in v2.2.0, v2, v3, and v4 of microcluster (all still
delete the token in the admission transaction with no retry path)

Assisted-by: openrouter/deepseek-pro-latest

bug fix: LP 2169835
OPEN-4867

Test run compiling the change into sunbeamd: https://pastebin.canonical.com/p/KMZNkcXfg9/

[roosterfish]

Hey @hmlanigan have you tried using the latest LTS version v3?
I would first like to understand the root cause of this. In the linked launchpad issue there is this sentence:

Both followers initially received HTTP 500 from POST /core/control.

Can you please share the returned error message? It could be the leader wasn't yet ready/started when performing the join operation. There is the (m *MicroCluster) Ready func exactly for this purpose. It's worth giving it a try first.

[hmlanigan]

@roosterfish I have not tried v3 because we're at the last stages of the 2026.1 release and trying to get a release candidate. Trying to move to v3 at this point is too risky for our testing. As well, AI tells me that the problem exists in all version of the library.

Working on the other questions asked.

[hmlanigan]

@roosterfish

The logs I'm working off are from SolQA and hard to sort thru, sent DeepSeek AI after the answer:

The returned body isn't in the sosreport. The charm posts to /core/control with Python requests and only calls raise_for_status(), so the only thing logged is the status line (500 Server Error: Internal Server Error for url: .../core/control); it never logs response.text, and clusterd doesn't log the error it returns (debug off). I searched all three node bundles for every string StartAPI can return and none is present.

From the journals I can still tell you what it was. The join passed d.db.Join and the PreJoin hook but never reached PostJoin, so StartAPI failed between them — i.e. in trust confirmation or the OnNewMember notification loop. That means the body was one of:

Failed to confirm new member "" on any existing system (): , or
Failed to notify any existing cluster member at "10.241.128.84:7000"
with error_code: 500. The only concrete daemon error is Follower 2 failing to reach Follower 1: Failed to send database upgrade request ... https://10.241.128.85:7000/core/internal/database: connect: connection refused.

On the leader-readiness idea: the leader was up well before the joins — bootstrapped 10:00:12–14 and heartbeating at 10:00:24/:34/:44/:55 and 10:01:05/:15, while the joins came at 10:01:17/18. So it looks like a join/rollback race rather than the leader not being Ready

From what I understand Ready() is used to see in the local cluster is ready. It has no indication that the remote cluster can be joined successfully. Please confirm.

DeepSeek is also telling me that adding a Ready() equivalent might shrink the race window, but not close it. https://pastebin.canonical.com/p/sZ42vwT82M/

[roosterfish]

From what I understand Ready() is used to see in the local cluster is ready. It has no indication that the remote cluster can be joined successfully. Please confirm.

Yes Ready() needs to be called on the remote/leader that you want to join. It will ensure its API and DB is ready for consumption.
I don't know the automation which is in place to form your cluster, but maybe there is an option to synchronize here and hold back the joiners until the leader is ready?

If you cannot get the actual error from the reports, can you please try reproducing it? I would really like to understand the root cause first before we start thinking about a potential code change here for any of the stable LTS branches.

[hmlanigan]

@roosterfish how do we get token if the leader isn't ready?

I can try to reproduce, it is a race, however we are not seeing error messages from the 500 message. What kind of error messages are you looking for, please be specific.

I will also try to reproduce outside of sunbeam.

[roosterfish]

@roosterfish how do we get token if the leader isn't ready?

That was just an idea. I don't know if your leader gets stopped/started in between.

I can try to reproduce, it is a race, however we are not seeing error messages from the 500 message. What kind of error messages are you looking for, please be specific.

The exact JSON response from the Microcluster API would be ideal including the error message.