Skip to content

internal/rest/resources: Trust a joining member before it starts dqlite - #839

Open
jrtorsella wants to merge 1 commit into
canonical:v4from
jrtorsella:LXD-5252
Open

jrtorsella wants to merge 1 commit into
canonical:v4from
jrtorsella:LXD-5252

Conversation

@jrtorsella

Copy link
Copy Markdown
Contributor

This PR addresses an issue where microcluster called into go-dqlite's Join() logic before a cluster member was trusted by all other members. Go-dqlite attempted to reach the other cluster members, determined because they were unreachable that they were offline, and would attempt to promote the joining member on that basis, before finishing the join.

The changes in this PR are:
1) Before responding to the joiner with the join token in
clusterPost(), the leader calls propagateTrustStoreEntry() (which
is factored out of the relevant logic where this work is usually
done so that it can be called in this case). This sends the
joiner's trust store entry to all the members in the cluster on a
best-effort basis, warning if it fails or takes longer than 10
seconds and continuing.
2) The 'pending' member record in the database for the new node is
not modified and its lifecycle does not change, but it is
handled differently by the code that builds the trust store entry
lists that are sent by the leader to each member each heartbeat.
Previously, the leader ignored a pending entry, and built the
trust store list based only on non-pending entries. With this
change, pending entries also form part of the heartbeat's list of
trusted members.
3) The joiner no longer attempts to send its own entry to every
member. Previously this was a catch-22 situation and ineffective.

fixes #825 .
https://warthogs.atlassian.net/browse/LXD-5252?atlOrigin=eyJpIjoiNzM4NmU0MTc1MDM3NDcxMGIyYTE3YTA0YjllMTk0YzkiLCJwIjoiaiJ9

@jrtorsella jrtorsella changed the title internal/rest/resources: trust a joining member before it starts dqlite internal/rest/resources: Trust a joining member before it starts dqlite Oct 13, 2026
@jrtorsella
jrtorsella requested a balanced review from Copilot October 13, 2026 20:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The join ordering is corrected with focused coverage; only minor error-message wording remains.

1 open finding
What changed in this PR

Ensures joining members are trusted cluster-wide before dqlite starts.

Changes:

  • Propagates joiner trust entries from the leader.
  • Includes pending members in heartbeat trust-store updates.
  • Removes ineffective post-join trust propagation and adds tests.
File Description
internal/​rest/​resources/​truststore.go Extracts trust propagation logic.
internal/​rest/​resources/​truststore_test.go Tests propagation scenarios.
internal/​rest/​resources/​heartbeat.go Retains pending members in trust stores.
internal/​rest/​resources/​cluster.go Propagates trust before returning join data.
internal/​daemon/​daemon.go Removes late joiner-side propagation.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread internal/rest/resources/truststore.go
@jrtorsella
jrtorsella marked this pull request as ready for review October 13, 2026 23:17
This PR addresses an issue where microcluster called into go-dqlite's
Join() logic before a cluster member was trusted by all other members.
Go-dqlite attempted to reach the other cluster members, determined
because they were unreachable that they were offline, and would attempt
to promote the joining member on that basis, before finishing the join.

The changes in this PR are:
    1) Before responding to the joiner with the join token in
       clusterPost(), the leader calls propagateTrustStoreEntry() (which
       is factored out of the relevant logic where this work is usually
       done so that it can be called in this case). This sends the
       joiner's trust store entry to all the members in the cluster on a
       best-effort basis, warning if it fails or takes longer than 10
       seconds and continuing.
    2) The 'pending' member record in the database for the new node is
       not modified and its lifecycle does not change, but it is
       handled differently by the code that builds the trust store entry
       lists that are sent by the leader to each member each heartbeat.
       Previously, the leader ignored a pending entry, and built the
       trust store list based only on non-pending entries. With this
       change, pending entries also form part of the heartbeat's list of
       trusted members.
    3) The joiner no longer attempts to send its own entry to every
       member. Previously this was a catch-22 situation and ineffective.
Signed-off-by: Joseph Robert Torsella <joey.torsella@canonical.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Every new cluster joiner promotes itself because the joiner thinks only the leader is a voter

2 participants