Repository navigation
test: Community across versions and node restarts, and a gossip stall regression test - #598
Merged
Merged
Conversation
…he actor iroh-gossip 0.101.0 dispatches OutEvent::SendMessage with an awaiting send on a bounded per-peer channel (SEND_QUEUE_CAP = 64, net.rs:686), so one neighbour whose stream credit runs out parks the single actor loop and the node serves no topic, including new joins, while QUIC stays healthy. 0.102.0 replaces that channel with a byte-bounded queue that drops or aborts instead of awaiting. The test drives the real net actor over QUIC: a neighbour completes the membership handshake through the public protocol state, then holds its streams open and reads nothing. It asserts the two symptoms a Community node shows, that traffic to the other neighbours keeps flowing and that a fresh join still completes.
Homes and the Community node update at different times, so the release has to hold for every pairing. The journey starts one Community node and two Homes from two named builds, joins them to one Community, then restarts the node and brings up a fresh Home against it, over the four arrangements. It stages only the Runtime and the localhost provider against the manifest checksum the Runtime verifies, so it runs on a machine that has no media tools for a full source Home.
One arrangement that fails now records its reason and the run continues to the others, and the settle window is a flag so a shake-out run is quick.
The shared process census owns a Runtime by its data-root path, so a Home started from a build target escaped cleanup. Each Home now runs its own copy, a stuck cleanup still returns a result, and a failed arrangement keeps a log tail before its fixture tree goes away.
The release gate asks what a Community node does while one joined Home holds its connection open and takes no more frames. The wedge arrangement stops that Home, asks a fresh Home to join, and reports the Community link lines it saw.
scripts/setup-source-home.sh refuses to install a source Home without ffmpeg and ffprobe, so a rented box could not run the installed journeys. The box provisioner reads this list from the checked commit and installs a package only when its program is absent. Co-authored-by: Cursor <cursoragent@cursor.com>
The wedge test could pass for the wrong reason. It took one second without
progress as a settled flood, at any count. It dropped broadcast errors and
task panics, so a silent neighbour that let go of its streams freed a wedged
iroh-gossip 0.101.0 hub and the test passed. The wait for the flood and the
endpoint shutdown had no time limit.
The flood now ends in one of three ways: it completes, it stalls, or the
fixture fails with its own error. A stall needs two consecutive samples
without progress above a pressure floor of 392 broadcasts. The floor is the
64-message send queue of 0.101.0 plus the 1,250,000-byte default stream
receive window in 3800-byte payloads. A flood that stays at or below the
floor fails with "the fixture did not build enough pressure". A broadcast
error, a panic in the flood, and an error or a panic in the silent neighbour
fail the test with that cause. One deadline, the sum of the phase budgets
plus a margin for bind and shutdown, covers every step.
On iroh-gossip 0.102.0 the test passes in 1 s to 4 s. On 0.101.0 it fails as
before: the hub stalls after 455 of 500 broadcasts, delivery stops, and a
fresh join stops.
cargo test --manifest-path elastos/Cargo.toml -p elastos-server \
--test gossip_actor_wedge -- --nocapture
Co-authored-by: Cursor <cursoragent@cursor.com>
A mixed-version run needs the Community node and the Homes to come from different installed builds. InstalledBuild binds an installed source Home to the checkout its installation receipt names, and Owner replaces the positional tuple for one running Home. The journey takes a Home build and an optional node build. The receipt records the node identity, and acceptance binds the node and each Home to the Runtime of its own build. One build still serves both by default, so the installed CI step is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
The journey passed when no dead-link warning appeared, and a remembered neighbour count hides a dead link from that warning. The journey now collects positive evidence instead. A table names four transitions: the first start, a Community node restart under running Homes, a fresh Home that joins the restarted node, and the restart of a stopped Home. After each one, every Home is a live process that reports a Carrier peer through its own status command. Then every member Home posts one fresh Community message, every other member reads that exact text, and one direct message arrives with its exact text. Acceptance refuses a receipt that lacks a transition, a Home, a live process, a peer count of 1 or more, an ordered Community pair or the direct message, and a text that an earlier transition already used. Co-authored-by: Cursor <cursoragent@cursor.com>
Ctrl-C reached the journey's clean-up through KeyboardInterrupt, but SIGTERM ended the process and left its Homes and fixture behind. The journey now installs a SIGTERM handler that raises, so the same clean-up stops every owned process, removes the fixture and writes the failure receipt. The handler stays in place during the clean-up and the journey restores the previous handler when it returns. Co-authored-by: Cursor <cursoragent@cursor.com>
Acceptance compares the set of ordered Community pairs with the set the transition requires, and it checks the exact text of every recorded row. A repeated row therefore either leaves a pair out or passes the same checks twice, so the separate row count refused nothing on its own. Co-authored-by: Cursor <cursoragent@cursor.com>
…rney The driver staged its own Homes, read Runtime log lines and passed when no dead-link line appeared. An independent review refused that as proof. The driver now takes a released and a candidate installed source Home, and for each of the four node and Home pairings it runs the installed two-Home journey, which proves every transition with live processes, peer counts and exact message texts. One receipt names each arrangement, both build identities and the whole journey receipt. An arrangement passes only when the journey returns and its receipt passes acceptance for those two builds. A failed clean-up, a deadline, a leftover fixture or a missing arrangement fails the run, and SIGTERM stops it with a failed receipt. The wedge mode is removed. SIGSTOP also stops the QUIC acknowledgements of the stopped Runtime, so its connection times out and the mode did not hold the condition it named. The gossip actor wedge test covers a peer that keeps healthy transport and withholds stream credit. Co-authored-by: Cursor <cursoragent@cursor.com>
Homes and the Community node update at different times, so a release must hold for every pairing. A Linux job installs the head and the released tag into two lean source Homes and runs the four arrangements against them. The tag is a workflow input with the current release as its default. Co-authored-by: Cursor <cursoragent@cursor.com>
…hholds gather_runtime_status returns early for a gateway Runtime and leaves runtime.peer_count unset, so every Home in this journey reported no count. The gateway control surface exposes one peer route on purpose, so each Home now proves its own Carrier Provider answers get_ticket through its own attach secret. The fresh Community and direct traffic after each transition carries the proof that the link works. Co-authored-by: Cursor <cursoragent@cursor.com>
A separate job needs its own source Home, and the release policy keeps one source Home per job. Three gated steps in source-home-linux build the released tag beside the candidate over the same build directories and run the four arrangements, so the second build reuses what the first one compiled. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Merged on Anders's explicit authorization in chat (15 Nov 2026); required checks green on 3438813. |
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deepen notes
gossip_joinline before it subscribes (carrier.rs:1322), and a gateway Runtime reports no peer count (home_cmd.rs:2479-2483). So neither a log line nor a peer count is accepted as proof.What this adds (tests and CI only, no production code)
elastos/crates/elastos-server/tests/gossip_actor_wedge.rs: a regression test for upstream iroh-gossip issue 47 (fixed in 0.102.0, which develop pins since carrier: remove distributed-topic-tracker, iroh-gossip 0.102 (#570) #587). A neighbour that stays connected and reads nothing must not stop delivery to others or a fresh join. One outer deadline, a pressure floor so the test cannot pass before the load exists, and flood errors or task panics fail it.scripts/gossip-mixed-version-journey.pyruns the installed two-Home journey with the Community node and the member Homes each on the released tag or the candidate, all four pairings. After each of four transitions (start, node restart, a fresh Home joining, a member Home restarting) every Home must be alive, its own Carrier Provider must answer, and Community posts created after the transition must arrive with their exact text in every direction, plus one direct message. No text may repeat across transitions.source-home-linux(check out the released tag, install it over the same build directories, run the four pairings) on merge groups,developand release pushes and manual dispatch. Pull-request pushes run the same-version journey only. Added time about 17 to 19 minutes; the job timeout goes from 120 to 180 minutes. The tag is themixed_version_taginput, defaultv2.0.1.scripts/ci-tools.txt: names the test tools a rented Linux box installs, so the remote check can install a source Home.Evidence at
3438813a5v2.0.1and candidate, installed path, Linux x86-64Review
Written with Cursor (Claude Opus). Reviewed by GPT (
codex-review.sh): PASS, nothing blocking, at head3438813a5db25a86ef800e3e4b6cac241a10362d, tree941434f3e806b0c6146ab50206ce7390149c5448. The first round's review was FAIL (the journey could pass without a working link); both of its findings are fixed here.Not tested
v2.0.1.Follow-ups from the review (not blocking)
Refs #410, #570.