Skip to content

test: Community across versions and node restarts, and a gossip stall regression test - #598

Merged
andersalm merged 15 commits into
developfrom
test/gossip-mixed-versions
Nov 15, 2026
Merged

andersalm merged 15 commits into
developfrom
test/gossip-mixed-versions

Conversation

@andersalm

Copy link
Copy Markdown
Contributor

Deepen notes

  1. Expert view. One installed journey serves same-version and mixed-version runs: the two-Home journey gains a node build choice, and the mixed-version script is an 81-line driver over it. Each check was first shown to observe what it claims before a matrix was built on it.
  2. Raw material. The failed review of the first round and the code it anchors: the Runtime prints its gossip_join line before it subscribes (carrier.rs:1322), and a gateway Runtime reports no peer count (home_cmd.rs:2479-2483). So neither a log line nor a peer count is accepted as proof.
  3. Methods. Measurement on running Runtimes, and a negative control: the regression test built against the library version that has the defect.
  4. Independent check. The negative control fails as it must, the four arrangements pass on the installed path with a real browser, the remote full check passes, and a review by a different model family passes.

What this adds (tests and CI only, no production code)

  • elastos/crates/elastos-server/tests/gossip_actor_wedge.rs: a regression test for upstream iroh-gossip issue 47 (fixed in 0.102.0, which develop pins since carrier: remove distributed-topic-tracker, iroh-gossip 0.102 (#570) #587). A neighbour that stays connected and reads nothing must not stop delivery to others or a fresh join. One outer deadline, a pressure floor so the test cannot pass before the load exists, and flood errors or task panics fail it.
  • Mixed-version gate: scripts/gossip-mixed-version-journey.py runs the installed two-Home journey with the Community node and the member Homes each on the released tag or the candidate, all four pairings. After each of four transitions (start, node restart, a fresh Home joining, a member Home restarting) every Home must be alive, its own Carrier Provider must answer, and Community posts created after the transition must arrive with their exact text in every direction, plus one direct message. No text may repeat across transitions.
  • CI: three steps in source-home-linux (check out the released tag, install it over the same build directories, run the four pairings) on merge groups, develop and release pushes and manual dispatch. Pull-request pushes run the same-version journey only. Added time about 17 to 19 minutes; the job timeout goes from 120 to 180 minutes. The tag is the mixed_version_tag input, default v2.0.1.
  • scripts/ci-tools.txt: names the test tools a rented Linux box installs, so the remote check can install a source Home.

Evidence at 3438813a5

Check Result
Regression test with iroh-gossip 0.102.0 (the pin) ok: 500 of 500 broadcasts accepted, delivery and fresh join immediate
Same test with iroh-gossip 0.101.0 (negative control) FAILED: stalled after 455 of 500, delivery 15.0 s, fresh join 30.0 s
Four pairings, released v2.0.1 and candidate, installed path, Linux x86-64 all passed; 56 fresh texts, 56 distinct
Remote check and remote full check PASS (full: gate 25m13s, test-elastos 8m21s, test-capsules 3m27s)

Review

Written with Cursor (Claude Opus). Reviewed by GPT (codex-review.sh): PASS, nothing blocking, at head 3438813a5db25a86ef800e3e4b6cac241a10362d, tree 941434f3e806b0c6146ab50206ce7390149c5448. The first round's review was FAIL (the journey could pass without a working link); both of its findings are fixed here.

Not tested

  • macOS and Linux ARM64 mixed-version runs; a Home older than v2.0.1.
  • A real network between the Homes (relay, NAT, packet loss); load from many senders.
  • The stall at Runtime level on an installed Community; the library-level test holds that condition.

Follow-ups from the review (not blocking)

  • Raise the flood from 500 to 700 messages so the test also exercises the 0.102.0 drop path (about 21 s more).
  • The default baseline tag must be advanced with each release, together with the update-hop pin.
  • The mixed-version steps do not run on pull-request pushes; a manual dispatch on this branch shows them before merge.

Refs #410, #570.

andersalm and others added 15 commits November 14, 2026 12:46
…he actor

iroh-gossip 0.101.0 dispatches OutEvent::SendMessage with an awaiting send on
a bounded per-peer channel (SEND_QUEUE_CAP = 64, net.rs:686), so one neighbour
whose stream credit runs out parks the single actor loop and the node serves
no topic, including new joins, while QUIC stays healthy. 0.102.0 replaces that
channel with a byte-bounded queue that drops or aborts instead of awaiting.

The test drives the real net actor over QUIC: a neighbour completes the
membership handshake through the public protocol state, then holds its streams
open and reads nothing. It asserts the two symptoms a Community node shows,
that traffic to the other neighbours keeps flowing and that a fresh join still
completes.
Homes and the Community node update at different times, so the release has to
hold for every pairing. The journey starts one Community node and two Homes
from two named builds, joins them to one Community, then restarts the node and
brings up a fresh Home against it, over the four arrangements.

It stages only the Runtime and the localhost provider against the manifest
checksum the Runtime verifies, so it runs on a machine that has no media tools
for a full source Home.
One arrangement that fails now records its reason and the run continues to the
others, and the settle window is a flag so a shake-out run is quick.
The shared process census owns a Runtime by its data-root path, so a Home
started from a build target escaped cleanup. Each Home now runs its own copy,
a stuck cleanup still returns a result, and a failed arrangement keeps a log
tail before its fixture tree goes away.
The release gate asks what a Community node does while one joined Home holds
its connection open and takes no more frames. The wedge arrangement stops that
Home, asks a fresh Home to join, and reports the Community link lines it saw.
scripts/setup-source-home.sh refuses to install a source Home without ffmpeg
and ffprobe, so a rented box could not run the installed journeys. The box
provisioner reads this list from the checked commit and installs a package
only when its program is absent.

Co-authored-by: Cursor <cursoragent@cursor.com>
The wedge test could pass for the wrong reason. It took one second without
progress as a settled flood, at any count. It dropped broadcast errors and
task panics, so a silent neighbour that let go of its streams freed a wedged
iroh-gossip 0.101.0 hub and the test passed. The wait for the flood and the
endpoint shutdown had no time limit.

The flood now ends in one of three ways: it completes, it stalls, or the
fixture fails with its own error. A stall needs two consecutive samples
without progress above a pressure floor of 392 broadcasts. The floor is the
64-message send queue of 0.101.0 plus the 1,250,000-byte default stream
receive window in 3800-byte payloads. A flood that stays at or below the
floor fails with "the fixture did not build enough pressure". A broadcast
error, a panic in the flood, and an error or a panic in the silent neighbour
fail the test with that cause. One deadline, the sum of the phase budgets
plus a margin for bind and shutdown, covers every step.

On iroh-gossip 0.102.0 the test passes in 1 s to 4 s. On 0.101.0 it fails as
before: the hub stalls after 455 of 500 broadcasts, delivery stops, and a
fresh join stops.

  cargo test --manifest-path elastos/Cargo.toml -p elastos-server \
    --test gossip_actor_wedge -- --nocapture

Co-authored-by: Cursor <cursoragent@cursor.com>
A mixed-version run needs the Community node and the Homes to come from
different installed builds. InstalledBuild binds an installed source Home
to the checkout its installation receipt names, and Owner replaces the
positional tuple for one running Home.

The journey takes a Home build and an optional node build. The receipt
records the node identity, and acceptance binds the node and each Home to
the Runtime of its own build. One build still serves both by default, so
the installed CI step is unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>
The journey passed when no dead-link warning appeared, and a remembered
neighbour count hides a dead link from that warning. The journey now
collects positive evidence instead.

A table names four transitions: the first start, a Community node restart
under running Homes, a fresh Home that joins the restarted node, and the
restart of a stopped Home. After each one, every Home is a live process
that reports a Carrier peer through its own status command. Then every
member Home posts one fresh Community message, every other member reads
that exact text, and one direct message arrives with its exact text.

Acceptance refuses a receipt that lacks a transition, a Home, a live
process, a peer count of 1 or more, an ordered Community pair or the
direct message, and a text that an earlier transition already used.

Co-authored-by: Cursor <cursoragent@cursor.com>
Ctrl-C reached the journey's clean-up through KeyboardInterrupt, but
SIGTERM ended the process and left its Homes and fixture behind. The
journey now installs a SIGTERM handler that raises, so the same clean-up
stops every owned process, removes the fixture and writes the failure
receipt. The handler stays in place during the clean-up and the journey
restores the previous handler when it returns.

Co-authored-by: Cursor <cursoragent@cursor.com>
Acceptance compares the set of ordered Community pairs with the set the
transition requires, and it checks the exact text of every recorded row.
A repeated row therefore either leaves a pair out or passes the same
checks twice, so the separate row count refused nothing on its own.

Co-authored-by: Cursor <cursoragent@cursor.com>
…rney

The driver staged its own Homes, read Runtime log lines and passed when
no dead-link line appeared. An independent review refused that as proof.
The driver now takes a released and a candidate installed source Home,
and for each of the four node and Home pairings it runs the installed
two-Home journey, which proves every transition with live processes,
peer counts and exact message texts.

One receipt names each arrangement, both build identities and the whole
journey receipt. An arrangement passes only when the journey returns and
its receipt passes acceptance for those two builds. A failed clean-up, a
deadline, a leftover fixture or a missing arrangement fails the run, and
SIGTERM stops it with a failed receipt.

The wedge mode is removed. SIGSTOP also stops the QUIC acknowledgements
of the stopped Runtime, so its connection times out and the mode did not
hold the condition it named. The gossip actor wedge test covers a peer
that keeps healthy transport and withholds stream credit.

Co-authored-by: Cursor <cursoragent@cursor.com>
Homes and the Community node update at different times, so a release must
hold for every pairing. A Linux job installs the head and the released tag
into two lean source Homes and runs the four arrangements against them.
The tag is a workflow input with the current release as its default.

Co-authored-by: Cursor <cursoragent@cursor.com>
…hholds

gather_runtime_status returns early for a gateway Runtime and leaves
runtime.peer_count unset, so every Home in this journey reported no count.
The gateway control surface exposes one peer route on purpose, so each Home
now proves its own Carrier Provider answers get_ticket through its own attach
secret. The fresh Community and direct traffic after each transition carries
the proof that the link works.

Co-authored-by: Cursor <cursoragent@cursor.com>
A separate job needs its own source Home, and the release policy keeps one
source Home per job. Three gated steps in source-home-linux build the released
tag beside the candidate over the same build directories and run the four
arrangements, so the second build reuses what the first one compiled.

Co-authored-by: Cursor <cursoragent@cursor.com>
@andersalm
andersalm merged commit d1aedca into develop Nov 15, 2026
34 checks passed
@andersalm

Copy link
Copy Markdown
Contributor Author

Merged on Anders's explicit authorization in chat (15 Nov 2026); required checks green on 3438813.
Review: GPT PASS on 3438813, tree 941434f3e (round 2; the round 1 findings are fixed); pull request files unchanged since.
The mixed-version steps also passed on this head by manual dispatch (run 43672302149).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant