Repository navigation
Open
Milestone
Description
What happens
After the computer changes network (for example office Wi-Fi to a phone hotspot), Community shows unreachable and never recovers. Only restarting Home brings it back.
Seen on 0.8.0-alpha.23, macOS, 19 Oct (Shanghai, on a 5G hotspot).
What the log shows
update-controller/runtime.log:
- Carrier logs
provider connect failed: connect timed outrepeatedly after the network change. - The Community link watchdog rebuilds gossip 6 times, then logs:
Carrier gossip rebuilds are capped until this Home restarts; the Community link watchdog only rejoins from now on - Rejoin alone does not recover. Community stays unreachable.
- Restarting Home:
gossip_joinsucceeds and Community works again.
Code: elastos/crates/elastos-server/src/collaboration_carrier.rs, MAX_COMMUNITY_GOSSIP_REBUILDS = 6 and the step logic around the "capped until this Home restarts" message.
Why it matters
A normal user has no idea they need to restart Home. They just see "unreachable" with nothing to do.
Suggestions
- Reset the rebuild count when the network changes (new local address or default route), or keep rebuilding with a growing back-off instead of a hard cap.
- Show a Retry action next to "unreachable" that does a full rebuild, so the user is never told to restart.
Acceptance record
- Candidate: Unpublished fix round 1 at 276fce5f5 on fix/451-community-recover-core, clean. Probe candidate unchanged.
- Exercised checks: Linux all targets passed 4070 tests with 26 ignored; fmt, Clippy and source gate passed. Mac behaviour passed 2374 tests. Required and follow-up breaks produced eight intended assertions; four focused baselines passed. Both quiet checks passed.
- Review links with reviewed SHA: Private independent development and adversarial source reviews passed at 74e6644ed, carried to 276fce5f5 by exact Linux formatting output and ancestry proof. Author checks passed.
- Installed release evidence: Released and candidate fixtures installed. All four node and Home pairings passed, with 56 unique transition texts and 80 exact arrivals. Gateway, managed Home and compiled Chat UI checks passed. Cleanup reported zero survivors. Update and Undo are unproved.
- Remaining gate: Admitted signed Update and Undo proof, designer wording and train acceptance. Repository local rehearsal refused Linux. Silent link loss without a local failure and physical target events remain open.
- Next owner: Release owner owns Update and Undo admission and train integration; designer owns wording acceptance. Later-release owner decides probe and library-hook scope.
Earlier evidence remains in the reviewed source correction, historical source gate, and installed Linux measurement. These records keep their original scope.
Done means
- Home restores its Community link by itself after node restart, message stall, long outage, sleep/wake, network change, offline return, local join failure/timeout, receiver end and a stale remembered neighbour. Test: isolated local nodes; measure the bound for each case and receive fresh exact post text in both directions after recovery.
- One record of working-link evidence drives repair, CLI status, Chat and System Community. Test: sustained local failures and ended receivers override remembered neighbours; keep a healthy quiet Community connected beyond every production timer.
- Repairs continue for unlimited uptime with bounded cost. Test: exceed 120 repairs, measure the retry rate, and prove old tasks, sockets and lookups are released while a full hub keeps its existing members.
- Reconnection status names what Home is doing and when it checks next. Test: exercise every status through the real route and both Runtime roles; inspect desktop and phone UI with retained history/draft and usable Direct controls. Recovery copy is a designer proposal.
- Failure reasons remain bounded and safe. Test: retain the reason, log state changes and at most once per minute while unchanged, and keep credentials and private operator data out of output.
- Existing network contracts keep working. Test: isolated Home starts no Community join; managed Home and gateway behave alike; run all four released-tag/candidate node/Home pairings from test: Community across versions and node restarts, and a gossip stall regression test #598 with recipient-confirmed fresh text.
- Update to the candidate and Undo to 2.0.1 preserve a working Home. Test: isolated installed fixtures; retain identity, configuration and user data; name every state file touched and bind the installed artifacts to the tested source.
- Startup states explain their prerequisites. Test: configured profile/default grant, stored Profile contexts, missing grant and failed local join; verify that signing in is separate from starting collaboration.
- Preserve historical Chat journeys
CHAT-01,CHAT-04,CHAT-05,CHAT-07andCHAT-08. Test: room selection, honest delivery, composer focus/draft, participants, reopen and retained history during recovery. UseF-27andF-28as heading/control regression input; record their current disposition without treating the historical workbook as current proof.
Metadata
Metadata
Assignees
Labels
No labels
Activity
andersalm commentedon Oct 19, 2026
Source check: alpha.23
bfefe92f6and qualified alpha.24c97c549e2have identicalcollaboration_carrier.rsand watchdog caller (blobsc107341e,8a51165c). The six-rebuild counter advances before each provider result and survives peer recovery and observation gaps.The source documents the cap as a bound on retained address-lookup tasks. A counter-reset change needs to preserve that resource bound.
OMP owns triage. Bind the reported log to the visible Home's PID and installed artifact, then measure Community peer recovery across the network switch; file-provider timeouts and a successful join response measure separate steps.
andersalm commentedon Oct 19, 2026
Triage for 0.8.0-alpha.24 (OMP): not fixed in alpha.24; listed as a known issue in the 0.8.0 notes; fix after measurement.
MAX_COMMUNITY_GOSSIP_REBUILDS(collaboration_carrier.rs:32-35) may not help. The stale part may be the Carrier endpoint or socket after the network change, not gossip.[0.8.0]/[0.8.0-alpha.24]: "After a network change, Community can stay unreachable until you restart Home (Community stays unreachable after a network change until Home restarts #451)."changelog: known issue: after a network change Community can stay unr…
andersalm commentedon Oct 19, 2026
Second report (relayed by Anders, alpha.23): the Home moved hotel → office → hotspot. Carrier logged
provider connect failed: connect timed out, then "gossip rebuilds are capped until this Home restarts". Community stayed dead on a working network until Home restarted.Correction to my triage (6261748589): I wrote that the 6 rebuilds "did not recover either". The code shows why they could not help: they are spent while the network is down. In
collaboration_carrier.rsat3a043849e:rebuildsis never reset, neither on recovery nor across gaps. It counts for the whole process lifetime, so earlier outages count against later ones.community_link_watchdog_paces_a_home_without_peers_and_caps_rebuilds). Shorter outages each spend some of the six.This explains why rebuilds are disabled. It does not yet prove that a rebuild on the new network heals, rather than an endpoint rebind or a restart. Existing recovery tests script
rebuild_heals.Plan (devil's advocate: MODIFY; alpha.24 publishes as approved, with this issue in its notes):
Scope: fix before 0.8.0 stable (#93). Not in alpha.24.
andersalm commentedon Oct 19, 2026
OMP’s 12:21 COT assignment moves the isolated #451 measurement to a fresh Linux host: two signed alpha.24 Homes, a subject-only Docker network change, short and exhausted-cap outages, peer/resource observations and a separate direct-message delivery check.
Community recovery needs evidence from its gossip path; direct messages use a separate Carrier path. Record actual rebuild attempts, authenticated peer recovery and resource lifetime before selecting the fix; container network changes give Linux evidence, with the reported Mac journey still to prove.
The requested test correction is committed at
df1565359e857913ff5bb7a78bc9a7e87784206d: it checks leave before the repair timeout and preserves the later controls. Source checks pass; its unit run remains open after remote transport failures before execution. OMP owns review carry-over and integration; the stable recovery gate remains open.andersalm commentedon Oct 19, 2026
Codex owns the preparatory watchdog test work assigned in #93. The approved source base is
3beb816438fd567145a8d725e22465bd83277cf0, tree5bc1173ec18ec947c40516f76917891f55301145; the branch isfix/451-network-recovery-fixtures, withdevelopas the eventual review target.This candidate will add bounded deterministic test controls for repeated outage, recovery, exhaustion and provider results where the current contract supports them. Production repair remains dependent on the installed measurement in 6266156985. Scripted fixture recovery measures the test control only. The installed network-switch journey still needs artifact-bound peer recovery, resource use and recipient-confirmed delivery evidence.
Source work uses an isolated worktree. Linux Rust checks use remote slot 6 from a clean committed candidate. The test work creates zero test Homes or Runtime services. Next: inspect the production caller and existing fixtures, then record the smallest useful candidate and exact checks here.
andersalm commentedon Oct 19, 2026
Prepared test-only candidate
df1565359e857913ff5bb7a78bc9a7e87784206d(treeaca731dc4497ca2877a6154c1a5121a010e4e7de) onfix/451-network-recovery-fixtures. The review correction checks leave cancellation at 126 s, before the 150 s repair timeout, with counters copied outside the lock. The later worker-alive and shutdown checks remain; production source is byte-identical.Local diff checks, pinned workspace and chain-provider formatting, product-data validation and all 99 product-data tests pass. The obsolete Home entropy script is absent from this base. The full slot 6 gate passed on predecessor
ca6515b66; the new candidate's remote check remains open after three SSH transport failures; zero checks or unit tests ran, and the fresh build host was removed. The local branch is three commits ahead and zero behind verified remote develop.Claude's review permits carry-over for this 13-line test correction; the reviewed head remains an ancestor and its other tests remain. OMP owns review confirmation and integration of this local branch. Endpoint cleanup and recipient-confirmed delivery remain in the installed measurement, which selects the production repair.
andersalm commentedon Oct 19, 2026
Measurement on installed alpha.24 (Linux, two Homes, interface swap to a new subnet): a gossip rebuild on the new network heals in about 5 s, without a restart. A rejoin does not heal. A restart heals.
Full record below. Evidence and harness:
/tmp/451-harness/(local). The fix design follows in the next comment.#451 measurement: Community and direct messages after a network change (installed 0.8.0-alpha.24, Linux)
Measured on Mon 19 Oct 2026, 17:24–19:40 UTC, on a rented Namespace Linux box (destroyed afterwards). The Mac, its real Home (8090) and the seed were not touched. Harness:
/tmp/451-harness/. Evidence copied from the box:/tmp/451-harness/out/box/.Answer
For choosing the fix: after a network change the Home must run a gossip rebuild once the new network is usable. A rejoin is not enough. A rebind or restart is not required. The fix therefore has to trigger a rebuild on a network change, and the per-process cap must not block that rebuild. The cap blocks it today because the count is never reset. A Retry action that runs a rebuild would heal too. A rebuild on the old schedule only heals when the timing happens to work: within about 20 s before reconnect, or at the next scheduled rebuild (up to 300 s later), and never once 6 rebuilds are spent.
Method
linux/amd64:8x16(8 vCPU, 16 GB), kernel 7.1.4, Docker 29.3.0. Instances78183cdljlb2, destroyed at 19:37 UTC;nsc listreturnsnull.ubuntu:24.04containers (imagesha256:68ee2200…aff33ac),subjectandreceiver, both on the docker bridgenet1(172.31.1.0/24), with--cgroupns=host.ritaand the publichttps://elastos.elacitylabs.com/install.sh.rc-tests/box-run.shdoes it.release-head.jsonreported0.8.0-alpha.24, channelcanary,release_sha256 6a692802…6f6144, signerdid:key:z6Mkgw…Barbfe.elastos --versiongiveselastos 0.8.0-alpha.24.~/.local/bin/elastosandupdate-controller/runtimehave the same sha25625ec5553a3f81c0af3aacf046d7e1ff9ff927be9128fc3c9cd7a683d9c22d81f, on both containers.3beb816438fd567145a8d725e22465bd83277cf0.docker network disconnectleaves loopback only.docker network connectto a new bridge (net2…net7 = 172.31.2–7.0/24), giving a new subnet and a new IP, like Wi-Fi to a hotspot.route.sh), on from 17:57:51 UTC:carrier.rs:1517-1543,1764-1766).update-controller/runtime.log, which is theelastos homegateway process that runs Carrier and the watchdog. Only watchdog and Carrier lines were kept (logfilter.sh), with non-test IPs and long ids redacted.transport.availablefromGET /api/apps/chat-room/summary, polled every 3 s, read-only (watch.js).elastos homeandelastos serve(sampler.sh).cap.sh; per-minute counts only, no addresses).dm-send.js/dm-recv.js): send from People → Message, wait up to 180 s for the message on the other Home, then the reverse direction. Times below are from send to first seen.elastos home, kill what is left, start again withelastos home --browser, unlock with the saved passkey.RUST_LOG(the debug filter for carrier and direct messages). The update controller clears the environment of the Home it launches (update_controller.rs:227,env_clear, then an allowlist), so the DEBUG connect and delivery lines are not available on an installed Home. Packet capture was used instead.Watchdog schedule seen (all runs)
available=falseat 17:38:59 for a cut at 17:38:22).Timeline per run (UTC; "reconnect" =
docker network connect)Run A: T = 3 min, first process (no route between bridges yet)
Community link recovered steps=2available=trueRun C: two 3-min outages, then a long one; same process (pid 17814, started 18:07:52, 0 rebuilds)
Community is unreachable(since 18:17:37) until the restart, 21 min on a working network.available=trueat 18:53:24. DMs 54.8 s / 2.2 s.Run B: long outage; the 6th rebuild runs on the new network (pid 31725, 0 rebuilds at start)
out/box/capB-summary.txt):Community link recovered steps=7at 19:18:42.The first
run.sh Bstarted at 18:53:13 was set to wait for the cap line. At 18:55:23, during the same outage, I switched it to "reconnect after 5 rebuilds" (run.sh B resume 7; logged inevents.log). That way the decisive case, a rebuild on the new network, was measured instead of a repeat of C.Run D: mechanism check, 6-min outage (pid 39774, from B's restart, 0 rebuilds)
Community link recovered steps=4Recovery summary
The healing window is between 24 s (healed) and 41 s (did not heal) from the last rebuild to reconnect.
Why, from reading the source (not run; iroh-gossip 0.101.0, iroh 1.0.2,
carrier.rsat3beb816):iroh-gossip net.rs:1014-1030). The QUIC handshake runs until the idle timeout, which defaults to 30 s (noq-proto transport.rs:555-556, not overridden iniroh quic.rs:153-163). A dial still in flight when the network returns gets through. That fits the 24–41 s cutoff [INFERENCE].Pending { queue: [Join] }in the actor's peer map. The dial failure does not remove it (net.rs:435-444,hyparview.rs:346-355). OnlyDisconnectPeerremoves a peer entry (net.rs:732-736).net.rs:695-701), and a leave does not clear the actor-wide peer map (hyparview.rs:357-362). That is why rejoins sent zero packets.carrier.rs:1286-1313). ElastOS already names this case incarrier.rs:1280-1283("a peer left pending after a failed dial").warn "dial failed"underiroh_gossip. The installed log filter hides it (noRUST_LOG, which the update controller clears).Resources per rebuild (
elastos home, fromsamples-subject.tsv)elastos servewas unaffected (41.7–48.3 MB, 10–12 threads).elastos homewent from 66.5 to 116.9 MB in 2 h without any rebuild, so normal traffic outweighs anything a rebuild adds.collaboration_carrier.rs:32-35(an address lookup and its eviction task) could not be counted from outside.Other observations
transport.availablewent false about 37 s after the cut and true at recovery. While stuck it said "Community is unreachable. Retained messages stay available."route.shfor that reason.Limits
Harness (
/tmp/451-harness/)box-setup.sh: containers, Playwright, install.u.sh: run asrita.start.sh: start Home and the driver.driver.mjs: Playwright driver, fromrc-tests/runs/a19-linuxpair.signup.js,unlock.js,disc.js,disc-request.js,accept.js,people.js: sign-up and contact setup.watch.js: Community state.dm-send.js,dm-recv.js,dm-list.js,dm-pair.sh: DMs.sampler.sh: resources and watchdog counts.logfilter.sh: bounded, redacted log copy.route.sh: route between bridges.cap.sh: packet counts.run.sh A|B|C|D: the runs.andersalm commentedon Oct 19, 2026
Fix design for 0.8.0 (OMP; devil's advocate MODIFY applied: keep a lifetime resource bound; no new Retry repair path this week).
iroh-gossip-0.101.0/src/net/address_lookup.rs:39-89): each rebuild leaves oneGossipAddressLookupon the endpoint. That is a map whose entries expire after 5 min, so it empties, plus one task that wakes every 30 s for as long as the endpoint keeps the lookup. It is small but permanent, so the lifetime bound stays.andersalm commentedon Oct 19, 2026
Claude delta review PASS covers
d795ae534..850729c29b10bacd73dd4d055d192343be596fc2/ treeb5c60c5e1a908008fdd8ac1ca9f18c0c3fff484c: restart visibility, cap documentation, address filtering, post-cap pacing and log wording are resolved. Independent source/artifact hashes match the committed stamp. This is source review; installed recovery remains open.Both completed
850729c29remote checks end with a missing exit status after passing subsets, with no full gate verdict. The reviewer ran the artifact validator and 10 Node tests; Cargo and compiler-byte reproduction remain separate checks.Seven normal source picks into the train preserve their changes and history. The combined Chat artifact commit is
769544347; later train headfdb762f9886e746b4a72fcf6676c1860fc1a75a5/ treebedbb31e2edc38668fe4162c2b203c58a4bf6045adds the copy consistency test and notes. Current-head checks and installed Linux/Mac Community, DM and resource acceptance remain open.andersalm commentedon Oct 19, 2026
The 23 focused Rust checks pass for the candidate in the Acceptance record, and controls with the old recovery settings fail as expected. The configured Linux gate is running; installed tests and Claude review remain open.
andersalm commentedon Oct 19, 2026
Clippy stopped on the single-range fixture. The fixture now uses an explicit one-range iterator, and its data and format checks pass; the current Acceptance record keeps the prior Rust results separate. The corrected candidate still needs its Rust gate, Claude review and installed acceptance.
andersalm commentedon Oct 19, 2026
Claude review of the production fix at
d795ae534(3beb81643..d795ae534): FAIL, with 2 blocking items outside the watchdog. Report:/tmp/claude-review-net451d795.md.TROUBLESHOOTING.mdstill describe the old six-rebuild limit.The fix round is with the Codex #451 builder.
andersalm commentedon Oct 19, 2026
Prepared
9c58b4178c4134bbecda9e5883b93c6005e00a43(treefb542e2df217ac0836e5c19e8eb8bbabda9ec834) with the F1, F2, F3, F5 and log corrections; the Acceptance record lists the exact checks and retained prior proof.The 15 Chat and artifact-validator fixtures, 99 product-data tests, data validation, format and whitespace checks pass. The shared build lease was busy; Chat WASM, bindings and stamp still need an admitted build, and the source/artifact check exits 1.
OMP owns that build, the new Rust tests and full gate, Claude review, and installed Linux and Mac acceptance. The Mac run will record which trigger starts recovery; #451 stays open.
andersalm commentedon Oct 21, 2026
Status (20 Oct): the fix (
850729c29, ported to the train as769544347) is on the alpha.25 train (release/train-0.8.0-rc10, current headf674ff637). Remaining: the Linux harness rerun and the Mac Wi-Fi switch check on signed alpha.25. Stays open until then.andersalm commentedon Nov 11, 2026
Reopened: the code shipped in 2.0.0, but the installed check in Done means was never recorded, and AGENTS.md closes a task only when Done means is proven. The check is owed in 2.0.2.