Skip to content

Community stays unreachable after a network change until Home restarts #451

Open

Description

@SashaMIT

What happens

After the computer changes network (for example office Wi-Fi to a phone hotspot), Community shows unreachable and never recovers. Only restarting Home brings it back.

Seen on 0.8.0-alpha.23, macOS, 19 Oct (Shanghai, on a 5G hotspot).

What the log shows

update-controller/runtime.log:

  1. Carrier logs provider connect failed: connect timed out repeatedly after the network change.
  2. The Community link watchdog rebuilds gossip 6 times, then logs:
    Carrier gossip rebuilds are capped until this Home restarts; the Community link watchdog only rejoins from now on
  3. Rejoin alone does not recover. Community stays unreachable.
  4. Restarting Home: gossip_join succeeds and Community works again.

Code: elastos/crates/elastos-server/src/collaboration_carrier.rs, MAX_COMMUNITY_GOSSIP_REBUILDS = 6 and the step logic around the "capped until this Home restarts" message.

Why it matters

A normal user has no idea they need to restart Home. They just see "unreachable" with nothing to do.

Suggestions

  • Reset the rebuild count when the network changes (new local address or default route), or keep rebuilding with a growing back-off instead of a hard cap.
  • Show a Retry action next to "unreachable" that does a full rebuild, so the user is never told to restart.

Acceptance record

  • Candidate: Unpublished fix round 1 at 276fce5f5 on fix/451-community-recover-core, clean. Probe candidate unchanged.
  • Exercised checks: Linux all targets passed 4070 tests with 26 ignored; fmt, Clippy and source gate passed. Mac behaviour passed 2374 tests. Required and follow-up breaks produced eight intended assertions; four focused baselines passed. Both quiet checks passed.
  • Review links with reviewed SHA: Private independent development and adversarial source reviews passed at 74e6644ed, carried to 276fce5f5 by exact Linux formatting output and ancestry proof. Author checks passed.
  • Installed release evidence: Released and candidate fixtures installed. All four node and Home pairings passed, with 56 unique transition texts and 80 exact arrivals. Gateway, managed Home and compiled Chat UI checks passed. Cleanup reported zero survivors. Update and Undo are unproved.
  • Remaining gate: Admitted signed Update and Undo proof, designer wording and train acceptance. Repository local rehearsal refused Linux. Silent link loss without a local failure and physical target events remain open.
  • Next owner: Release owner owns Update and Undo admission and train integration; designer owns wording acceptance. Later-release owner decides probe and library-hook scope.

Earlier evidence remains in the reviewed source correction, historical source gate, and installed Linux measurement. These records keep their original scope.

Done means

  • Home restores its Community link by itself after node restart, message stall, long outage, sleep/wake, network change, offline return, local join failure/timeout, receiver end and a stale remembered neighbour. Test: isolated local nodes; measure the bound for each case and receive fresh exact post text in both directions after recovery.
  • One record of working-link evidence drives repair, CLI status, Chat and System Community. Test: sustained local failures and ended receivers override remembered neighbours; keep a healthy quiet Community connected beyond every production timer.
  • Repairs continue for unlimited uptime with bounded cost. Test: exceed 120 repairs, measure the retry rate, and prove old tasks, sockets and lookups are released while a full hub keeps its existing members.
  • Reconnection status names what Home is doing and when it checks next. Test: exercise every status through the real route and both Runtime roles; inspect desktop and phone UI with retained history/draft and usable Direct controls. Recovery copy is a designer proposal.
  • Failure reasons remain bounded and safe. Test: retain the reason, log state changes and at most once per minute while unchanged, and keep credentials and private operator data out of output.
  • Existing network contracts keep working. Test: isolated Home starts no Community join; managed Home and gateway behave alike; run all four released-tag/candidate node/Home pairings from test: Community across versions and node restarts, and a gossip stall regression test #598 with recipient-confirmed fresh text.
  • Update to the candidate and Undo to 2.0.1 preserve a working Home. Test: isolated installed fixtures; retain identity, configuration and user data; name every state file touched and bind the installed artifacts to the tested source.
  • Startup states explain their prerequisites. Test: configured profile/default grant, stored Profile contexts, missing grant and failed local join; verify that signing in is separate from starting collaboration.
  • Preserve historical Chat journeys CHAT-01, CHAT-04, CHAT-05, CHAT-07 and CHAT-08. Test: room selection, honest delivery, composer focus/draft, participants, reopen and retained history during recovery. Use F-27 and F-28 as heading/control regression input; record their current disposition without treating the historical workbook as current proof.

Activity

  1. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Source check: alpha.23 bfefe92f6 and qualified alpha.24 c97c549e2 have identical collaboration_carrier.rs and watchdog caller (blobs c107341e, 8a51165c). The six-rebuild counter advances before each provider result and survives peer recovery and observation gaps.
    The source documents the cap as a bound on retained address-lookup tasks. A counter-reset change needs to preserve that resource bound.
    OMP owns triage. Bind the reported log to the visible Home's PID and installed artifact, then measure Community peer recovery across the network switch; file-provider timeouts and a successful join response measure separate steps.

  2. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Triage for 0.8.0-alpha.24 (OMP): not fixed in alpha.24; listed as a known issue in the 0.8.0 notes; fix after measurement.

    • Why not a quick fix: the report shows the 6 gossip rebuilds did not recover either. Only a process restart did. So raising or removing MAX_COMMUNITY_GOSSIP_REBUILDS (collaboration_carrier.rs:32-35) may not help. The stale part may be the Carrier endpoint or socket after the network change, not gossip.
    • The cap is deliberate: each rebuild leaks an address lookup and its eviction task on the iroh 1.0.2 endpoint. A rebuild loop without a bound is a resource leak.
    • Next, to measure: after a network change, does a rebuild recover once the new network is usable, or does only an endpoint rebind or process restart? Capture provider connect results per rebuild, with the endpoint's local addresses before and after.
    • Candidate fixes, depending on the result:
      • rebind the Carrier endpoint when the local address set changes;
      • or reset the rebuild count on a network change;
      • and a Retry action next to "unreachable" that does the same repair.
    • Notes line in [0.8.0] / [0.8.0-alpha.24]: "After a network change, Community can stay unreachable until you restart Home (Community stays unreachable after a network change until Home restarts #451)."
    • Scope: Anders decides before 0.8.0 stable whether the fix is required (0.8.0: Release 0.8.0 #93).
  3. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Second report (relayed by Anders, alpha.23): the Home moved hotel → office → hotspot. Carrier logged provider connect failed: connect timed out, then "gossip rebuilds are capped until this Home restarts". Community stayed dead on a working network until Home restarted.

    Correction to my triage (6261748589): I wrote that the 6 rebuilds "did not recover either". The code shows why they could not help: they are spent while the network is down. In collaboration_carrier.rs at 3a043849e:

    • The watchdog steps after 60 s without peers. The first step rejoins; each later step rebuilds; the waits double up to 300 s.
    • rebuilds is never reset, neither on recovery nor across gaps. It counts for the whole process lifetime, so earlier outages count against later ones.
    • The sixth rebuild comes at 1,380 s of continuous silence (test community_link_watchdog_paces_a_home_without_peers_and_caps_rebuilds). Shorter outages each spend some of the six.
    • After the cap, only rejoins run, so a working network does not recover.

    This explains why rebuilds are disabled. It does not yet prove that a rebuild on the new network heals, rather than an endpoint rebind or a restart. Existing recovery tests script rebuild_heals.

    Plan (devil's advocate: MODIFY; alpha.24 publishes as approved, with this issue in its notes):

    1. Measure now, on an installed Home:
      • a short outage (1–2 rebuilds spent), then the network back: does the next rebuild heal?
      • an exhausted outage: compare rejoin, a forced rebuild, and a restart;
      • resource growth per rebuild;
      • record real message delivery, not only the Live label.
    2. Fix after the measurement, in a focused alpha.25 before 0.8.0 stable. A refilling budget alone is rejected: it bounds the leak rate, not the leaked tasks (6 + 48/day at one per 30 min). The candidate fix is whichever repair the measurement shows heals with bounded resources, for example rebinding the Carrier endpoint, which drops the old endpoint's leaked lookups, with any counter reset on a successful network change.
    3. UI: the Community status must say something true and actionable when the link is down, for example "Community can't connect. Restart Home to reconnect", until a Retry exists. A Retry action is added only if it runs the repair that the measurement proves.

    Scope: fix before 0.8.0 stable (#93). Not in alpha.24.

  4. added this to the 0.8.0 milestone on Oct 19, 2026
  5. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    OMP’s 12:21 COT assignment moves the isolated #451 measurement to a fresh Linux host: two signed alpha.24 Homes, a subject-only Docker network change, short and exhausted-cap outages, peer/resource observations and a separate direct-message delivery check.
    Community recovery needs evidence from its gossip path; direct messages use a separate Carrier path. Record actual rebuild attempts, authenticated peer recovery and resource lifetime before selecting the fix; container network changes give Linux evidence, with the reported Mac journey still to prove.
    The requested test correction is committed at df1565359e857913ff5bb7a78bc9a7e87784206d: it checks leave before the repair timeout and preserves the later controls. Source checks pass; its unit run remains open after remote transport failures before execution. OMP owns review carry-over and integration; the stable recovery gate remains open.

  6. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Codex owns the preparatory watchdog test work assigned in #93. The approved source base is 3beb816438fd567145a8d725e22465bd83277cf0, tree 5bc1173ec18ec947c40516f76917891f55301145; the branch is fix/451-network-recovery-fixtures, with develop as the eventual review target.

    This candidate will add bounded deterministic test controls for repeated outage, recovery, exhaustion and provider results where the current contract supports them. Production repair remains dependent on the installed measurement in 6266156985. Scripted fixture recovery measures the test control only. The installed network-switch journey still needs artifact-bound peer recovery, resource use and recipient-confirmed delivery evidence.

    Source work uses an isolated worktree. Linux Rust checks use remote slot 6 from a clean committed candidate. The test work creates zero test Homes or Runtime services. Next: inspect the production caller and existing fixtures, then record the smallest useful candidate and exact checks here.

  7. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Prepared test-only candidate df1565359e857913ff5bb7a78bc9a7e87784206d (tree aca731dc4497ca2877a6154c1a5121a010e4e7de) on fix/451-network-recovery-fixtures. The review correction checks leave cancellation at 126 s, before the 150 s repair timeout, with counters copied outside the lock. The later worker-alive and shutdown checks remain; production source is byte-identical.
    Local diff checks, pinned workspace and chain-provider formatting, product-data validation and all 99 product-data tests pass. The obsolete Home entropy script is absent from this base. The full slot 6 gate passed on predecessor ca6515b66; the new candidate's remote check remains open after three SSH transport failures; zero checks or unit tests ran, and the fresh build host was removed. The local branch is three commits ahead and zero behind verified remote develop.
    Claude's review permits carry-over for this 13-line test correction; the reviewed head remains an ancestor and its other tests remain. OMP owns review confirmation and integration of this local branch. Endpoint cleanup and recipient-confirmed delivery remain in the installed measurement, which selects the production repair.

  8. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Measurement on installed alpha.24 (Linux, two Homes, interface swap to a new subnet): a gossip rebuild on the new network heals in about 5 s, without a restart. A rejoin does not heal. A restart heals.

    • Counts carry over: with the process-lifetime cap spent, Community stays unreachable on a working network until restart. That reproduces the field symptom.
    • Silent window: between reconnect and the next scheduled rebuild (up to 5 min), the Home sends nothing.
    • Leak: no growth per rebuild was measurable (RSS +0.18 MB over 5 rebuilds; threads and fds flat).

    Full record below. Evidence and harness: /tmp/451-harness/ (local). The fix design follows in the next comment.


    #451 measurement: Community and direct messages after a network change (installed 0.8.0-alpha.24, Linux)

    Measured on Mon 19 Oct 2026, 17:24–19:40 UTC, on a rented Namespace Linux box (destroyed afterwards). The Mac, its real Home (8090) and the seed were not touched. Harness: /tmp/451-harness/. Evidence copied from the box: /tmp/451-harness/out/box/.

    Answer

    • A gossip rebuild on the new network heals, without a restart. It restores both Community and direct messages. In run B the 6th rebuild ran 4 min 23 s after the network came back, and Community recovered 5 s later. In run D the 3rd rebuild ran 2 min 47 s after reconnect, and Community recovered 5 s later. Direct messages (DMs) were delivered both ways 6–93 s after the rebuild. No endpoint rebind was needed.
    • A rejoin does not heal. In run C, four rejoins on a working network over 15 min sent no packets, and Community stayed down.
    • A restart also heals, in about 0–5 s. This is the only repair left once the cap is spent.
    • Between reconnect and the next rebuild, the Home is silent. Its Carrier sends no UDP packets at all on the new network, not to the Community seed and not to the other Home. Over the same minutes the receiver kept sending about 250–300 packets/min to the seed.
    • The Community stays unreachable after a network change until Home restarts #451 field symptom is reproduced (run C). Two short outages spent 2 rebuilds. A long outage spent the other 4. The Home then reconnected with no rebuild left: 21 min unreachable on a working network, until a restart.
    • Short outages seem to heal on their own only because of timing. In those runs a rebuild had happened 15–24 s before the network came back. With 41–76 s between the last rebuild and reconnect, nothing healed until the next rebuild (runs C-long, B, D).
    • No resource growth was measurable per rebuild. Over 5 offline rebuilds (run B process) the Home's RSS went from 64.03 to 64.21 MB, with threads (15) and fds unchanged. The source says each rebuild leaks one address lookup plus its eviction task; that is below what RSS/threads/fds can show.

    For choosing the fix: after a network change the Home must run a gossip rebuild once the new network is usable. A rejoin is not enough. A rebind or restart is not required. The fix therefore has to trigger a rebuild on a network change, and the per-process cap must not block that rebuild. The cap blocks it today because the count is never reset. A Retry action that runs a rebuild would heal too. A rebuild on the old schedule only heals when the timing happens to work: within about 20 s before reconnect, or at the next scheduled rebuild (up to 300 s later), and never once 6 rebuilds are spent.

    Method

    • Box: Namespace linux/amd64:8x16 (8 vCPU, 16 GB), kernel 7.1.4, Docker 29.3.0. Instance s78183cdljlb2, destroyed at 19:37 UTC; nsc list returns null.
    • Two Homes: two ubuntu:24.04 containers (image sha256:68ee2200…aff33ac), subject and receiver, both on the docker bridge net1 (172.31.1.0/24), with --cgroupns=host.
      • Each has a fresh non-root user rita and the public https://elastos.elacitylabs.com/install.sh.
      • Both were set up the way rc-tests/box-run.sh does it.
    • Install: release-head.json reported 0.8.0-alpha.24, channel canary, release_sha256 6a692802…6f6144, signer did:key:z6Mkgw…Barbfe.
      • elastos --version gives elastos 0.8.0-alpha.24.
      • ~/.local/bin/elastos and update-controller/runtime have the same sha256 25ec5553a3f81c0af3aacf046d7e1ff9ff927be9128fc3c9cd7a683d9c22d81f, on both containers.
      • Source per the 0.8.0: Release 0.8.0 #93 proof: 3beb816438fd567145a8d725e22465bd83277cf0.
    • Sign-up and contacts: each Home signed up with a Playwright 1.49.1 Chromium virtual passkey ("451 Subject", "451 Receiver"). Both were in Community (the default network) and showed "Community is connected".
      • They became contacts through Discovery: request, then Inbox Accept. Discovery was then turned off on both.
      • No message was posted to the public Community conversation. All test messages were DMs between the two test Homes.
    • Outage control (SUBJECT only):
      • docker network disconnect leaves loopback only.
      • After T, docker network connect to a new bridge (net2…net7 = 172.31.2–7.0/24), giving a new subnet and a new IP, like Wi-Fi to a hotspot.
      • RECEIVER and the box never lost their network. The Mac's network was never touched.
    • Direct path between bridges (route.sh), on from 17:57:51 UTC:
      • Docker isolates bridges from each other.
      • DMs are a direct QUIC dial between Homes. Production Carrier is direct-only, with no relay and no public address lookup (source reading by the scout: carrier.rs:1517-1543,1764-1766).
      • So the bridges were made routable to each other, without NAT. Each bridge's NAT to the internet was unchanged. Without this, any cross-bridge DM fails whatever the Home's state.
    • Observation:
      • Watchdog lines: update-controller/runtime.log, which is the elastos home gateway process that runs Carrier and the watchdog. Only watchdog and Carrier lines were kept (logfilter.sh), with non-test IPs and long ids redacted.
      • Community state: transport.available from GET /api/apps/chat-room/summary, polled every 3 s, read-only (watch.js).
      • Resources: every 10 s, RSS, threads, fds and sockets of elastos home and elastos serve (sampler.sh).
      • Packets (from 18:53): UDP packet headers on the bridges, captured from the host netns (cap.sh; per-minute counts only, no addresses).
    • DM check (dm-send.js / dm-recv.js): send from People → Message, wait up to 180 s for the message on the other Home, then the reverse direction. Times below are from send to first seen.
    • Restart: TERM elastos home, kill what is left, start again with elastos home --browser, unlock with the saved passkey.
    • Hidden debug lines could not be enabled. I tried RUST_LOG (the debug filter for carrier and direct messages). The update controller clears the environment of the Home it launches (update_controller.rs:227, env_clear, then an allowlist), so the DEBUG connect and delivery lines are not available on an installed Home. Packet capture was used instead.

    Watchdog schedule seen (all runs)

    • Peer loss shows about 37 s after the cut (available=false at 17:38:59 for a cut at 17:38:22).
    • The rejoin comes about 100 s after the cut. Rebuilds then follow at +65 s, +125 s, +245 s and +305 s (capped wait).
    • The counter is per process and never reset. The cap line is logged at the step after the 6th rebuild, and only rejoins run after that.

    Timeline per run (UTC; "reconnect" = docker network connect)

    Run A: T = 3 min, first process (no route between bridges yet)

    Event Time
    Cut (net1) 17:38:22
    Rejoin 17:40:02
    Rebuild 1 17:41:07
    Reconnect net2 ≈17:41:22 (15 s after the rebuild)
    Community link recovered steps=2 17:41:27, available=true
    • Community recovered: yes, 3–5 s after reconnect, with no step after reconnect.
    • DMs: inconclusive. Both directions stayed "Sending · Delivery unconfirmed" from 17:41 onward. Until 17:57:51 that was a harness artifact, because there was no path between bridges.
      • After the route was added, two fresh DMs still failed for 7.5 min (18:01:54 and 18:01:59, each waited 90 s).
      • After restarting SUBJECT at 18:05:17, all four pending DMs were delivered by 18:06:40.

    Run C: two 3-min outages, then a long one; same process (pid 17814, started 18:07:52, 0 rebuilds)

    Outage Cut Steps while down Reconnect Recovered DM S→R / R→S
    C1, 182 s 18:08:08 rejoin 18:09:39, rebuild 18:10:44 ≈18:11:08 net4 (24 s after the rebuild) 18:11:09.6 (≈2 s) 12.7 s / 14.6 s
    C2, 182 s 18:12:43 rejoin 18:14:24, rebuild 18:15:24 ≈18:15:43 net5 (19 s after) 18:15:44.6 (≈2 s) 2.2 s / 3.2 s
    C-long, 881 s 18:17:01 rejoin 18:18:39; rebuilds 18:19:44, 18:21:49, 18:25:54, 18:30:59 (this process's 6th) ≈18:31:42 net6 (43 s after) no not delivered (≥183 s each)
    • Counts carry over. The two short outages spent 2 rebuilds, so the long outage reached the cap after 4 rebuilds, at 838 s. A fresh process needs about 1,380 s to reach it.
    • After reconnect (C-long):
      • The cap line was logged at 18:36:04. Rejoins followed at 18:36:04, 18:41:09, 18:46:14 and 18:51:19.
      • Community stayed Community is unreachable (since 18:17:37) until the restart, 21 min on a working network.
    • Packets: captures in SUBJECT's network namespace at 18:41–18:42 (40 s), 18:42:54–18:46:14 and 18:49:43–18:51:43 (the last one includes the 18:51:19 rejoin and DM retries). They contain zero UDP packets other than DNS and mDNS.
    • Restart at 18:53:13: Home up at 18:53:26, available=true at 18:53:24. DMs 54.8 s / 2.2 s.

    Run B: long outage; the 6th rebuild runs on the new network (pid 31725, 0 rebuilds at start)

    • Outage: cut at 18:54:29. Rejoin 18:56:07. Rebuilds 18:57:12, 18:59:17, 19:03:22, 19:08:27 and 19:13:32 (5th).
    • Reconnect net3 at ≈19:14:14, after 1,185 s and 41 s after the 5th rebuild.
    • Packets from SUBJECT (out/box/capB-summary.txt):
      • 19:14–19:17: 0/min. The receiver sent 243–290/min to the seed in the same minutes.
      • 19:18: 126, then hundreds and thousands per minute.
    • 6th rebuild at 19:18:37 (4 min 23 s after reconnect). Community link recovered steps=7 at 19:18:42.
    • DMs: 18.4 s / 15.4 s. This healed without a restart.
    • The cap line was never logged, because the link recovered before the next step. The process now has 0 rebuilds left.
    • Restart at 19:20:23 (for comparison): live after 5 s. DMs 160 s / 11.4 s. The first DM after a restart waits on retry backoff.

    The first run.sh B started at 18:53:13 was set to wait for the cap line. At 18:55:23, during the same outage, I switched it to "reconnect after 5 rebuilds" (run.sh B resume 7; logged in events.log). That way the decisive case, a rebuild on the new network, was measured instead of a repeat of C.

    Run D: mechanism check, 6-min outage (pid 39774, from B's restart, 0 rebuilds)

    Event Time
    Cut 19:25:19
    Rejoin 19:27:03
    Rebuild 1 19:28:08
    Rebuild 2 19:30:13
    Reconnect net7 ≈19:31:29 (76 s after rebuild 2)
    Rebuild 3 19:34:18
    Community link recovered steps=4 19:34:23
    • Packets from SUBJECT: 0/min for 19:31–19:33, while the network was working. 228 at 19:34.
    • Live after: 172 s after reconnect, and 5 s after the rebuild.
    • DMs: 93.3 s / 6.2 s.

    Recovery summary

    Run Outage Last rebuild before reconnect Rebuild after reconnect Community recovered DMs after
    A 3 min 15 s none needed yes, 3–5 s inconclusive (no route); restart healed
    C1 3 min 24 s none needed yes, ≈2 s yes, 13–15 s
    C2 3 min 19 s none needed yes, ≈2 s yes, 2–3 s
    C-long 14.7 min 43 s none (cap spent) no, 21 min; rejoins send nothing no
    B 19.75 min 41 s 6th, after 4 min 23 s yes, 5 s after that rebuild yes, 15–18 s
    D 6.2 min 76 s 3rd, after 2 min 49 s yes, 5 s after that rebuild yes, 6–93 s
    Restart (after C, B, A) – – – yes, 0–5 s after Home is up yes, 2–160 s

    The healing window is between 24 s (healed) and 41 s (did not heal) from the last rebuild to reconnect.

    Why, from reading the source (not run; iroh-gossip 0.101.0, iroh 1.0.2, carrier.rs at 3beb816):

    • Dial timeout: a gossip dial has no timeout of its own (iroh-gossip net.rs:1014-1030). The QUIC handshake runs until the idle timeout, which defaults to 30 s (noq-proto transport.rs:555-556, not overridden in iroh quic.rs:153-163). A dial still in flight when the network returns gets through. That fits the 24–41 s cutoff [INFERENCE].
    • A failed dial is never retried: the seed stays Pending { queue: [Join] } in the actor's peer map. The dial failure does not remove it (net.rs:435-444, hyparview.rs:346-355). Only DisconnectPeer removes a peer entry (net.rs:732-736).
    • A rejoin only adds to that queue: it sends no dial (net.rs:695-701), and a leave does not clear the actor-wide peer map (hyparview.rs:357-362). That is why rejoins sent zero packets.
    • A rebuild starts a new actor with an empty peer map, so its first Join dials again (carrier.rs:1286-1313). ElastOS already names this case in carrier.rs:1280-1283 ("a peer left pending after a failed dial").
    • DMs: they share the endpoint. [INFERENCE] They come back after a rebuild because the receiver's address is learned again through the new actor's gossip address lookup.
    • The failed dial is logged as warn "dial failed" under iroh_gossip. The installed log filter hides it (no RUST_LOG, which the update controller clears).

    Resources per rebuild (elastos home, from samples-subject.tsv)

    Process Rebuilds RSS (MB) Threads fds / sockets
    pid 31725 (B) 5 rebuilds while offline (18:57–19:13) 64.03 → 64.05 → 64.16 → 64.06 → 64.21 15 123/76 ↔ 128/81 (alternates; no trend)
    pid 17814 (C) 6 rebuilds over 28 min with Community traffic 62.1–66.0 16 122–132
    • elastos serve was unaffected (41.7–48.3 MB, 10–12 threads).
    • Control: the receiver's elastos home went from 66.5 to 116.9 MB in 2 h without any rebuild, so normal traffic outweighs anything a rebuild adds.
    • Verdict: nothing measurable per rebuild at this granularity. The per-rebuild leak described in collaboration_carrier.rs:32-35 (an address lookup and its eviction task) could not be counted from outside.

    Other observations

    • The Community status was truthful. transport.available went false about 37 s after the cut and true at recovery. While stuck it said "Community is unreachable. Retained messages stay available."
    • "Sending · Delivery unconfirmed" stays up while a DM cannot be delivered. The first DM after a heal or restart can take 90–160 s, because of the retry backoff (5/10/20/40/60 s per the source).
    • Out of scope for Community stays unreachable after a network change until Home restarts #451, source reading only: DMs are direct QUIC between the two Homes, with no relay. Two Homes with no direct IP path between them (for example behind different NATs) would have no DM path at all. This harness needed route.sh for that reason.

    Limits

    • Linux docker veth, where the interface is removed and a new one added. A macOS Wi-Fi to hotspot change keeps the interface and is not covered.
    • The seed is the shared public Community node, and its side was not observed.
    • One box, with one sample per case. B and D agree on the mechanism.
    • No DEBUG logs were available (env cleared by the update controller).

    Harness (/tmp/451-harness/)

    • box-setup.sh: containers, Playwright, install.
    • u.sh: run as rita.
    • start.sh: start Home and the driver.
    • driver.mjs: Playwright driver, from rc-tests/runs/a19-linuxpair.
    • signup.js, unlock.js, disc.js, disc-request.js, accept.js, people.js: sign-up and contact setup.
    • watch.js: Community state.
    • dm-send.js, dm-recv.js, dm-list.js, dm-pair.sh: DMs.
    • sampler.sh: resources and watchdog counts.
    • logfilter.sh: bounded, redacted log copy.
    • route.sh: route between bridges.
    • cap.sh: packet counts.
    • run.sh A|B|C|D: the runs.
  9. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Fix design for 0.8.0 (OMP; devil's advocate MODIFY applied: keep a lifetime resource bound; no new Retry repair path this week).

    • Leak per rebuild, from source (iroh-gossip-0.101.0/src/net/address_lookup.rs:39-89): each rebuild leaves one GossipAddressLookup on the endpoint. That is a map whose entries expire after 5 min, so it empties, plus one task that wakes every 30 s for as long as the endpoint keeps the lookup. It is small but permanent, so the lifetime bound stays.
    • Design:
      1. A named lifetime cap of 120 rebuilds, replacing 6. At most 120 retained lookups and tasks, roughly 4 wakeups per second, well under 1 MB [estimate]. The comment states this cost.
      2. Per-outage pacing: keep today's backoff (60 s doubling to 300 s) for the first 6 rebuilds of one outage; after that, rebuild every 15 min while the outage lasts. The 120 cap then covers about 28 h of continuous outage, or many days of normal network changes.
      3. Network-change trigger: while Community has no peer, a change in the host's non-loopback address set schedules one rebuild right away and resets the outage backoff. It is debounced (the set must be stable for 10 s), gated on the unhealthy state, counts against the cap, and keeps the scheduled fallback. This closes the measured silent window (up to 5 min).
      4. Truthful status: after the cap is spent and Community is unreachable, the status says "Community can't connect. Restart Home to reconnect." No Retry button in 0.8.0. That needs Runtime authorization and serialisation with the watchdog, so it moves to after 0.8.0.
    • Tests:
      • flip Codex's fixtures: no rejoin-only after 6; 15-min pacing; cap at 120; the network-change trigger debounced and gated;
      • rerun the Linux harness (runs B, C and D) on the candidate;
      • macOS Wi-Fi to hotspot in the acceptance round.
    • Owner: the Codex Community stays unreachable after a network change until Home restarts #451 builder lane. Claude reviews it.
  10. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Claude delta review PASS covers d795ae534..850729c29b10bacd73dd4d055d192343be596fc2 / tree b5c60c5e1a908008fdd8ac1ca9f18c0c3fff484c: restart visibility, cap documentation, address filtering, post-cap pacing and log wording are resolved. Independent source/artifact hashes match the committed stamp. This is source review; installed recovery remains open.
    Both completed 850729c29 remote checks end with a missing exit status after passing subsets, with no full gate verdict. The reviewer ran the artifact validator and 10 Node tests; Cargo and compiler-byte reproduction remain separate checks.
    Seven normal source picks into the train preserve their changes and history. The combined Chat artifact commit is 769544347; later train head fdb762f9886e746b4a72fcf6676c1860fc1a75a5 / tree bedbb31e2edc38668fe4162c2b203c58a4bf6045 adds the copy consistency test and notes. Current-head checks and installed Linux/Mac Community, DM and resource acceptance remain open.

  11. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    The 23 focused Rust checks pass for the candidate in the Acceptance record, and controls with the old recovery settings fail as expected. The configured Linux gate is running; installed tests and Claude review remain open.

  12. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Clippy stopped on the single-range fixture. The fixture now uses an explicit one-range iterator, and its data and format checks pass; the current Acceptance record keeps the prior Rust results separate. The corrected candidate still needs its Rust gate, Claude review and installed acceptance.

  13. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Claude review of the production fix at d795ae534 (3beb81643..d795ae534): FAIL, with 2 blocking items outside the watchdog. Report: /tmp/claude-review-net451d795.md.

    • Sound: the lifetime cap of 120 (counted before send, never refunded), the per-outage pacing (60 s doubling to 300 s, then 900 s), the debounced and gated address trigger with the timed fallback, and the getifaddrs wrapper on macOS and Linux.
    • F1, blocking: after the cap, Home's Chat still shows only "Community unreachable". The new restart guidance reaches only a status row that is hidden inside Home.
    • F2, blocking: the developer CHANGELOG and TROUBLESHOOTING.md still describe the old six-rebuild limit.
    • Also fixed in the same round:
      • F3: link-local addresses no longer count as network changes, so fe80 churn on utun, awdl or veth can't trigger rebuilds.
      • F5: after the cap, the rejoin interval grows back to 300 s.
      • The startup log line is reworded.
    • F4, kept: a fast switch while the link still looks healthy waits for the timer path, about 120 s, which is not worse than before. The Mac acceptance run records which path fired.

    The fix round is with the Codex #451 builder.

  14. andersalm commented on Oct 19, 2026

    @andersalm
    Contributor

    Prepared 9c58b4178c4134bbecda9e5883b93c6005e00a43 (tree fb542e2df217ac0836e5c19e8eb8bbabda9ec834) with the F1, F2, F3, F5 and log corrections; the Acceptance record lists the exact checks and retained prior proof.
    The 15 Chat and artifact-validator fixtures, 99 product-data tests, data validation, format and whitespace checks pass. The shared build lease was busy; Chat WASM, bindings and stamp still need an admitted build, and the source/artifact check exits 1.
    OMP owns that build, the new Rust tests and full gate, Claude review, and installed Linux and Mac acceptance. The Mac run will record which trigger starts recovery; #451 stays open.

  15. self-assigned this
    on Oct 21, 2026
  16. andersalm commented on Oct 21, 2026

    @andersalm
    Contributor

    Status (20 Oct): the fix (850729c29, ported to the train as 769544347) is on the alpha.25 train (release/train-0.8.0-rc10, current head f674ff637). Remaining: the Linux harness rerun and the Mac Wi-Fi switch check on signed alpha.25. Stays open until then.

  17. modified the milestones: 2.0.0, 2.0.2 on Nov 11, 2026
  18. andersalm commented on Nov 11, 2026

    @andersalm
    Contributor

    Reopened: the code shipped in 2.0.0, but the installed check in Done means was never recorded, and AGENTS.md closes a task only when Done means is proven. The check is owed in 2.0.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions