Skip to content

fix: retry transient finite reads before recovery actions - #1791

Merged
kalvinnchau merged 3 commits into
mainfrom
am/background-retry
Oct 28, 2026
Merged

kalvinnchau merged 3 commits into
mainfrom
am/background-retry

Conversation

@kalvinnchau

@kalvinnchau kalvinnchau commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Transient read failures can expose Retry controls before the network recovers.
Finite read owners now make one initial attempt plus up to three quiet retries
before the existing Retry action appears. Loaded data stays in place and obsolete
work is cancelled.

Policy (src/features/relay/read-retry.ts):

  • Backoff ceilings follow session recovery's first three delays (1/2/4 s) with
    equal jitter. No retry starts after 10 s. Exhaustion rethrows the last failure.
  • An admission hint within the current ceiling is honored. A longer hint surfaces
    immediately to the existing cooldown.
  • Only transient failures retry. Denied, invalid and deterministic local
    failures, cancellations and writes never retry. HTTP classification uses one
    shared transientStatus (408/429/5xx).
  • One retry owner per read path. Internal roster, head, catch-up and preference
    reads keep the session recovery coordinator.
  • In the session, access revocation and cache clears end a read's backoff under
    the same rule that retires its in-flight requests.
  • Media gets one quiet recovery budget per source. A later failure of the same
    source shows the existing failed state.

Covered surfaces: hidden conversations, public and global search, directory,
thread and older history, profiles, repositories and Git reads, staff reads,
GitHub objects, GIFs, Builderlab agents, the inbox, relay agent inventory and
media reloads. The hidden-DM check and inbox use the session's retry rather than
their own. Agent-provider
catalogs (Codex, Claude Code, goose, ACP) and profile settings keep their
existing behavior.

Tests:

  • read-retry.test.ts owns the policy matrix: boundaries, exhaustion, quiet
    window, permanent failures, admission hints, cancellation and late success.
  • Owner tests cover wiring and one representative failure per owner: session
    (including inbox and cached-only), channel lifecycle and sidebar roster churn,
    hidden DMs, agent inventory, Git, relay staff, GitHub, Builderlab and media.
  • One browser case added, none removed. hidden-conversation-retry.spec.mjs checks
    the actual sidebar, session and broker wiring through exhaustion and the
    existing manual Retry. Component tests cannot prove that wiring.
  • history-loading.spec.mjs: two older-page quota cases now use a hint longer
    than the first backoff ceiling. A 0 s quota (1 s hint) is now retried quietly
    by design, so their deliberate-retry assertions need a quota that surfaces.
    Both failed in CI before the change and pass in Chromium and WebKit after.
  • The session backoff retirement is covered by session-read-retry.test.ts
    (cache clear) and navigation.test.tsx (access revocation). Removing either
    call fails its test.

Not exercised: native signing/IPC, live-relay enforcement and native media
playback.

@kalvinnchau
kalvinnchau requested review from a team as code owners October 28, 2026 05:18
@kalvinnchau

Copy link
Copy Markdown
Contributor Author

🤖 Retry behavior before and after this change, based on the source comparison from base 3c8c568e to head fad9d4be (the PR has since advanced).

Operation Before After When it still fails
Finite reads: search, profiles, older messages, thread pages, repositories, staff lists, provider catalogs Generally one attempt, then existing error/Retry action Initial attempt + up to 3 automatic retries for transient failures: 4 attempts total per finite read Stop; surface the existing error/action. Retained views keep previously loaded data.
Hidden-conversation refresh Transient failures entered indefinite session recovery 4 attempts maximum per refresh Keep confirmed hidden IDs; show “Hidden conversations could not be refreshed.” and “Retry hidden conversations”. Manual retry or a later completed roster refresh can start a new bounded refresh.
Existing session background reads: live/roster, initial messages, catch-up, templates/preferences Indefinite recovery Unchanged Continue background recovery; delay grows through 1/2/4/8/16/30 seconds, capped at 30 seconds.
Voice/video resource loading Explicit recovery after load failure Original load + 3 reloads, only for browser network error code 2 Existing failure/control appears. No automatic play().
Usage / avatars / config sync / pairing / updates Already had their own retry/polling owners Unchanged; no extra retry layer Existing behavior. Usage keeps one retry after 500 ms; avatars keep 2/10/30-second recovery.
Writes and interactive actions: send, delete, save, upload, sign-in, installation, permissions Existing explicit/durable recovery Unchanged User action or the existing operation owner handles recovery. No new silent replay.

Finite retry waits are 0.5–1 s, 1–2 s, then 2–4 s: 3.5–7 seconds of backoff plus request execution time, subject to existing deadlines/cancellation. Four attempts is per finite read, not a screen-wide or session-wide request cap.

Permission failures, invalid responses and deterministic local/setup failures surface immediately. Caller cancellation stops obsolete work. A relay admission delay longer than the next retry ceiling surfaces to the existing cooldown controls. Exhaustion markers prevent nested finite retry owners from restarting an exhausted budget; hidden-conversation exhaustion also stays out of indefinite session recovery.

Sources: finite retry helper, surface inventory, background recovery. Source explanation only; this is not a code review or a claim of runtime validation.

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Carl, an automated reviewer, commenting via Wes's GitHub account.

Reviewed at 4181e83. I recommend fixing the two blockers below before merge. The rest is non-blocking. The core policy holds up: backoff retirement on revoke and cache clear, the epoch/generation fences, last hidden IDs kept through exhaustion, and no write paths or other nested budgets retried.

Recommended blockers

  • session.read/readPage retries compound with useHiddenDms' own loop (inline on session.ts).
  • Media recovery gets a fresh budget whenever metadata loads, so reloads never terminate (inline on use-media-read-recovery.ts).

Non-blocking: the agent-library retry can never fire, the hidden-conversation refresh restarts on every roster commit, and the inbox retries a cached-only failure (all inline). Optional: the lifecycle recoverable(...) source at session.ts:1854-1858 can no longer report failed, so its retry is unreachable. Trim it or add a comment.

Validation gaps: source review plus two temporary targeted Vitest probes (inventory, media). The media evidence is jsdom event ordering; native playback is untested. No app runs. CI is green at this head.

Comment thread src/features/relay/session.ts
Comment thread src/features/messages/use-media-read-recovery.ts
Comment thread src/features/agents/library.ts Outdated
Comment thread src/features/channel-navigation/ChannelInventory.tsx
Comment thread src/features/relay/inbox-feed.ts Outdated
kalvinnchau and others added 3 commits October 28, 2026 10:09
bound reads to four attempts with jitter and caller cancellation.
retain existing recovery owners, deadlines, and explicit mutation retries.

Co-authored-by: am <6e30cd56c30e030cd31bb0939b94a7c257c9a09d5ba2d92cf2735da45629f248@buzz.block.builderlab.xyz>
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
drop the nested retry budget marker and per-read elapsed caps; retryRead
now owns a 10s quiet window. share transientStatus across gifs, github,
builderlab, relay-staff and git. retire session backoffs with the same
rule the scheduler uses for in-flight reads. revert provider-catalog,
codex, claude-code, goose, acp and profile-settings retry plumbing and
the surface inventory doc. consolidate duplicated tests.

Co-authored-by: peon <9ac6794b000690b7e814eb1805ad32405d0bec7d52838de3a86cf967565dacc0@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
drop the hidden-dm hook's own read loop, leaving session.read as the only
retry owner. spend one quiet media recovery budget per source so a
failure after loadedmetadata surfaces. retry the relay source inside
inventoryReader instead of a library wrapper that could not fire. keep
the hidden-conversation refresh signal per lifecycle so roster commits
join it. route inbox reads through finiteRead so cached-only failures
surface at once.

Co-authored-by: peon <9ac6794b000690b7e814eb1805ad32405d0bec7d52838de3a86cf967565dacc0@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kalvin Chau <kalvin@block.xyz>
@kalvinnchau

Copy link
Copy Markdown
Contributor Author

🤖 Optional recoverable(...) lifecycle source: kept, with a comment at its definition. It still reports busy to the recovery coordinator during visibility loads, but visibility exhausts without transient, so its retry is not reached. Removing it would change the coordinator's busy accounting.

@wesbillman wesbillman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Carl, an automated reviewer, commenting via Wes's GitHub account.

Re-review at 36fbdfd: both recommended blockers and all three non-blocking findings from the previous review are fixed, and 36fbdfdc4 introduces no new defect. The two original commits are range-diff identical after the rebase. CI is green.

  • session.read is now the only retry owner for hidden-DM reads (at most 4 reads per DM).
  • Media gets one quiet budget per mounted source, so metadata alone no longer replenishes it. The same targeted probes for media and inventory now pass 2/2 (previously 0/2).
  • Inventory retries the relay source before aggregation, and partial-source results are preserved.
  • Hidden-conversation roster commits join the in-flight refresh. Reconnect self-heal still runs through asOf.
  • Inbox reads go through finiteRead, so cached-only failures surface at once and revoke/clear retire their backoffs.

Optional: when finite reads keep failing while live discovery keeps arriving, the hidden-conversations role="alert" drops out and reappears about every 10 s, and screen readers re-announce it each time.

Validation gaps: source review plus the targeted Vitest probes; no app runs. Native media playback is unverified.

@kalvinnchau

Copy link
Copy Markdown
Contributor Author

🤖 Thanks for the re-review. Leaving the optional alert re-announcement as is. The one-line fix (do not emit loading over a standing error) would also take away the immediate response to a manual "Retry hidden conversations" click. Keeping both would need extra state to tell automatic roster-driven refreshes from manual ones, which is more than this edge case (finite reads failing continuously while live discovery keeps committing) justifies here. Main already shows the same drop-and-return for non-transient errors on each roster commit. Happy to take it as a follow-up if wanted.

@kalvinnchau
kalvinnchau merged commit fffd985 into main Oct 28, 2026
38 checks passed
@kalvinnchau
kalvinnchau deleted the am/background-retry branch October 28, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants