Skip to content

fix(members): wait for memory preparation before opening a member thread - #11097

Closed
CrysisDeu wants to merge 1 commit into
mainfrom
fix/member-thread-store-unavailable
Closed

CrysisDeu wants to merge 1 commit into
mainfrom
fix/member-thread-store-unavailable

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Summary

Opening a Crew Member's DM (POST /api/members/{slug}/thread) resolves the member's memory binding through require_member_memory_store, and that resolver fails closed while the gateway is still restoring and indexing memory after a start. The route did not wait on the shared preparation task the way a chat turn does at its admission seam, so the Members page's first open after every restart met a hard 503:

/api/members/kirocrew-conductor/thread -> HTTP 503 store_unavailable
memory store 'default' is unavailable: Memory is being restored and prepared during gateway startup. Retry when recovery completes.

The fence clears seconds later, but the page raised its error notice with no retry.

Change

  • Backend (dashboard/handlers/members.py): the open awaits wait_for_memory_preparation(state.memory_startup_task) before resolving the binding, the same bounded grace a turn gets. A grace that still expires answers 503 with a distinct code, memory_preparing, plus Retry-After. store_unavailable keeps covering refusals only the owner can clear (a failed restore, a broken binding), so a client never sits retrying those. memory_startup.memory_preparation_pending() is the one-line predicate that tells the two apart.
  • Frontend (MembersPage.tsx): a memory_preparing refusal re-POSTs on a bounded timer (3s gap, 10 re-asks; each POST already holds server-side) while the column keeps its existing "Opening the conversation…" state. Past the cap the failure is reported the way any other open failure is. No new user-facing strings.
  • Docs: the memory spec's preparation-barrier section names the route and the new code.

Tests

  • test/test_members_dm_thread.py::TestMemoryPreparationGrace: hold-then-bind within the grace (no slot born while held); retryable memory_preparing past the grace with the shared task untouched and the retry succeeding once preparation lands; structural failure keeps store_unavailable and no Retry-After. Mutation check: the first two fail on origin/main's handler.
  • MembersPage.test.tsx: retry loop until the thread binds (no notice, timer stops on success); cap reached reports the failure with its report and stops re-asking. Both fail with the page change reverted.
  • Ran locally: test_memory_startup.py, test_members_dm_thread.py, test_member_memory_runtime.py (240 passed); src/pages/members/ vitest (155 passed); tsc, eslint, black/isort/flake8/mypy, comment-history gate, i18n check, jscpd, docs lint, error-code contract.

Opening a Crew Member's DM (POST /api/members/{slug}/thread) resolves the
member's memory binding, and that resolver fails closed while the gateway
is still restoring and indexing memory after a start. The route did not
wait on the shared preparation task the way a chat turn does, so the
Members page's first open after every restart met a hard 503
(store_unavailable: "Memory is being restored and prepared during gateway
startup") on a fence that cleared seconds later, and the page raised its
error notice with no retry.

The open now waits on the same task with the same bounded grace a turn
gets at its admission seam. A grace that still expires answers 503 with a
distinct code, memory_preparing, plus a Retry-After header: unlike
store_unavailable, which covers refusals only the owner can clear (a
failed restore, a broken binding), this one is cleared by preparation
finishing, so a client may re-ask on its own. The Members page does so on
a bounded timer while keeping the column in its "Opening the
conversation..." state, and reports a failed open only past the cap.

Tests pin the hold-then-bind path, the retryable refusal past the grace
(shared task untouched, retry succeeds once preparation lands), the
fail-closed code for a structural failure, and the page's retry loop and
cap. The memory spec documents the route's place in the preparation
barrier.
@CrysisDeu
CrysisDeu requested review from a team and buluoray September 15, 2026 20:14
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ✅ PASS

UX-level review of c2c8b3dd75dffbc981c853bf6595f4947515a600 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: PASS

Verify the multi-minute silent "Opening the conversation…" wait after a restart is acceptable without naming its cause.

Suggestions

  • In MembersPage.tsx, on the first memory_preparing retry, add one secondary line under the existing pending state (e.g. "Waiting for memory to finish preparing after restart…") — the client already holds the reason in errorReport.code and a minutes-long generic loader invites a page refresh that restarts the loop.

[UX-REVIEWED] c2c8b3d

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of c2c8b3dd75dffbc981c853bf6595f4947515a600 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

One page-wide retry timer over per-member retry state lets a stale member's retry silently kill the visible member's chain, re-sticking the exact startup window this fixes.

Watch

  • preparingTimerRef is a singleton but preparingRetriesRef/threadReqSeq are per-member. Interleave: open A during preparation (timer set) → switch to B (B POSTs, errors memory_preparing, sets the timer) → A's still-live retry (its seq is unchanged, so the seq guard passes) errors and clearTimeouts B's pending re-ask. B — the active, rendered member — is left permanently in "Opening the conversation…" with no timer, no failure notice, and nothing to re-ask until the user re-clicks. The comment "The timer is one per page: a newer open supersedes it through the sequence check" only holds for re-opens of the same member; a cross-member interleave defeats it.
    Clears when: the timer is keyed per member (mirroring the counters), or a member switch cancels/retires the other member's chain, with a test covering the A→B interleave.

Suggestions

  • The server sends Retry-After: 5 but the client paces on an independent MEMORY_PREPARING_RETRY_MS = 3_000; honoring the header removes the second source of truth for the same interval.

[DESIGN-REVIEWED] c2c8b3d

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging c2c8b3dd75dffbc981c853bf6595f4947515a600. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/dashboard/handlers/members.py:369 -- preparation wait resumes with a stale registry snapshot
await wait_for_memory_preparation(getattr(state, "memory_startup_task", None))
non-default V1 member deleted while preparation is pending -> handler resumes with pre-wait cfg -> creates a live slot and durable DM binding for the deleted member
Anchor: residual/crash-data-loss-corruption
Fix: reload the config and binding after the wait, then recompute and revalidate the slug owner before creating anything.

FINDING -- website/src/pages/members/MembersPage.tsx:730 -- "preparingRetriesRef.current[m.name] = 0" resets the exhausted budget, so reopening starts another ten retries instead of preserving the stated cap -> Fix: retain the capped count for terminal memory_preparing responses and reset it only after success or a different error.

[GPT-REVIEWED] c2c8b3d
[BLOCK-MERGE] c2c8b3d

Adjudication (Opus 4.8) — is blocking on each finding proportionate?

The adjudicable block is empty (0 findings). One fenced finding, F1.

F1 — members.py:369 — Harm rung: data-corruption class (fence-classified unbounded). Conditions confirmed this run: stale cfg/binding captured pre-wait (members.py:338–339); the newly added wait_for_memory_preparation widens the staleness window only while startup preparation is pending (members.py:369); the guard that would refuse a deleted member keys off slug_owners computed from the same stale cfg (members.py:350 → 388), so a member deleted during the wait still passes; the V1 path skips the V2 slug-owner revalidation block (members.py:533, guarded on memory_version == 2); a durable binding and live slot are then created for the deleted member (members.py:481–488, 588–594). Recovery exists — the next open reloads fresh cfg and fails closed at members.py:388 (409 member_pin_mismatch, dm.json untouched) — but it does not contain the residual in-memory slot pinned to a now-deleted agent, and I cannot certify from code opened here that no turn can execute against it. The trigger is a startup-window race, not a mutually-contradicting or writer-impossible condition, and V1 is not a legacy-only artifact, so the FLAG bar (residual a human would plausibly accept) is not met. Torn → UPHOLD-FENCED.

[ADJUDICATION] c2c8b3dd75dffbc981c853bf6595f4947515a600 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] c2c8b3dd75dffbc981c853bf6595f4947515a600
[ADJUDICATION-FENCED] c2c8b3dd75dffbc981c853bf6595f4947515a600 fenced=1 flagged=0
UPHOLD-FENCED F1 src/kiro_crew/dashboard/handlers/members.py:369 -- The staleness guard at line 388 relies on pre-wait slug_owners, the V1 path skips revalidation, and the durable binding plus in-memory slot for a deleted member are only partially self-correcting; the trigger is a plausible startup race, not an extreme condition, so residual risk is not clearly acceptable.
[GPT-ADJUDICATED-FENCED] c2c8b3dd75dffbc981c853bf6595f4947515a600

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c2c8b3dd75dffbc981c853bf6595f4947515a600: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of c2c8b3dd75dffbc981c853bf6595f4947515a600 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The 3s/10× re-POST loop re-implements react-query's retry/retryDelay, and the Retry-After: 5 header ships with zero readers — the page ignores it.

Not justified as shipped

  • Item 4 — duplicate of @tanstack/react-query retry/retryDelay: "re-POSTs a memory_preparing refusal on a bounded timer (3s gap, 10 re-asks)" is hand-rolled with preparingRetriesRef, preparingTimerRef, an unmount effect and a setTimeout, while the same useMutation already expresses predicate-gated retries — retry: is used at 15+ prod sites in website/src (grep retry:|retryDelay). The comment's deltas (unmount cancel, supersession cancel) only avoid stray idempotent POSTs to a get-or-create endpoint.
    Subtraction: replace the refs/timer/effect block in MembersPage.tsx with retry: (n, err) => findReport(err.message)?.code === MEMORY_PREPARING_CODE && n < MEMORY_PREPARING_RETRIES, retryDelay: MEMORY_PREPARING_RETRY_MS on openThread.
    Clears when: the plumbing is replaced by the mutation options, or a concrete failure the built-in retry cannot avoid (what a post-unmount POST actually breaks) is named.
  • Item 3 — zero consumers: the Retry-After: 5 header (MEMORY_PREPARING_RETRY_AFTER_SECONDS, memory.py) has no reader — grep retry-after in website/src: 0 matches — and the route's only admissible client (owner dashboard; app callers get 404) re-asks at its own 3s, already diverging from the header's 5.
    Subtraction: delete MEMORY_PREPARING_RETRY_AFTER_SECONDS and the header (the memory_preparing code alone drives the client), or read the header and delete MEMORY_PREPARING_RETRY_MS.
    Clears when: a counted consumer reads the header, or the header and constant are deleted.

What this change ships

Inventory (7 items) — 5 justified

Intent: FIX — opening a Crew Member's DM right after a gateway restart should succeed once memory preparation finishes instead of failing immediately with a 503 notice.

  1. Opening a member thread after a restart now holds on the shared preparation task and succeeds — justified
  2. A hold that outlasts the grace answers a distinct retryable code, memory_preparing, instead of the owner-actionable store_unavailable — justified
  3. That refusal carries a Retry-After: 5 header — zero consumers (no client-side reader; the page uses its own 3s gap)
  4. The Members page silently re-POSTs the open up to 10 times at 3s, keeping "Opening the conversation…" — duplicate of useMutation retry/retryDelay (second spelling of the library's retry)
  5. Past the retry cap the open is reported as a failure like any other, with its error report — justified
  6. New backend predicate memory_preparation_pending() telling retryable from fail-closed refusals — justified
  7. The memory spec's preparation-barrier section now names this route and code — justified

[FIRST-PRINCIPLES-REVIEWED] c2c8b3d

@github-actions

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed c2c8b3dd75dffbc981c853bf6595f4947515a600 — this comment is updated in place on each push.

Review details

No blocking issues — one advisory finding on the retry-counter reset.

FINDING — website/src/pages/members/MembersPage.tsx:730 — the failed branch runs preparingRetriesRef.current[m.name] = 0 even when the cap was hit on a memory_preparing error, so a later open of a still-preparing member re-enters the full 10-retry cycle (each POST holds server-side for the admission grace ≈ minutes) instead of reporting at once, contradicting the same PR's contract ("every later open of that member reports at once … until a confirmed open resets the count") → Fix: reset the counter to 0 only for non-memory_preparing errors; for a memory_preparing error past the cap leave the count at/above MEMORY_PREPARING_RETRIES so subsequent opens report immediately (the confirmed-open reset already lives in onSuccess).

Candidate 2 dropped: only one member is active/displayed at a time, and activate() fires a fresh postThread for the newly-active member; a dropped timer only affects a member that is no longer on screen, and re-activating it (the sole way to view it again) re-POSTs — no observable stuck state.

[OPUS-REVIEWED] c2c8b3d

Verdict parsed from the review's SHA-scoped output markers for commit c2c8b3dd75dffbc981c853bf6595f4947515a600.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c2c8b3dd75dffbc981c853bf6595f4947515a600: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Sep 15, 2026
@dwu96 dwu96 added the needs-pr-triage PR scanner: awaiting automated triage label Sep 21, 2026
@chenmingwei23 chenmingwei23 added drive-to-green PR claimed by drive-to-green pipeline and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 21, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: chenmingwei23]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author.

Assessment: GPT flags a real reachable defect in members.py: after the newly added preparation wait, the handler resumes with a stale cfg/binding and can create a live slot + durable DM binding for a member deleted during the wait. Fix: reload config and binding after the wait, revalidate the slug owner before creating anything; also retain the capped retry count for terminal memory_preparing responses (MembersPage.tsx). Branch conflicts with main — rebase too.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

@bolichen97

Copy link
Copy Markdown
Contributor

Closing as superseded: origin/main no longer fences the member-DM open on memory preparation

This PR works around the 503 store_unavailable that POST /api/members/{slug}/thread returned while the gateway was still preparing memory. PR #11780 (bc0d2b0, merged 2026-09-19) took that fence off the open path instead of waiting on it. On main the open now answers right away, even while preparation is still pending.

Where main has it:

  • src/kiro_crew/dashboard/handlers/members.py:591-599 (main): _member_thread_slot now picks "the member's deterministic DM key without opening learned memory" through require_member_memory_store(cfg, member, require_directory=False). At this head it is still require_member_memory_store(cfg, member) (members.py:294).
  • src/kiro_crew/memory_stores.py:650-712 (main): the resolvers no longer call require_memory_ready. bc0d2b0 removes all three calls.
  • Reviewers ran the same open against both trees and left preparation pending. The merge base returned 503 store_unavailable with the message quoted in the description. Main returned 200 (slot_key=member-code-reviewer).
This PR (c2c8b3d) On main
wait on wait_for_memory_preparation (members.py:369) not needed, because there is nothing to wait for
503 memory_preparing + Retry-After (memory.py:265-280, members.py:384-385) cannot be reached; main never sends this code
memory_preparation_pending() (memory_startup.py:117) nothing on main would call it
3 s x 10 re-POST (MembersPage.tsx:322-326, :681) would never run; main keeps the plain onError (MembersPage.tsx:1145-1152)
spec paragraph (memory-skills-hooks.md:1853-1860) would describe behavior main does not have

Rebasing this PR (as the 2026-09-21 autofix note proposed) would make main worse. On main, wait_for_memory_preparation (memory_startup.py:117-132) raises at once after a failed restore and raises after 30 s otherwise. Opens that main answers with 200 would first wait up to 30 s, and then get a 503 whenever preparation fails or takes longer than 30 s. This is also why all three TestMemoryPreparationGrace cases (test/test_members_dm_thread.py:1815) fail on main: each one expects a wait or a 503, and main returns 200.

What is left: main has no test that checks "a member-DM open while preparation is pending returns 200 without waiting". That would be a small test PR on its own and is no reason to keep this one open. Nothing else is worth keeping.

If the open still hits the fence on main in some case I missed, please reopen with a repro.

From a full open-PR audit against origin/main c66f95b (2026-09-25); findings were checked by independent reviewers reading the code. Reply here if this is wrong.

@bolichen97 bolichen97 closed this Sep 25, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 25, 2026
@dwu96 dwu96 removed the drive-to-green PR claimed by drive-to-green pipeline label Sep 25, 2026
@bolichen97
bolichen97 deleted the fix/member-thread-store-unavailable branch November 6, 2026 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants