Skip to content

feat(members): prune sync-generated crewmates once at startup - #12798

Merged
iamwhatever merged 1 commit into
mainfrom
feat/crewmate-optin-existing-agents
Sep 24, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
feat/crewmate-optin-existing-agents

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Older dashboards called POST /api/agents/sync on every chat mount, and that
sync made a crewmate out of every custom agent under ~/.kiro/agents: a
config.agents row with no member_id, on the shared default memory store,
bound to the agent by name. An existing user therefore opens the Crewmates page
to one crewmate per custom agent, most of them never used and none with a
memory of its own.

Why it matters

These are exactly the users the Crewmates launch is for. Without this, the launch
lands on a roster full of crewmates nobody asked for, and the ones the user does
chat with are shared-memory shells rather than real members.

What changed (motivation → approach → change)

Implements the accepted RFC: docs/request-for-change/rfc-crewmates-launch.md
(screen 03, existing users) — as a one-time startup migration, no UI
(amendment #13032).

Piece What it does
crewmate_prune_migration.py (new) Once, when the dashboard gateway starts (not the headless --slack-only entrypoint): finds the rows an older sync generated, removes the never-chatted ones, leaves the chatted ones untouched, keeps any it cannot judge, writes a marker. The whole pass runs under an exclusive cross-process lock (<config dir>/crewmate_prune.lock, platform_compat.file_lock, waited for without give-up: a contender that returned early would open its own barrier while the holder can still delete) and the marker is created O_EXCL, so two gateways on one data home cannot both prune
dashboard/server.py _kick_crewmate_prune runs the pass as a tracked background task right after the listener binds (the _kick_* shape); nothing on the startup path awaits it, so a session-count-scaled scan never gates readiness. Slot restores need not wait: a removed row has no DM binding and no session naming it, so no slot can be rebuilt for it. _register_crewmate_prune_gate, installed next to the workflow gate BEFORE the bind, clears crewmate_prune_settled and adds one middleware that holds every request whose method is not GET/HEAD/OPTIONS, on any path (/api/* and POST /v1/chat/completions alike), plus every request under /api/members whatever its method (the roster read folds and retires a member's legacy activity file and appends reconciles to the member log — the places the pass reads), until the pass sets the event in finally (60 s, then 503 prune_in_progress; a 503 writes nothing). The non-HTTP writers (subagent pump, channel agent resume, cron) start only after await_crewmate_prune_settled returns — in slack/gateway.py run() past KIROCREW_READY, so readiness never waits — and it returns only once the pass has RETURNED: past its 60 s budget it sets crewmate_prune_abandon (the pass polls it before each candidate and inside the config lock before each delete, keeps the rest under doubted, writes the marker, returns) and keeps waiting. No writer ever runs beside a pass that can still delete; the pass always returns (non-blocking opens, bounded lock acquires). Fast path: one is_set() read
dashboard/state.py crewmate_prune_settled event (set by default so tests and the CLI never wait); crewmate_prune_abandon, the stop signal the pass's worker polls
channel_transcript_migration.py migrate_channel_transcripts(remove=...): the startup merge of orphaned dashboard copies into their channel transcripts still lands before the session restores, but while the prune has not settled the copies are kept — an orphan's first line is the only record of the agent its dashboard surface ran as, the evidence the pass reads — and _kick_deferred_transcript_removal (a tracked task, off the readiness path) re-runs the migration with removal on once the pass has returned; the re-merge is byte-identical

A candidate is a row that is exactly what the sync wrote: its name is its
kiro_agent; the spec is on disk, user-authored, not the runtime's own or a
private copy; its description equals the spec's; every other field is at its
default, no undeclared key (_is_fresh_sync_shape, on the raw row); a name
config.local.json touches is never one. Chatted means any of: a session's metadata names
it in agent or execution_context; its member activity log holds a session pointer for it
(record_activity; survives the agent switch that rewrites the metadata); or
its DM thread was opened. All three are read strictly here.
Never-chatted → deleted under the base config lock and, nested, the overlay's
and the spec's (overlay then spec, both held until the delete commits), only
while the row still has the sync shape, the overlay does not name it and the
spec still carries that description; a change is refused (no marker). Chatted → untouched, shared store, no member_id: a memory
binding is identity, chosen at creation (memory-skills-hooks.md). Removal
needs a complete history; the pass always finishes.
Two kinds of session file
are kept apart (_session_agents_named): one that reads and parses is
evidence whatever it says — every agent its record names, or nobody (an older
build's first line, a record naming no agent, another agent) — and never voids
the pass; one
that cannot be read (open/read failure, gone since the listing, not UTF-8,
empty, over budget, not JSON) makes the history incomplete, and an incomplete history removes nothing:
every candidate is kept (doubted), the files listed (unreadable_sessions).
Per candidate, an activity log that cannot be loaded or whose segments are
present but empty (a torn write the store reads as absent), an unreadable
legacy activity file or DM binding each keep that crewmate. The marker still
lands, so one bad file never makes the prune re-run on
every boot; a kept row loses nothing. Agent-writable files are opened with
open_file_no_reparse, regular files only (a FIFO is "no record", never a
hang). Only config.json rows move. A completed pass
writes <config dir>/crewmate_prune_migrated.json and logs one INFO line,
removed N unused auto-generated crewmates: <names>, plus a WARNING for any
kept on doubt.

What changed vs the RFC's screen 03. The RFC offered a user with custom
agents but no crewmates to add them. That path is not this PR's job: #12224
ended the enrol-on-mount behaviour, and the Crewmates page's empty state with
New crewmate (#12924) is the way from a custom agent to a crewmate. An
existing user carries the opposite — the crewmates sync already made — so this
PR settles those once, at startup, with no UI (RFC amendment #13032, merged).
Two refinements past its wording: an edited generated row is kept; an
incomplete history keeps every candidate.

Specs: crew-mode.md (new section; the catalog's note on the sync's exclusions
updated), docs/feature-map/README.md (new row).

Finding (GPT 5.6, span eced8e08c12f, crewmate_prune_migration.py) Response
Spec lock ends before deletion commits (:756) The spec lock is now entered on the same ExitStack as the overlay's, inside the base config lock, overlay then spec — the order _write_bindings and the template create keep — and both are released only after update_config_locked has renamed the base file into place. A spec edit that waits on the lock therefore lands after the row is gone, never between the description re-read and the delete. Two tests: the base file already lacks the row when the spec lock is released; the nesting order is overlay then spec.
Transcript scan races startup transcript deletion (:508) Two changes. The pass no longer skips a file that is gone between the listing and the open: it is listed under unreadable_sessions, the history is incomplete, every candidate is kept (test added). And the one startup step that deletes a transcript — migrate_channel_transcripts, which merges an orphaned dashboard copy into its channel transcript and drops the copy's agent — now runs with remove=False while the pass has not settled: the merge still lands before the restores, the copy stays with its first line intact, and _kick_deferred_transcript_removal removes the copies once the pass has returned, off the readiness path (nothing on it waits). Tests: remove=False merges but keeps the copy and its first line, and a removing pass then converges; the deferred removal starts only after the event is set, with removal on.
Legacy agent switches erase the only usage evidence (:441) Per-session evidence is now the union of every agent the metadata record names: agent, execution_context.selection_name, execution_context.template_id (an interrupted switch can leave them apart). Nothing below that line carries agent provenance: ConversationLog.append writes role, content, ts, tools, source and meta.mid; api_chat_slot_agent rewrites the metadata line in place and appends no row, so no switch marker exists and no earlier agent can be recovered from a transcript. A session that ran as a crewmate, switched, and had every field overwritten is found only through the member activity log. Residual, accepted by the maintainers' ruling: a session from before the activity log that did exactly that leaves no trace; its crewmate row is removed, its agent file and shared store stay, New crewmate re-enrols it. A pass is not voided by such sessions existing.

Backwards compatibility

Compatible for configuration and files: no key changes shape, no agent file is
touched, transcripts stay. One behaviour changes on purpose: a never-chatted
sync-generated crewmate disappears from the roster on the first start of this
build — its custom agent is still there and New crewmate re-enrols it in one
step. A chatted one keeps its name, thread and exact memory binding.

Tests

CI note: Backend Tests (3.12, 1) and macOS Tests (2) reds on this branch are
test_pdf_extract.py (TestChildCaps::test_main_reports_a_wrapped_memory_error_and_exits_failed
on Linux; TestBound::test_the_child_itself_exits_with_the_memory_report_under_the_profile
plus a TestChildCaps worker crash on macOS), the xdist flake main also shows
(tracking issue #13030); not chased here. Backend Tests (3.12, 3) red is
test_dashboard_server_startup_coverage.py::TestReserveDashboardPort::test_bind_once_matches_kernel_listener_posture
(OSError: [Errno 99] binding on the runner), red on main too; not chased here.
Backend Tests (Windows) (2) red is
test_taskrunner_coverage.py::TestLogTask::test_on_loop_offloads_the_write
(an off-loop write not observed within 100 sleep(0) ticks on the Windows
runner); the task runner is not in this diff, not chased here.

  • test/test_crewmate_prune_migration.py — seeds an old-style config (one
    chatted, one never-chatted, one hand-created with a member_id, one
    sync-shaped row that owns a non-empty memory store) and asserts exactly the
    never-chatted one on the shared store is removed, the chatted one keeps its
    exact binding (shared store, no member_id), the hand-created one and the one
    with memory are untouched, the marker names what happened, and a second boot
    is a no-op;
    a session that merely named the crewmate counts as chatted, and so does an
    activity record in the member event log or the legacy activity.jsonl /
    .1 (for the exact name only — a colliding slug's other member does not);
    a session file that does not parse, is not UTF-8, is empty, has a first line
    over budget, cannot be opened or vanishes after the listing keeps every candidate and is named in the
    marker, while one that reads but names another agent, no agent, or is JSON
    that is not a record is evidence and removes as usual; a missing
    conversation log keeps every candidate, an unresolvable or
    containment-refused binding path, a malformed or non-UTF-8 binding file, an
    unparseable or over-budget activity file, a corrupt event log and a torn
    (zero-byte segment) event log each keep that crewmate only, a FIFO or link
    at a session or activity path is "no record" and does not hang, and a
    doubted pass is not re-run; a doubt on one candidate does not spare the
    next; a binding naming another crew is not this crewmate's; a row that gains
    a model — or an overlay leaf — between the judgement and the lock is refused
    with no marker; a legacy row missing keys is still removed; a row with a
    model, triggers, a star, an avatar, another workspace, a description that
    differs from the spec's, an undeclared key or a renamed key is never a
    candidate, nor is a name config.local.json touches; a row or spec
    description edit under the lock refuses the delete, and the overlay and spec
    locks outlive the base write, overlay then spec; the candidate filter (spec gone, package
    spec, runtime-owned, private copy) admits only the user's own; a pass told
    to stop keeps every unjudged row under doubted, still writes the marker,
    and a stop that lands inside the config lock leaves that row; the gate holds
    GET /api/members and the routes under it (not a path that merely shares
    the letters); the writer-side wait returns as soon as the pass settles, and
    past its budget sets the stop signal and still does not return until the
    pass has; the cross-process lock: a pass waits for a holder and then finds
    its marker, a holder that outlasts the wait is waited for (never given up
    on), a marker that appears while waiting ends the wait,
    the marker is created exclusively, a failure inside the pass is not read as
    contention, and the lock is released after a pass that returns or raises.
  • test/test_agent_spec_hardened_reads.py — the strict-reader call-site
    ratchet gains the prune's under-lock spec re-read, labelled
    crewmate_prune / dashboard like the other dashboard-surface reads.

Manual verification

N/A — a startup migration with no UI; the unit tests above run the real config
loader against an isolated home.

Related Issues

no linked issue: launch-review item, no tracking issue filed.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and pepmach September 22, 2026 12:29
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Intent: Offer an existing user who already has custom agents but no crewmate a one-time, one-screen way to turn the agents they use into crewmates, then drop them into the most recently used one's chat.
Not a goal: Renaming the Crewmates page or its copy, changing how a crewmate is created (the step calls the existing create route), a way to re-open the step from the UI, or listing package-installed and runtime-owned specs as candidates.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision ffd011dd29d4bbe46e170cc0a5249b39098750a6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of ffd011dd29d4bbe46e170cc0a5249b39098750a6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

A fixed 60-second budget ties first-boot write availability to history size; overrun abandons the prune permanently for exactly the heavy-history installs it targets.

Watch

  • The pass's cost "scales with the user's history" (its own comment), yet the budget is one constant: past 60 s every mutating request — chat send, POST /v1/chat/completions — answers 503, await_crewmate_prune_settled sets crewmate_prune_abandon, the remainder lands in doubted, and the O_EXCL marker makes that outcome final ("a doubted pass is not re-run"). The heaviest histories are the likeliest to overrun, so the launch-target users with the most sync clutter are the ones most likely to keep it, silently, after a one-boot write outage.
    Clears when: a measured pass duration on a realistically large history (thousands of session files) shows it settles well under 60 s, or an abandoned/doubted pass leaves state a later boot retries.

Suggestions

  • Record doubted/abandoned rows in the marker as retryable and re-judge only those on the next boot — the per-boot cost objection applies to full re-runs, not to a bounded retry of named rows.

[DESIGN-REVIEWED] ffd011d

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed ffd011dd29d4bbe46e170cc0a5249b39098750a6 — this comment is updated in place on each push.

Review details

Two startup waits on the new prune can each hang forever, so a stuck lock holder leaves the gateway permanently un-ready with every mutating request 503ing.

FINDING — src/kiro_crew/crewmate_prune_migration.py:876 — the acquire loop while True: / except OSError: report.lock_waits += 1 retries forever and never consults abandoned(), and it treats every OSError as contention even though platform_compat._posix_acquire_blocking re-raises non-EAGAIN/EACCES/EWOULDBLOCK errnos precisely so the caller stops; a stale holder (or ENOLCK/EIO on the lock file) means prune_synced_crewmates never returns, crewmate_prune_settled is never set in _kick_crewmate_prune's finally, and the dashboard's own file_lock timeout — documented as existing because "an unbounded wait on a stuck holder is not a wait but a hang" — is defeated by the wrapper → Fix: in the except OSError arm, return report when abandoned() is true or when the errno is not a contention errno, instead of unconditionally re-entering the loop.

FINDING — src/kiro_crew/dashboard/server.py:3455 — after the 60 s bounded wait expires and crewmate_prune_abandon is set, await_crewmate_prune_settled falls into a bare await state.crewmate_prune_settled.wait() with no ceiling, so the abandon signal only helps if the pass is at a point that polls it — with the worker blocked in the lock loop above, GatewayOrchestrator.run never reaches cron, heartbeat, subagent dispatch or the dashboard workers, and no supervisor or health check sees a reason → Fix: wrap that second wait in asyncio.wait_for with its own ceiling and log-and-continue on timeout rather than blocking the boot sequence indefinitely.

FINDING — src/kiro_crew/dashboard/server.py:6133 — in standalone start_dashboard, await await_crewmate_prune_settled(state, before="channel agent resume") sits ahead of state.ready = True and record_boot_to_ready(...), so the prune is on the readiness path after all and up to 60 s (unbounded, per the finding above) of it is charged to boot-to-ready with the dashboard answering prune_in_progress 503s meanwhile → Fix: move the _resume_channel_agents() call into a tracked background task that awaits crewmate_prune_settled first — the shape _kick_deferred_transcript_removal already uses — leaving readiness unblocked.

[OPUS-REVIEWED] ffd011d

Verdict parsed from the review's SHA-scoped output markers for commit ffd011dd29d4bbe46e170cc0a5249b39098750a6.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ffd011dd29d4bbe46e170cc0a5249b39098750a6: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of ffd011dd29d4bbe46e170cc0a5249b39098750a6 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ffd011d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ffd011dd29d4bbe46e170cc0a5249b39098750a6: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of ffd011dd29d4bbe46e170cc0a5249b39098750a6 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

PruneReport.lock_waits and skipped_marker have zero production consumers — only tests read them; the WARNING line and marker already record the same facts.

Not justified as shipped

  • Item 7 — zero consumers: lock_waits (1 write, 3 test asserts, 0 production reads; grep lock_waits) and skipped_marker (3 writes, 6 test asserts, 0 production reads; grep skipped_marker) are returned to a caller that reads only prune.removed (_kick_crewmate_prune), and neither reaches the marker body.
    Subtraction: delete both fields from PruneReport (crewmate_prune_migration.py:221,224); the lock-wait WARNING and the marker's presence already carry what they count.
    Clears when: the fields are deleted, or a production reader of either is named.

What this change ships

Inventory (8 items) — 7 justified

Intent: FIX — existing users' Crewmates rosters stop carrying the never-used crewmates an older enrol-on-mount agent sync auto-created. Covered by the base's accepted RFC rfc-crewmates-launch.md (acceptance §7 names this exact prune; §8 names PR #12798).

  1. First start after upgrade removes never-chatted sync-generated crewmates from the roster, once — justified
  2. Chatted sync-generated crewmates stay exactly as they were (shared store, no member_id) — justified
  3. Two new persisted files under the config dir: prune marker and cross-process lock — justified
  4. On the first boot, every mutating request (any path) and every /api/members request waits for the pass, 503 prune_in_progress after 60 s — justified
  5. Channel agent resume, cron dispatch and the subagent pump start only after the pass returns — justified
  6. Orphaned channel-transcript copies survive the startup merge until the pass settles, then a deferred task removes them (remove kwarg, 2 production callers with differing values) — justified
  7. PruneReport.lock_waits / skipped_marker diagnostic fields — zero consumers: tests only
  8. New crew-mode.md section and feature-map row, same commit — justified (AGENTS.md same-commit doc rule)

[FIRST-PRINCIPLES-REVIEWED] ffd011d

@CrysisDeu CrysisDeu mentioned this pull request Sep 22, 2026
5 tasks done
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 22, 2026
@CrysisDeu
CrysisDeu force-pushed the feat/crewmate-optin-existing-agents branch from 8e16e47 to f50a573 Compare September 22, 2026 22:04
@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=2c6bd599dbfc

  • Escape triggers the permanent dismissal

fixed in f50a573: Escape now calls the step's local close only — nothing is recorded, and the step is offered again on the next visit. "Not now" is the sole exit that writes dashboard.crewmate_optin_done. Test: 'Escape closes the step for this visit and records nothing'.

@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=6e939fd5404d

  • "Not now" is permanent while promising later

rebutted: the label is the launch decision's copy (RFC docs/request-for-change/rfc-crewmates-launch.md §03: "Either action records the step as done; it never shows again"). What "later" keeps is the durable path, and the aside footnote states it on the same screen: "You can add or remove crewmates any time" (the roster's + creates one crewmate at a time). Escape, the accidental exit, no longer records (see 2c6bd599dbfc).

@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=6cec04b2f454

  • "Meet CrewMates" casing

rebutted: "CrewMates" is the launch's product name (RFC title "CrewMates launch"), used once, in the aside headline; "crewmate(s)" in every sibling string is the common noun. Same split as "Kiro Crew" vs "crew".

@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=a78a798a86fd

  • "3 chats" vs "used in 0 chats"

rebutted: both strings are the launch copy for this screen; the zero case is deliberately a sentence so a never-used agent reads as a fact about the agent rather than a count of nothing, which is what decides its unchecked default.

@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=d0076d666cde

  • Required rendered evidence missing

fixed: the description now carries seven screenshots (step dark/light, page after the add dark/light, zero selection, partial add refused, Not now refused) — real components rendered against a stubbed /api via Vite + Playwright, each frame asserting its state before capture. Backend paths are exercised by CI.

@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=e7068a5fb50f

  • crewmates === 0 gate ambushes a user who deletes their last crewmate

fixed in f50a573: the page records the step as over the first time it sees a roster with a crewmate and the gate still false, so a user who built crewmates through + is past the step for good. Test: 'a roster that already holds a crewmate records the step as over without opening it'.

@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=d7dc5a0d55ef

  • Item 9 — source field with zero consumers

fixed in f50a573: the field is deleted from the candidate row, the TypeScript type, the test and the spec shape line.

@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=8f1377a0688d

  • masked presentation text is persisted as the crewmate description

fixed in f50a573: a description the roster mask would rewrite is dropped server-side (empty string), never masked, so neither the secret nor the mask sentinel can reach POST /api/agents. Test: 'a credential-shaped description is dropped, not masked'.

@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=8f1377a0688d

  • opt-in load failures are silently treated as not due

fixed in f50a573: the read retries twice and goes stale after 30s, and a failed read records nothing — the gate is written only by an exit — so the step is offered on the next visit rather than lost. Spec (crew-mode.md) states the rule.

@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=96b17a80a753

  • the new dashboard feature is absent from the feature map

fixed in f50a573: docs/feature-map/README.md gains the 'Crewmate opt-in' row (page, handler, endpoints); the Feature Map Gate passes against origin/main.

@CrysisDeu

CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor Author

span=184a80bd1ee8

  • catch(() => undefined) discards a failed completion write

fixed in f50a573: the completion write is awaited; a refused record keeps the step open with every row locked and the reason shown, and "Not now" retries the record, landing on the most recent crewmate only once it succeeds. Test: 'a refused record after a finished add keeps the step open, locked, and Not now retries it'.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Sep 22, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=2f6082500a24

  • Startup claim incorrectly includes headless gateways — fixed in 8af15eb90d.

The feature-map row now reads "Once, when the dashboard gateway starts (start_dashboard; the headless --slack-only entrypoint has no dashboard and does not run it)". The spec section and the PR body say the same.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=e3d13945529f

  • bare open() on agent-writable paths: a FIFO hangs start_dashboard — fixed in 8af15eb90d.

Both path families now go through platform_compat.open_file_no_reparse(path, nonblocking=True) followed by stat.S_ISREG(os.fstat(fd).st_mode): a link is refused in the same operation as the open (no is_symlink() check-then-open window), a FIFO returns at once and is refused as not a regular file, and either reads as "no record". The session scan reads one line per file under a 64 KiB cap (_read_first_line_of_regular_file); the legacy activity file is streamed with readline(budget + 1) under eventlog.service.MAX_LEGACY_ACTIVITY_BYTES, the same budget the fold applies, and over budget is doubt for that candidate. Tests: FIFO at a session path and at the activity path return without hanging; a link at a session path is no record; an over-budget activity file keeps the crewmate.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=9b784b9cbeed

  • session-count-scaled scan awaited between the bind and KIROCREW_READY — fixed in 8af15eb90d.

Taken as suggested: _kick_crewmate_prune(state) runs the pass as a tracked background task (asyncio.create_task, state._background_tasks, the _kick_knowledge_orphan_reclaim shape) right after _arm_listener_guard, and start_dashboard awaits crewmate_prune_settled (bounded by the gate's 60 s; a timeout logs and restores anyway) immediately before the slot restores, so the ordering the pass needs survives without gating readiness. The event is set in the task's finally. Comments, spec and body no longer claim the awaited pass was off the boot path.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased: head 8af15eb on main de20586 (#13230, the shard-3 bind fix, in base), same migration. Folded in from the efe7097 round: doubt is per candidate (product-owner ruling) -- a session file that cannot be read is skipped as evidence, not held against everyone; session and legacy activity files are opened with open_file_no_reparse + S_ISREG (a FIFO or link is no record, never a hang) and the activity file is streamed under MAX_LEGACY_ACTIVITY_BYTES; the pass is kicked as a tracked background task after the bind and awaited before the slot restores, so it is off the readiness path; the feature-map row names dashboard startup only.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=eced8e08c12f

  • Description-only customization is deleted — fixed in 5a2b9ae37b.

_is_fresh_sync_shape now takes the bound spec's current description and requires the row's to equal it (the sync copied it from the spec); a row whose description differs — the owner rewrote it, or the spec moved on — is not the sync's row and is never a candidate. _synced_candidates returns each candidate with that spec description, and remove_never_chatted re-runs the same test against the same description inside the config lock, so a description edit that lands between the judgement and the delete is refused. Tests: test_a_row_whose_description_differs_from_the_spec_is_kept, test_a_description_edit_under_the_lock_refuses_the_delete, test_every_field_including_the_description_is_the_owners_signal. Spec and body updated.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased: head 5a2b9ae on main 1f9c252, same migration. Folded in from the 8af15eb round: a candidate's description must equal the bound spec's current description, checked at judgement and again inside the config lock; a description the owner rewrote is a customization and the row is kept.

@CrysisDeu

CrysisDeu commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor Author

span=d376e6f4be5f

  • prune scan still gates gateway readiness — fixed in 2c8636ea97.

The await crewmate_prune_settled.wait() before the slot restores is removed; nothing on the startup path awaits the pass any more, so start_dashboard returns and KIROCREW_READY prints independently of the session count. The ordering it enforced was not needed: a row the pass removes has, by the pass's own evidence rule, no DM binding and no session whose metadata names it (either would have kept it), so no restore can rebuild a slot for it. The request gate armed before the bind still holds every mutating request until the pass settles. Docstrings, spec and body say so.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased: head 2c8636e on main 6af4ede, same migration. Folded in from the 5a2b9ae round: the startup path no longer awaits the prune before the slot restores (a removed row has no DM binding and no session naming it, so no slot can be rebuilt for it); readiness is independent of the session count. Opus produced no verdict on 5a2b9ae (turn cap), so it re-runs on this head.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=d376e6f4be5f

  • Detached prune races non-HTTP session writers — fixed in fc4e1cb6b5.

Taken as suggested: every writer that binds an agent to a session without an HTTP request now waits for the pass. await_crewmate_prune_settled(state, before=…) (server.py) is awaited in GatewayOrchestrator.run right after the memory barrier — before the subagent pump, the deferred channel agent resume and cron dispatch start, and past KIROCREW_READY, so readiness does not wait — and in the standalone dashboard before its inline _resume_channel_agents(). Bounded by the request gate's 60 s (the pass sets the event in finally). _kick_crewmate_prune's docstring, the module docstring, the spec and the body name the writers and where each waits. Moving the pass before the bind instead was not taken: that is the boot path the earlier round moved it off.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased: head fc4e1cb on main 8c015af, same migration. Folded in from the 2c8636e round: the non-HTTP session writers (subagent pump, channel agent resume, cron) now await the prune before they start -- in the orchestrator after the memory barrier, past KIROCREW_READY -- so no crewmate can be bound while the pass is judging it; the standalone dashboard waits before its inline channel resume.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=eced8e08c12f

  • Stale spec snapshot can delete a concurrently customized crewmate — fixed in 6b3f77b9a3.

Taken as suggested: the delete now re-reads the bound spec inside the base config lock, under agents_spec_lock (nested in the order every other spec writer keeps, released before the overlay lock), through read_agent_spec_strict + spec_str — the same gate and normalisation discovery uses — and refuses unless the file's current description equals the one the row was judged with. A spec that vanished, stopped reading as a spec, or whose lock cannot be taken is doubt, and doubt refuses (no marker, re-judged next boot). _synced_candidates now records the spec filename per candidate (SyncedCandidate) so the delete re-reads that exact file. Tests: a spec description rewritten on disk between judgement and lock, and a spec unlinked in that window, both refuse the delete and leave the row. Module docstring, crew-mode.md and the body updated.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased: fc4e1cb6b5 → 6b3f77b9a3 onto main c0dbcd7abc. Same change plus: the delete re-reads the bound spec under agents_spec_lock inside the config lock (GPT F1 on fc4e1cb6b5); test_slack_gateway_overload_wiring stub gains dashboard_state = None (shard 7 red on fc4e1cb6b5 was this branch's). Windows (4) test_diag_procs timing red is not this diff's; named in the body.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=d376e6f4be5f

  • prune barrier admits evidence writers before migration settles — fixed in b484d96680.

Taken as suggested, both halves. (1) The roster read is not a pure read: GET /api/members runs MemberEventLogService.ensure per row — which folds a member's pre-log activity.jsonl into the event log and retires the file — and reconcile_member_config appends to that log; both are the places _activity_names_member reads, so a fold beside the pass could move a crewmate's only record out from under it. The gate now holds every request under /api/members whatever its method (_CREWMATE_PRUNE_GATE_HELD_PREFIXES), alongside every non-GET/HEAD/OPTIONS request on any path. (2) The writer-side wait no longer starts session writers beside a running pass. await_crewmate_prune_settled returns only once crewmate_prune_settled is set, which the pass does in finally after it has returned. Its 60 s budget now bounds how long the pass may keep deleting, not how long the writer waits: on timeout it sets DashboardState.crewmate_prune_abandon, which prune_synced_crewmates polls before each candidate and remove_never_chatted polls again inside the config lock right before the delete; the pass keeps whatever it has not judged (listed under doubted), writes its marker and returns, and only then does the writer start. The pass always returns: non-blocking opens, bounded lock acquires (platform_compat.file_lock raises on a stuck holder), either path ending in finally. A held HTTP request that outlives the budget is still answered 503 and writes nothing. Tests: the roster route and a route under it are held while a look-alike path is not; a pass told to stop keeps every unjudged row and writes the marker; a stop landing inside the lock leaves that row; the writer-side wait sets the stop signal past its budget and still does not return until the pass has. Module docstring, crew-mode.md and the body updated.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased: head 0029cc6 on main c5c014a (clean rebase, same diff plus one fix). Round b484d96: Backend Tests (3.12, 7) and (Windows) (7) were this PR's: the strict-reader call-site ratchet in test_agent_spec_hardened_reads.py did not list the prune's under-lock spec re-read. The site is now labelled crewmate_prune / dashboard and listed. Coverage Gate was downstream. Review lanes had not finished on that head and re-run on this one.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=eced8e08c12f

  • Best-effort activity loss deletes a used crewmate — fixed in 1e141a7547.

Taken as suggested: removal now needs a complete history. _session_first_line_agent keeps two kinds of session file apart. A file that READS AND PARSES is evidence whatever it says — the agent its metadata names, or nobody (an older build's first line that is not a metadata record, a record with no agent, a session that ran as another agent) — and never voids the pass. A file that CANNOT BE READ (open/stat/read failure, not UTF-8, empty, first line over the 64 KiB budget, first line not JSON) makes the history incomplete, and an incomplete history removes nothing: every candidate is kept under doubted with the reason, the files listed under unreadable_sessions, the marker still written so the pass does not re-run and hold writers every boot. The same fail-closed rule now covers the durable activity evidence: a member event log whose segment files are present but hold nothing the store will read (a zero-byte segment, which CrewLog.exists answers "absent" for and which is what a torn write leaves) is HistoryUnreadable for that candidate, alongside a log that will not load and a legacy activity file that cannot be read, parsed or streamed under budget. What remains: a dropped record_activity append leaves no trace on disk (members.py logs the loss and returns False), so a session that ran as a crewmate in a persistent slot, whose append was refused, and whose slot then switched agents with the DM thread never opened is undetectable from history; the row it would lose is the config.agents binding only — the agent file and every transcript stay, and New crewmate re-enrols it. Tests: an unparseable, non-UTF-8, empty, over-budget and unopenable session file each keep every candidate; a file naming another agent, a metadata record with no agent, and JSON that is not a record are evidence and remove as usual; a torn event log keeps that crewmate. Module docstring, crew-mode.md, the feature map and the body updated.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=eced8e08c12f

  • Process-local barrier races another gateway — fixed in 1e141a7547.

Taken as suggested: the barrier is now on disk. prune_synced_crewmates runs the whole pass — marker check, history scan, every delete, marker write — under an exclusive cross-process lock on <config dir>/crewmate_prune.lock (platform_compat.open_lock_file + file_lock, waited for up to PRUNE_LOCK_WAIT_S = 120 s, longer than the gateway's 60 s writer budget for the pass), and _write_marker creates the marker O_CREAT|O_EXCL with an fsync before close. A second gateway on the same data home therefore cannot run its pass beside the first: its pass waits on the lock — its own request barrier holds its writers meanwhile — then finds the marker and returns skipped_marker; one still waiting when the budget runs out removes nothing, writes no marker and reports skipped_locked with one WARNING, so the holder's marker settles later boots. The acquire is tried on its own (ExitStack) so only a refused lock reads as contention; a failure inside the pass propagates as itself. Tests (TestOneProcess): a pass waits for a holder (a second open file description, which is what a second process is to flock/msvcrt.locking) and then finds the holder's marker without writing over it; a holder that outlasts the wait leaves every row and no marker, and the next pass runs once it is released; the marker is created exclusively; an error inside the pass is not read as contention; the lock is released after a pass that returns and after one that raises. Module docstring, crew-mode.md, the feature map and the body updated.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=eced8e08c12f

  • Lock timeout releases writers while another process can still delete — fixed in fd63526a2c.

Taken as suggested: the contender's gates stay closed until it holds the lock or the shared marker appears. prune_synced_crewmates no longer gives up on PRUNE_LOCK: the acquire runs in a loop, and each PRUNE_LOCK_WAIT_S (120 s) that passes with another process holding the lock logs one WARNING ("still waiting ... session writers stay held until it finishes"), counts in the report's new lock_waits, and tries again. skipped_locked and the early return report are gone, so a pass never returns — and the gateway's crewmate_prune_settled event is never set — while the holder can still delete; the gateway's writers (_crewmate_prune_gate, await_crewmate_prune_settled) therefore stay held for the whole wait, and a session bound on this process cannot be absent from the holder's snapshot. Between tries the marker is checked without the lock: _write_marker is the pass's last step under the lock, so its presence means every delete is done, and a waiting pass may settle on it (skipped_marker) even before the holder lets go. A holder that dies releases flock/msvcrt.locking with its process, so the wait ends. Tests: a holder that outlasts several waits is waited for (lock_waits >= 2), then the contender runs the pass itself; a marker that appears while waiting ends the wait with nothing removed. Module docstring, PRUNE_LOCK_WAIT_S comment, crew-mode.md and the body updated.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased: 1e141a7547 -> fd63526a2c on main 2077450197. Folded in: GPT security BLOCK on 1e141a7547 (lock wait never gives up; skipped_locked removed), the three shard-7 test fixes that were mine (test_crewmate_prune_migration.py: a link at a session path is now "no record" via ELOOP, and two doubt-reason assertions read the file name the aggregate reason carries). Coverage Gate was downstream of shard 7.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=eced8e08c12f

  • Overlay lock is released before deletion commits — fixed in 31a80363de.

Taken as suggested: the overlay check and the base delete now run with both config locks held. remove_never_chatted no longer peeks the overlay through its own update_config_locked call (whose lock ended when the peek returned, before the base rename). Inside _mutate, after the row and spec re-checks, it enters _config_write_lock on config.local.json's sidecar (via _lock_target) on an ExitStack that outlives the callback, reads the overlay with read_config_for_update, and refuses if it names the row; the stack unwinds only after the outer update_config_locked has renamed the base file into place, so an overlay writer (config set --local agents.<name>…, the binding writers) that lands after the check finds the base row already gone, never a row it can still bind to. Lock order is base-then-overlay, the order agent_capabilities._write_bindings keeps. An overlay that cannot be locked or read (OSError, ConfigReadError) is doubt and refuses. Nothing is written to the overlay. Test added: a recording wrapper around _config_write_lock asserts that at the moment the overlay lock is released the base file on disk already lacks the row (one hold per delete). Module docstring, remove_never_chatted docstring, crew-mode.md and the body updated.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=eced8e08c12f

  • Legacy agent switches erase the only usage evidence — fixed forward in f7a88867bd, with a stated residual.

Usage evidence per session is now the union of every agent the metadata record names, not the one agent field: _session_agents_named (was _session_first_line_agent) returns agent plus execution_context.selection_name and execution_context.template_id; an interrupted switch can leave those apart (chat_persistence._restored_agent_name prefers the durable selection for that reason), and a name any of them holds is a name the session ran as. The "recover earlier usage from the rows" half of the suggested fix has nothing to read, verified against history.py and chat_handlers.py: ConversationLog.append writes a row as role, content, ts, tools, source thread/user and meta.mid — no agent — and api_chat_slot_agent rewrites the metadata line in place through update_metadata and appends no row, so a transcript holds no switch marker and no earlier agent. The module docstring, crew-mode.md and the body say so. The other half — treat every pre-activity-log session as incomplete — was ruled out by the maintainers (it makes the pass a no-op on exactly the installs it exists for, since every pre-#12224 install has such sessions). Residual, accepted and disclosed in the body's review-response table: a session from before the member activity log that ran as a synced crewmate, switched, and had every field overwritten leaves no trace; that crewmate's roster row is removed. What is lost is the config row only — the agent file stays, the shared store is untouched, and New crewmate re-enrols it in one click. Tests: test_every_agent_naming_field_of_the_record_counts (split record keeps both candidates), test_an_execution_context_of_the_wrong_shape_names_nobody.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased: 31a80363de -> f7a88867bd on main ad164a7f57. Same diff plus the GPT fix-forward above (metadata-record union). No inherited reds carried; Opus on the old head was still in progress and re-runs here.

An enrol-on-mount build enrolled every user-authored spec under ~/.kiro/agents
as a shared-memory crewmate through POST /api/agents/sync. A one-time startup
pass removes the ones never chatted with (config row only; the agent file and
any transcript stay), leaves the chatted ones exactly as they are, writes a
marker, and fails closed when chat history cannot be read.
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=eced8e08c12f

  • Spec lock ends before deletion commits — fixed forward in ffd011dd29.

The spec re-read under agents_spec_lock released the lock when the read returned, so a spec writer waiting on it could change the description between that check and the base delete. In remove_never_chatted._mutate the spec lock is now entered on the same ExitStack as the overlay's sidecar lock, inside the base config lock, overlay then spec — the order agent_capabilities._write_bindings and the template create keep — and both are released only after update_config_locked has renamed the base file into place. _current_spec_description is read under that held lock. A lock that cannot be taken refuses the delete, as before. Docstrings and crew-mode.md say so. Tests: test_the_spec_lock_is_held_until_the_base_delete_has_committed (the base file already lacks the row when the spec lock is released), test_the_locks_nest_base_then_overlay_then_spec.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

span=eced8e08c12f

  • Transcript scan races startup transcript deletion — fixed forward in ffd011dd29.

Two changes. (1) _agents_named_in_history no longer skips a file that is gone between the listing and the open: FileNotFoundError is now listed under unreadable_sessions like any file that cannot be read, the history is incomplete, and every candidate is kept (test test_a_session_that_vanishes_between_listing_and_open_keeps_every_candidate). (2) The one startup step that deletes a transcript, migrate_channel_transcripts — it merges an orphaned dashboard copy into its channel transcript and drops the copy's agent, the only record of the agent that dashboard surface ran as — is now serialized after the prune without putting the prune on the readiness path: it gains remove (default true); start_dashboard passes remove=state.crewmate_prune_settled.is_set(), so while the pass runs the merge still lands before the session restores but the copy stays with its first line intact, and _kick_deferred_transcript_removal (a tracked task) awaits the settled event and re-runs the migration with removal on; the re-merge is byte-identical and only the deletes are new. Nothing on the startup path waits for the pass. Tests: test_remove_false_merges_but_keeps_the_copy_and_its_first_line (and a removing pass then converges), test_deferred_transcript_removal_runs_only_after_the_pass_returns. Residual: a copy the migration removed on an earlier boot, under a build before this one, left no record of its agent; that is the pre-existing state of that install, not a window this pass opens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants