Skip to content

fix: carry slot state forward so superseded crew logs expire - #34264

Open
bolichen97 wants to merge 1 commit into
mainfrom
fix/crew-log-units-never-collected
Open

bolichen97 wants to merge 1 commit into
mainfrom
fix/crew-log-units-never-collected

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A crew log that a chat, crewmate or conductor has moved past expires after session.archive_retention_days like any other finished session, and everything that slot shows reads the same afterwards.

~/.kiro/crew/crew-log/sessions grows whatever the retention setting says. A reset, a discarded conversation, a rewind and a restart whose resume fails each open a new crew log for the same slot and leave the old one behind, closed reset or not closed at all. The sweep collects a crew log only when its newest close is destroyed, so none of those ever qualify: on the install in the issue, not one of about 465 crew logs (about 54 MB) could be collected.

Why it matters

Every long-lived chat, crewmate and conductor keeps every crew log it ever ran under, on every install, so disk use grows without bound and each read that folds a slot's crew logs gets slower as they pile up. The retention setting the user chose does nothing for them.

Not a goal

  • Expiring a crew log that no later crew log of its slot follows (a closed tab, an archived chat). Those are kept as before; whether they expire is a separate decision.
  • Carrying the context panel's turn history, the root card's tool tally, a board's count of left-out crew logs, or a worker's pre-board reports for the operator's rebuild. These are the accepted costs in the decision record.
  • Changing when the sweep runs (at most once an hour, when a transcript archive is written).
  • The cost of a slot's warm reads right after a reset (gateway: move the top GIL users into child processes #33697).

What changed (motivation → approach → change)

The old crew logs of a chat, crewmate or conductor that has moved to a new session expire with session.archive_retention_days, and what that chat, crewmate or conductor shows stays the same.

The sweep collected only a crew log closed destroyed, which a reset or a restart never writes. Collecting the others on age alone would delete live state: every slot fold (the state a chat, crewmate or conductor keeps across its sessions) reads every crew log that slot ran under.

So the slot's newest crew log carries the slot's state forward first, and only then is each older one closed superseded, which the sweep collects. The maintainer chose this over letting old slot state expire or keeping every crew log of a live slot: the decision record, indexed in docs/decisions/README.md.

Step What happens
Carry When session/opened starts a slot's next crew log, the emitter (crew_log/emit_session.py) folds every slot fold, the work tree the slot roots and the slot's status, each in its reader's own order, and writes each non-empty state into the new crew log as a session/carried entry (entry_types_lifecycle.py), in one group commit. A crew log the slot has already moved past when its fold runs or its carry is written carries nothing and closes nothing; the slot's later crew log carries it. A crew log the fold read that gains an entry before the write (a turn that ended meanwhile) makes the fold run again, at most twice, and then stand down. The session-tree entries the carry copies from an older crew log are marked carried: true (tree_carry.py), so the per-crew-log child count of otel_session_tree does not count a creator's children again under its new crew log. A tree carry that would leave a spawn past its reader's per-crew-log bound carries nothing and closes nothing, so the older crew logs keep every creating edge.
Read A fold over a slot that meets a carry for its own fold and slot replaces its state with it (projection_kernel.py, projection_readers.py), so the older crew logs stop changing any answer. The cold slot folds read each crew log from its first record through one helper, _from_record_one, so the whole-unit read call sites that test_crew_log_read_budget_gaps.py pins stay one. A read of one crew log is unchanged.
Close Each older crew log gets session/closed {superseded} unless a live turn still writes it, it gained an entry after the carry read it, it is already closed for good, or the gateway's session-key probe finds a key that maps it or cannot read the map. The probe lives in slack/gateway_runtime/admission.py and is registered at boot (service_boot.py).
Sweep crew_log/store_retention.py collects superseded beside destroyed, and still reads nothing outside the crew log tree.
Order A carrying crew log stays after the crew logs it carries in every later read and carry, so no event is read twice. A crew log an order file does not name is placed by the slot's succession, after the recorded crew logs before it (session_ledger_control.py, which the Issue Radar crew store crew_store_log.py uses too), and the other slot folds read the succession instead of the header clock, which a backward clock step inverts (projection_readers.py, projection_workers.py). The new crew log is also published in the ledger, work and panel order files.
Late writes A write that names a crew log its slot has moved past (a request resolved before the reset that arrives after the carry) is decided on the slot's writer lane, where the successor's carry is committed too. That covers the ledgers (emit_surfaces._record_on_slot_lane, used by session_ledger.record_update and the Issue Radar ledger's _append), a worker report's board copy (on_work_recorded), the board relay (work_relay._queue_run), the crew panel (on_panel_published) every dashboard write (_append_dashboard: the agentic and custom values, their refusals and failed runs, the stale wake, and the crewmate feed's goal and waiting snapshots), and the session tree's adoption, release and spawn writes (on_session_adopted, on_session_released, on_session_spawned, through emit_session._tree_lands_in, under late_writes.TREE_KEY), since the tree ranks the newer crew log's copied decision first. Each writer job asks crew_log/late_writes.hand_to_carrier, hands itself to the slot's newest crew log when a later one carried a fold the entry feeds or the named one is closed superseded (a carry an earlier process wrote is read off the later crew log, once per crew log per process, so a gateway restart in between changes nothing), appends, and publishes the unit order for the crew log the entry landed in; a ledger job then settles a Receipt with that crew log and seq. So the carry lands wholly before or after the write, and no clock or sampled seq decides where the entry goes or whether it landed. The handoff (session_handoff._move_tree_edge) reads its target's crew log again after its waits, and moves no edge when the target's slot was replaced in between.
Joins The two openers, the dashboard turn (dashboard/chat_turn/turn_admission.py) and the channel dispatcher (messaging/dispatch_crew_log.py), record the session key the slot's turns run under. The ledger carry (session_ledger.py) joins the crew logs headed with a caller's own spelling of that key, including a cron job's tab (cron_tab_name, moved into session_surface.py) and any spelling a ledger write or a live read recorded in the slot's bounded aliases control file (session_ledger_layout.py). crew_log/retention_joins.py keeps a superseded crew log another slot still reads through that join until that slot carries its own ledger. When an older build's crew log this build resumed has grown past the tail window since its last opening, the sweep searches its segments, newest first, for that opening, once per pass and only when a candidate's answer depends on it.

A crew log whose opening states no session key carries nothing and closes nothing, and so does a process with no probe registered (a CLI run, a test), so either keeps every crew log it kept before. The crew log specs (crew-log-core.md, crew-log-emitter.md, crew-log-projection.md, session-types.md), session-control.md, session-work-ledger.md, learn-cron-dashboard.md and the Issue Radar crew_ledger_spec.md describe the carry, the close and the costs.

Backwards compatibility

Compatible: nothing main accepts is rejected, renamed or removed; a crew log is closed superseded only after a later crew log of its slot holds every carried fold's state, and a crew log whose slot carries nothing is kept as before.

Reader: src/kiro_crew/session_ledger.py:read_state -- chat -- test_crew_log_slot_carry.py::test_retention_then_collects_the_replaced_unit_and_the_slot_keeps_its_state
Reader: src/kiro_crew/session_ledger.py:read_state -- cron -- test_crew_log_slot_carry.py::test_a_spelling_an_older_build_wrote_under_keeps_reading_across_the_first_carry
Reader: src/kiro_crew/session_ledger.py:crew_log_units -- chat -- a channel conversation's raw key: test_crew_log_slot_carry.py::test_a_channel_keys_raw_spelling_keeps_reading_across_the_carry_with_no_live_session
Reader: src/kiro_crew/crew_log/retention_joins.py:LedgerJoins -- chat -- a workflow's fallback tab reading a gone chat's records: test_crew_log_retention_joins.py::test_the_sweep_keeps_the_chats_older_unit_until_the_tabs_first_carry_lands
Reader: src/kiro_crew/apps/builtins/issue_radar/backend/crew_store_log.py:crew_log_units -- app -- test_issue_radar_crew_store_unit_order.py::test_a_crews_newest_unit_folds_last_where_its_slot_carry_lands
Reader: src/kiro_crew/crew_log/projection_readers.py:read_tree_projection -- crew page -- test_crew_log_worktree_read.py::test_the_tree_reads_the_same_after_the_roots_older_unit_is_collected
Reader: src/kiro_crew/crew_log/projection_readers.py:read_slot_projection -- crew page -- every slot fold (crew work items and skip index, dashboard values, refused writes, per-task costs): test_crew_log_slot_carry.py::test_a_carry_makes_the_slot_fold_independent_of_the_units_before_it
Reader: src/kiro_crew/dashboard/session_control_parts/session_status.py:_gone_slot_lifecycles -- subagent -- a gone worker still reads lost or closed: test_crew_log_slot_carry.py::test_a_gone_row_reads_the_same_once_the_older_unit_is_collected
Reader: src/kiro_crew/work_ledger.py:rebuild_from_projection -- crew page -- still rebuilds; a worker's pre-board reports in a collected worker crew log are left out, an accepted cost in the decision record (reports since then sit in the board's carried crew logs)
Reader: src/kiro_crew/crew_log/projection_workstreams.py:_own_board_units -- crew page -- still reads the newest 512; the count it left out reads zero once those crew logs are collected, an accepted cost in the decision record: test_crew_log_worktree_read.py::test_the_checkpoint_and_the_read_count_the_same_units_left_out_after_a_carry
Reader: src/kiro_crew/dashboard/handlers/usage_context.py:_usage_read -- chat -- still serves the panel; it covers only the turns of the crew logs still kept, an accepted cost in the decision record
Reader: src/kiro_crew/dashboard/card_lifecycle.py:_read_card_tools -- chat -- the tally loses a collected crew log's calls and then reads its list as not whole, so the card never says a tool was not called; an accepted cost in the decision record
Reader: src/kiro_crew/dashboard/cron_inject.py:cron_tab_name -- cron -- cron_tab_name moved from chat_utils_slots.py to session_surface.py and is imported back there, the same function: test_cron_tab_workspace.py
Reader: src/kiro_crew/slack/gateway_runtime/cron_delivery.py:cron_tab_name -- cron -- imports it through cron_inject, the same function: test_cron_tab_workspace.py
Reader: src/kiro_crew/dashboard/chat_api/resume.py:cron_tab_name -- chat -- imports it through chat_utils, which re-exports the same function
Reader: src/kiro_crew/crew_log/session_tree.py:latest_edges -- crew page -- a decision or spawn filed after a reset now lands in the slot's newest crew log: test_crew_log_carry_late_tree_writes.py::test_a_decision_filed_under_the_older_unit_after_the_tree_carry_is_the_tree
Reader: src/kiro_crew/dashboard/session_control_parts/session_handoff.py:handoff_target -- crew page -- tree_moved is false, with the handoff standing, when the target's slot was replaced while the move waited: test_session_handoff.py::test_a_target_replaced_while_the_handoff_waits_gets_no_tree_edge

Tests

  • test/test_crew_log_slot_carry.py (new): the carry, read, close and sweep end to end on a real store and emitter. A carry makes a slot fold independent of the crew logs before it, matches the warm and cold reads, and leaves other slots and folds alone; the replaced crew log is closed superseded and then collected while every fold reads the same; a key that still maps the old id, an unreadable map, a live turn, a missing probe or an opening with no session key each carry nothing and close nothing; a gone row reads lost the same once the older crew log is gone; every caller spelling (a cron:<id> key, a channel's raw key, a key an older build wrote under) keeps reading across the carry. A slot that moves on to a third crew log before the second one's carry folds, or while it folds, is carried by the third alone, and every ledger event reads once before and after the sweep.
  • test/test_crew_log_retention_joins.py (new): the sweep keeps a superseded crew log another slot reads through the ledger join until that slot's first carry lands, collects one nothing joins or one with no ledger record, keeps the crew log when the join cannot be read, and reads nothing outside the crew log tree.
  • test/test_crew_log_retention.py: a superseded close authorizes collection past the window and not inside it, and closed_for_good agrees with the sweep.
  • test/test_crew_log_worktree_read.py: the work tree reads the same after its root's older crew log is collected, and the checkpoint and the read count the same left-out crew logs after a carry.
  • test/test_issue_radar_crew_store_unit_order.py: the radar crew store folds a crew's newest crew log last where its carry lands, keeps a carried crew log it never recorded into after what it carries once the crew moves on, and holds each event once after a backward clock step and a second carry.
  • test/test_crew_log_slot_carry.py also pins the two review findings of c406128. A turn an older crew log ends, or a record it or a joined spelling's crew log gains, while the carry folds is not lost: the fold runs again, at most twice, and a crew log that gains an entry after the carry read it is not closed. A later carry after a backward clock step reads each turn once, every carried fold reads its crew logs in succession rather than by header clock, a crew log the order file does not name is placed by succession, and a carry its reader would not fold last stands down.
  • test/test_crew_log_workers_fold.py, test/test_work_unit_order_reset.py: the crewmate's workers fold reads its crew logs in succession, and a reset crew log that never recorded keeps its place, and the read stays warm, once a newer one opens.
  • test/test_turn_harness.py, test/test_dispatch_crew_log.py: the dashboard turn and the channel dispatcher state the session key in the opening.
  • test/test_crew_log_types.py, test/test_entry_types_facade.py, test/test_docs_crew_log_reference.py: session/carried is declared, documented and in the reference tables.
  • test/test_slack_gateway_composition_contract.py: the gateway's admission surface lists _register_session_key_probe, the boot hook that registers the session-key probe.
  • Round 17 review repairs: test/test_crew_log_tree_decision_retention.py pins that a carried session-tree decision follows previous_sid succession after a backward clock step, and that a sweep keeps the succession link of a unit that stays; test/test_crew_log_slot_carry.py pins that a joined spelling's units carry in succession. Each fails with its production hunk reverted and passes with it (run on the DevSpace lane with pytest -n0). The member scenario repro.sh fix for crew-log-units-never-collected exits 0 at this head.
  • Rebased onto main after review; the rebase patch unchanged apart from the repairs above.
  • Round 20 review repairs (head 8a4a567, on main bb7048c): test/test_crew_log_slot_carry_overlap.py (new) drives a ledger record that overlaps a slot reset through the real writer and emitter, each interleaving forced by a hook, no sleeps. An update the carry already read is not filed twice on a tied stamp; an update after the carry is kept when the clock steps back; a concurrent record's entry does not answer for this one; the unit order is published on the slot's lane; a carry an earlier process wrote is found from the superseded close; a resumed crew log's join holds past the tail window; a reset adds no child to the new crew log's session-tree count.
  • Reproduce, then verify, on DevSpace lane ds-kc-fix-d-lxybdhjh (Linux, Python 3.12), PYTHONPATH=$PWD/src pytest -n0 test/test_crew_log_slot_carry_overlap.py: at this head 7 passed, exit 0; with the production hunks reverted (git apply -R of the src/ diff, tests kept) 6 failed and 1 passed, exit 1 (the six are the review findings; the seventh guards the new cross-process path and passes on the old code too). One mutation per guard, each caught: no note_carried (1 failed), no superseded close check (1), the lane job never redirects (2), the otel fold counts carried spawns (1), unsettled crew logs never searched (1), the tree carry does not mark (1), the order published after the flush instead of on the lane (3).
  • The three CI reds of e4c88d0 at this head, pytest -n0 test/test_crew_log_read_budget_gaps.py::test_callers_of_the_whole_unit_reads_only_go_down test/test_issue_radar_facades.py -k crew_store test/test_facade_mirror_guards.py::test_every_mirrored_facade_manifest_names_what_it_forwards: 9 passed, exit 0.
  • Member scenario repro.sh <base|fix> for crew-log-units-never-collected on the same lane: base (main bb7048c) exit 1, all 3 replaced crew logs survive the sweep; fix (this head) exit 0, all 3 collected and the slot's current crew log kept.
  • flake8, isort --check-only, black --check on the 24 changed Python files and mypy --platform linux on the 13 changed source files, on the lane: each exit 0. The full suites and gates run in CI.

Both new test files pin the data home through the isolation floor's own MonkeyPatch (_floor_monkeypatch), so a test's monkeypatch.undo() cannot lift it.

Union run on this head's rebased commit cc32950de2 (DevSpace, Linux, Python 3.12) of the 13 changed and added test files plus test/test_gateway_runtime_bindings.py, test/test_cron_tab_workspace.py (which imports the moved cron_tab_name from its old path), test/test_crew_log_retention.py and test/test_crew_log_worktree_read.py: 1472 passed, 51 skipped (all 51 are test_docs_crew_log_reference.py examples that have no crew_key), exit 0. A related set of 174 test files (every file that names a changed crew log or ledger owner, plus test_crew_log_*, test_session_ledger*, test_work_* and the facade mirror tests) on the repair before the rebase: 6974 passed, 52 skipped, 2 xfailed, exit 0. The full backend and frontend suites run in CI.

Rebased onto main after review; patch unchanged: git range-diff 29a09dcca2..9f2fd9149b c77993833a..cc32950de2 prints 1: 9f2fd9149b = 1: cc32950de2, main's 43 new commits touch none of the crew log, ledger or radar modules, and the run above is on the rebased commit. Rebased once more onto 096f6d9 (a8e5320f0e = daa4466e0c; five more main commits, the one shared file is docs/decisions/README.md, one row apart): the PR's test files with test/test_gateway_runtime_bindings.py, test/test_cron_tab_workspace.py and test/test_check_decisions_history_selftest.py 1466 passed, 51 skipped; check_decisions_history.py against main and scripts/docs-lint.sh, exit 0. Between those two rebases one import in test/test_crew_log_slot_carry.py moved into its test body, so the file still collects on a tree without the new fold set and prove.py can read its assertions: that file 52 passed, and test_every_carried_fold_reads_its_units_in_succession_not_the_header_clock 20 of 20 green at -n0.

Repro of #23082 on the group commit, real store and emitter in a scratch data home: a slot's first crew log records a ledger entry, closes reset, and the slot opens its next crew log, then sweep_expired(0) runs. On origin/main 73e3275 the old crew log stays closed reset, the sweep removes nothing ((0, 0)) and the crew log stays on disk. On the group commit it is closed superseded, the sweep removes it ((1, 0)), and the slot's ledger reads the same goal and phase before and after.

Repro of a slot that moves twice before its first carry lands, on the group commit: with the newest-unit check taken out of the emitter (_still_slots_newest in crew_log/emit_session.py), both new tests fail, because the second crew log writes a ledger carry the third one already holds. Taking out only the check before the fold fails test_a_carry_whose_slot_moved_on_before_it_folded_stands_down (both later crew logs carry, ['u2', 'u3'] against ['u3']), and taking out only the check before the write fails test_a_carry_whose_slot_moved_on_while_it_folded_writes_nothing. With both checks in, both pass.

Repro of the two GPT findings on c406128 (span=2117439f419d, span=7db6df3cde56), DevSpace. At the fix, pytest -n0 of test_a_turn_the_older_unit_ends_while_the_carry_folds_is_not_lost, test_a_later_carry_after_a_backward_clock_step_reads_each_turn_once and test_issue_radar_crew_store_unit_order.py::test_a_later_carry_after_a_backward_clock_step_holds_each_event_once: 3 passed, exit 0. With the repair's production hunks reverted (the PR head before it, 59b5b730b9, with this head's tests): 3 failed, exit 1. The turn that ended during the fold is lost (turns_completed 0, not 1), the second carry counts u1's turn twice (2, not 1), and the radar reads two events twice. Across the four changed test files 29 failed and 133 passed reverted, against 542 passed at the fix with the gateway composition and bindings tests (exit 0).

User scenario, the real gateway (python -m kiro_crew gateway --test-mode on a scratch home with session.archive_retention_days=0, driven over its HTTP API with the packaged fake ACP backend): one chat slot is reloaded, reset and restarted, and a rewind on another slot runs the sweep. On main 29a09dc: exit 1, all 3 replaced crew logs survive the sweep. On this fix: exit 0, all 3 are collected and the slot's current crew log is kept.

Gates on this fix (DevSpace, each exit 0): scripts/check_black_formatting.py, scripts/check_subprocess_encoding.py, isort --check-only, flake8 on the 41 changed Python files, mypy --platform linux src/kiro_crew (also on the rebased commit), check_comment_history.py, check_brand_name.py, check_harness_parity.py, check_file_line_cap.py, scripts/docs-lint.sh, and pytest -n0 test/test_flake_pattern_ratchet.py (68 passed). Semgrep is not installed on that host; CI's SAST job runs it.

Proof: PROVEN test/test_crew_log_retention.py::test_a_superseded_close_authorizes_collection, test/test_crew_log_types.py::test_every_type_written_today_is_declared_and_nothing_else_is, test/test_crew_log_types.py::test_the_canonical_examples_cover_exactly_the_declared_types, test/test_crew_log_types.py::test_an_undeclared_field_is_refused_rather_than_written[session/carried], test/test_crew_log_workers_fold.py::test_a_crewmates_units_follow_succession_not_the_header_clock (head daa4466)

Determinism, on c406128: the 53 new and changed test functions (61 cases) ran 20 times at -n0 and 10 times at -n 4 --dist loadgroup --max-worker-restart=2 beside test_crew_log_emit.py and test_crew_log_checkpoint.py (178 passed each run), all green, and the ten changed test files ran green in a shuffled order (seed 7735, 1051 ids: 1000 passed, 51 skipped). For this head's repair: its 15 new and changed test functions (31 cases) ran 20 times at -n0 (31 passed each run), its four changed test files ran 10 times at -n 4 --dist loadgroup --max-worker-restart=2 (162 passed each run), and those four files ran green in a shuffled order (seed 23784, 162 ids, 162 passed).

Review repair on head bf7e6a4 (rebased onto main 54ccb78, one conflict in crew_log/emit_runtime.py on_session_closed resolved by keeping main's _on_settled_with_release and this PR's _forget_session; the radar carry repair changes the patch, so its lines were reviewed): the radar slot carry now folds a crew's units in the order the crew recorded them (crew_store_log.carry_units, reached from projection_readers._slot_units_for_fold), and carries nothing when that order cannot be found, so older units stay. DevSpace, Linux: pytest -n0 test/test_issue_radar_crew_store_unit_order.py 44 passed, exit 0; with the radar branch of _slot_units_for_fold disabled 2 failed (the new repro and test_a_carry_whose_recorded_order_cannot_be_found_carries_nothing), exit 1; ten scoped crew log files with -n 4 --dist loadgroup --max-worker-restart=2 508 passed, exit 0; black, isort, flake8 on the four changed Python files and mypy --platform linux src/kiro_crew exit 0; check_comment_history.py, check_brand_name.py, check_file_line_cap.py exit 0. local-gate.py was cut at 540 s at 89% with no failure shown; CI runs the full suites.

Disposition on c6962d3: the Backend Tests reds (test_core_issue_radar_decoupling, test_facade_mirror_guards) were this PR's own and failed on the head but not on its merge-base; the radar carry now asks a generic crew_log/unit_order_seam that the app fills from routes.register_routes, the facade manifests name the new forwarded names, and retention keeps the previous_sid links between a kept unit and the unit that carries it (new test fails without the change). Changed tests, black, isort, flake8, mypy --platform linux, comment-history, brand and the flake ratchet ran green on a DevSpace.

Round 21 review repairs (GPT 6.1 findings span=2117439f419d and span=182bfe589b57), verified at 4e948f4 on main 4dd046d. Head 259e89d on main 758e63c has the same src/ and one changed test line (the CI bullet below):

  • F1, resolution: narrowed -- every writer that publishes a carried fold's unit order now lands through the one slot-lane hand-over round 20 added (late_writes.hand_to_carrier), no new mechanism. The board copy of a worker report (emit_surfaces.on_work_recorded), the board relay (work_relay._queue_run) and the crew panel (on_panel_published) now pick their unit inside the writer job, on the slot's lane, the same way the ledgers do. The carried folds an entry type feeds come from the fold registry (late_writes.folds_fed_by). test/test_crew_log_carry_late_board_writes.py (new) puts each writer through a real reset and carry twice: once with the older unit closed superseded, once with a key still mapping it. It also checks that a waiter that gave up gets nothing landed by the job that was handed on.
  • F2, resolution: narrowed -- a tree carry that a reader's per-unit spawn bound would cut now carries and closes nothing, so the older units stay and keep every edge they held. In tree_carry.tree_carries, an older unit read short (hit_cap) answers None, and so do held plus carried spawns past SPAWNED_PER_UNIT_CAP. The slice that dropped the oldest spawns is gone. Pinned by test/test_crew_log_tree_carry_cap.py (new).
  • Reproduced, then verified, on DevSpace lane ds-kc-fix-d-lxybdhjh (Linux, Python 3.12) with PYTHONPATH=$PWD/src pytest -n0 test/test_crew_log_carry_late_board_writes.py test/test_crew_log_tree_carry_cap.py. At this head: 12 passed, exit 0. With this round's production hunks reverted (git apply -R of the src/ diff, tests kept): 12 failed, exit 1. The failures: a report reads 3 entries where 2 are expected, a relayed copy 2 where 1 is expected, the panel counts 3 publishes where 2 are expected, nothing is handed to the carrier, "176 children are carried nowhere, yet ['cap-c1', 'cap-c2'] were closed for retention", and after the sweep and a restart spawned_creator('kid-0000') is ''. One mutation per guard, each caught: no hit_cap stand-down (1 failed), no held-plus-carried bound (2 failed), any job releasing the work waiter, the handed-on one included (4 failed).
  • test/test_crew_log_slot_carry_overlap.py, which covers the round 20 hand-over this round moved into late_writes.hand_to_carrier: 7 passed at this head, and 7 passed with the hunks reverted.
  • The two shared CI reds of 8a4a567 are test_wait_discipline_ratchet.py::test_the_scan_matches_known_sites and test_updates_handlers_facade.py::test_every_name_served_from_outside_the_family_is_visible_to_type_checkers. Both failed at 8a4a567 (2 failed) and both pass at this head (2 passed, exit 0). The fix came from main.
  • Member scenario repro.sh <base|fix> for crew-log-units-never-collected, on the same lane: base (main 4dd046d) exit 1, all 3 replaced crew logs survive the sweep; fix (4e948f4, the same src/ as this head) exit 0, all 3 collected and the slot's current crew log kept. The run used a lane-owned copy of the script (~/kc-build/wt-babysit/r21-repro/repro.sh) whose only difference is its ROOT line, because another lane's cleanup removed worktrees under ~/kc-repro3 (a first attempt there was inconclusive, exit 2, with no worktree).
  • On the lane, each with exit 0: flake8, isort --check-only and black --check on the 7 changed Python files; mypy --platform linux on the 5 changed source files; check_comment_history.py, check_brand_name.py and check_file_line_cap.py. The full suites and gates run in CI.
  • CI on 4e948f4: test_facade_mirror_guards.py::test_no_test_patches_a_shared_name_on_an_owner_directly failed on Linux shard 7 and Windows shard 2, because the waiter test patched emit_runtime._hand_over on one owner. It now patches through the emit facade. On the lane that test failed at 4e948f4 (1 failed, exit 1) and passes with the fix (1 passed, exit 0). The other red, test_cloud_launch_customize.py::TestAmiArchitectureCheck::test_matching_architecture_proceeds_to_deploy, also failed on a clean main 4dd046d worktree (1 failed, exit 1), and main's test(cloud): let the AMI check's fake answer a failed launch's stack lookup #35746 fixes it. At 259e89d on the lane, the two new files, test_crew_log_slot_carry_overlap.py, that guard, TestAmiArchitectureCheck and the two shared-cause tests ran together: 25 passed, exit 0.
  • Rebased onto main after review (758e63c, which carries test(cloud): let the AMI check's fake answer a failed launch's stack lookup #35746; none of main's new commits touches a crew log file). Apart from the repairs above, the patch is unchanged (git range-diff prints 1: b9912f55aa = 1: 259e89da08).

Round 22 review repair (GPT 6.1 finding span=2117439f419d, "Late dashboard writes are acknowledged but lost behind the carry"), head 533d7b6 on main cf4578c:

  • Every dashboard write now lands through the same slot-lane hand-over as the ledgers, the board and the panel (late_writes.hand_to_carrier). emit_surfaces._append_dashboard is the one helper for the agentic and custom values, their refusals and failed runs, the stale wake, and the crewmate feed's goal and waiting snapshots (on_goal_recorded and on_waiting_changed, which went through _write before). The feed calls from the serving loop, where the emitter reads no file, so for a unit the writer has no slot key for (a closed one) its job reads the header on the writer and queues itself again on the slot's lane (late_writes.requeue_on_slot_lane).
  • The answer is given once (emit_surfaces._Landing): by the job that appended or found no log, by a refusal, or by the first job's drop or rejection at the writer's ceiling, never by a job that handed the entry on. A waiter that gives up abandons the entry, so a handed-on job that runs later appends nothing, and the feed is told the same answer through on_settled. An entry lost after it was handed on (its attempt budget spent in the newer unit) tells nothing, the way a ledger receipt stays unsettled: a waiter's timeout answers false.
  • test/test_crew_log_carry_late_dashboard_writes.py (new) puts each dashboard writer through a real reset and carry twice, once with the older unit closed superseded and once with a key still mapping it. It also checks that the answer comes once (handed on, retried, refused, rejected at the ceiling), that the feed reads no header on the serving loop, that a waiter that gave up gets nothing landed, and, by reading emit_surfaces' syntax, that every append of an entry type that feeds a slot-keyed carried fold goes through the hand-over.
  • Reproduced, then verified, on DevSpace lane ds-kc-fix-e-ld3tr722 (Linux, Python 3.12) with PYTHONPATH=$PWD/src pytest -n0 test/test_crew_log_carry_late_dashboard_writes.py. Before the repair, at 259e89d and at its rebase 79cb3d63e5, the five writer tests failed 10 of 10: custom showed first where second is expected, agentic 1 where 2, the mistake book 1 refusal where 2, goal ship v1 where ship v2, waiting approve the plan where review the PR, and every write had answered true. At this head: 31 passed, exit 0. With the production hunks reverted (emit_surfaces.py, late_writes.py and emit.py from 79cb3d63e5, tests kept): 29 failed and 2 passed, exit 1. The two that pass both ways are the ceiling-rejection checks, which _write already covered.
  • Member scenario repro.sh fix for crew-log-units-never-collected at this head, on the same lane with a lane-owned copy of the script (only its ROOT line differs): exit 0, all 3 replaced crew logs collected and the slot's current crew log kept.
  • Related files at this head on the lane, -n 4 --dist loadgroup --max-worker-restart=2: the new file, test_emit_facade.py, test_facade_mirror_guards.py, test_crewmate_feed.py, test_crewmate_feed_cold_store.py, test_crew_log_types.py, test_crew_log_carry_late_board_writes.py, test_crew_log_fold_eager_memory.py, test_crew_log_slot_carry_overlap.py, test_crew_log_slot_carry.py, test_agent_panel_routes_dashboard.py, test_agent_panel_routes_record.py, test_agent_panel_routes_record_app.py, test_slack_gateway_composition_contract.py and test_gateway_runtime_bindings.py: 1075 passed, exit 0. An earlier run caught two facade guard reds this repair caused (_Landing was not bound by the emit facade, and a test patched _queue_tail_repair on its owner); both are fixed at this head.
  • On the lane, each with exit 0: black --check, isort --check-only and flake8 on the 4 changed Python files; mypy --platform linux on the 3 changed source files; check_comment_history.py, check_brand_name.py and check_file_line_cap.py; pytest -n0 test/test_flake_pattern_ratchet.py (68 passed). The full suites and gates run in CI.
  • Rebased onto main after review (cf4578c). Main's 51 new commits touch two of this PR's files, service_boot.py and learn-cron-dashboard.md, and merge clean. Apart from the repair above, the patch is unchanged (git range-diff prints 1: 71865a40eb = 1: 533d7b6464).

Round 23 review repairs (buluoray's review of 533d7b6, and GPT 6.1 finding F1 at src/kiro_crew/crew_log/tree_carry.py:92, "Carried tree decisions override a later acknowledged handoff"), head b7e8724 on main 374b7d7:

  • buluoray, a late write after a restart: the writer now finds a carry an earlier process wrote. late_writes.carrier_after asks, cheapest first: is the named crew log the slot's newest (no log opened); did this process note a later carry or read one before; is the named crew log closed superseded; and then it reads each later crew log's own log once per process for a carry (_read_carries: a session/carried for the slot naming one of the writer's folds, or for the tree any carry or a tree entry marked carried). The read skips records without a carry marker unparsed and stops at the end of the carry's one group commit. What it finds is noted, keyed by data home, slot and crew log. A read that fails proves nothing, so that write goes where it always went and the next write reads again. test/test_crew_log_carry_across_restart.py (new) calls emit.reset_caches() between the carry and the late write, the way the review reproduced it: a ledger update and a dashboard value both land in the carrier, a dashboard waiter that gave up gets nothing landed, a later crew log is read once, and the newest opens no log.
  • GPT 6.1 F1, resolution: narrowed -- tree mutations join the one slot-lane hand-over the ledgers, board, panel and dashboard already use (late_writes.hand_to_carrier), no new mechanism. Adoption, release and spawn writes decide their crew log inside the writer job, on the slot's lane (emit_session._tree_lands_in), and hand themselves to the slot's newest crew log once a carry landed there. Every carry notes the tree (late_writes.TREE_KEY), whatever it copied of it, because a decision filed under an older crew log loses to any edge the newer one holds. on_settled is told once, by the job that appended, found no log or was refused, never by a job that handed the entry on (emit_tree._TreeSettle.handed_on). The handoff (session_handoff._move_tree_edge) captures the target's slot object before its first wait, re-reads the target's id after the last one with nothing suspending before the append is queued, and moves no edge when the slot was replaced. test/test_crew_log_carry_late_tree_writes.py (new) and three tests in test/test_session_handoff.py cover an adoption and a release filed after the tree carry (older crew log closed superseded, or still mapped by a key), the same after a restart, a carrier that holds an edge of its own, a refused handed-on decision told false once, a spawn filed under the creator's closed crew log that keeps its child through retention, and a handoff whose target resets or is replaced while it waits.
  • Human review item 3: the decision entry's Evidence bullet now links this pull request.
  • Reproduced, then verified, on DevSpace lane ds-kc-fix-c-6kwhp1e3 (Linux, Python 3.12), PYTHONPATH=$PWD/src pytest -n0 test/test_crew_log_carry_across_restart.py test/test_crew_log_carry_late_tree_writes.py plus the three new test/test_session_handoff.py tests. Before the repair, at the rebased old head e729c4cf1e and at 533d7b6: 6 failed and 2 passed, exit 1 (the late ledger update after a restart lands in the older crew log, 'late' in ['planned', 'tried', 'late']; the tree keeps the old parent, 'chat-b' == 'chat-c'; the handoff adopts under sid-before-reset). At the repair (5d43baf73c on main df73916) and again after the rebase (2a8f65f, the same src/ as this head): 21 passed, exit 0. With this round's production hunks reverted (git apply -R of the src/ diff from e729c4cf1e, tests kept): 20 failed and 1 passed, exit 1. The one that passes both ways is the same-process ledger case, which round 20 already covered.
  • Member scenario repro.sh <base|fix> for crew-log-units-never-collected, on the same lane: base (main df73916) exit 1, all 3 replaced crew logs survive the sweep; fix (5d43baf73c, and again at 2a8f65f, the same src/ as this head) exit 0, all 3 collected and the slot's current crew log kept. Script md5 c5f2be31, the same as the member's copy, under ~/kc-repro3/crew-log-units-never-collected with lane-owned base and fix worktrees.
  • On the lane, each with exit 0: black --check, isort --check-only and flake8 on the 8 changed Python files. The full suites and gates run in CI.
  • CI on 2a8f65f: test/test_crew_log_direct_segment_open_ratchet.py::test_raw_opens_near_crew_log_segments_only_go_down failed on Linux shard 4 and Windows shard 4 (the Coverage Gate failed with them), because late_writes._read_carries opens a crew log segment itself. It is now listed in that ratchet's SEGMENT_OPENERS with its bound, beside retention_joins._holds, the other needle-filtered scan this PR adds. On the lane that test failed at 2a8f65f (1 failed, exit 1), and the whole file passes at 30cb353135 (5 passed, exit 0); with the two new files and the three handoff tests, 26 passed, exit 0. This head is that commit rebased over main commits that touch none of this PR's files (feat(chat): let a ChatPanel show reasoning and tool steps expanded #35688 and later), git range-diff printing = for the patch.
  • Rebased onto main after review, to df73916, then ff1a93a, then d822fd7, then the main named above. The one conflict was docs/decisions/README.md, where main's new row and this PR's row sit side by side; both are kept. Apart from the repairs above, the patch is unchanged (git range-diff df73916fda..5d43baf73c origin/main..HEAD shows only that README context). Main's commits in between touch crew_log/eager.py (fix(crew-log): let the eager seal finish before a delete removes the log #35705), crew-log-core.md and learn-cron-dashboard.md, and merge clean; the last rebase's git range-diff again shows only that README context.
  • A recorded limit (crew-log-emitter.md, "What a carry costs a reader"): a tree entry the writer loses after the hand-over, its requeued follow-on dropped again on a wedged disk, tells nothing, so that waiter's fence stays up until the gateway restarts.

Manual verification

N/A: unit coverage sufficient. The new tests and the repro above drive the real emitter, store, projection and sweep on a scratch data home, which is the path a reset takes in a running gateway.

Screenshots / video

N/A: no frontend change.

Related Issues

Fixes #23082

Sibling sweep

Swept: git grep -nE "archive_retention_days|sweep_expired(|_TERMINAL_CLOSE_REASONS" -- src/
Sibling: src/kiro_crew/history_files.py -- not affected -- transcript archives expire by file age, not by a close reason
Sibling: src/kiro_crew/decisions/log.py -- not affected -- decision day-files expire by the date in their name
Sibling: src/kiro_crew/dashboard/notification_coordinator.py -- not affected -- passive notes expire on their own ttl (state_parts/notification_log.py), with no lifecycle close to wait for

Pattern harvest

Not generalizable: the crew log is the only store whose retention waits on a lifecycle close reason, and which crew logs may go was a maintainer decision about slot state, not a code shape a rule could catch.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 10, 2026 01:31
@bolichen97
bolichen97 requested a review from buluoray November 10, 2026 01:31
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b7e8724173bb33f105da2a6af27cd90f91437cfe touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b7e8724173bb33f105da2a6af27cd90f91437cfe — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: a maintainer decision record backs the retention change, the generic unit_order_seam keeps core free of app imports, and every uncertain case keeps the old crew logs.

Notes

  • Deletion can't be undone across a downgrade. After a sweep collects superseded crew logs, a build without this change ignores session/carried, so its slot folds come back with less state than before. Name that rollback cost in the backwards-compatibility section, which was cut off in the captured description.
  • The change to test_only_a_destroy_close_authorizes_collection is a decision, not a gap: docs/decisions/2026-11-08-… names the rule it replaces.

[DESIGN-REVIEWED] b7e8724

Description read

[DESCRIPTION-READ] 6bd20f612185e037c8a1098baa4faf7ce7d9cf7a97c8387e7cbb1d3cacb4a448

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging b7e8724173bb33f105da2a6af27cd90f91437cfe. 2 of 2 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/crew_log/emit_surfaces.py:98 -- Legacy migrations bypass the carrier handoff

if late_writes.hand_to_carrier(slot, unit, (fold,), _land, moved):
Trigger: A dashboard crew-work request enters _carry_legacy_forward for an idle unit; a concurrent reset carries and closes that unit before migration queues its rows. The migration calls on_radar_recorded without slot or fold, so this handoff appends into the superseded unit.
Outcome: Migration verifies the rows and marks their legacy sources consumed. After the old handle is released, retention deletes their only crew-log copy, and unmodified migrated items disappear from the ledger.
Anchor: residual/crash-data-loss-corruption
Fix: Route migrations through the slot/fold-aware append, verify each receipt’s landing unit and sequence, and commit migration markers only after those entries land.
Sites: apps/builtins/issue_radar/backend/crew_store_carry.py:560; session_ledger.py:852 likewise omits the slot and commits its migration marker after the old unit grows.
Remedy: narrow or remove the mechanism

BLOCKING -- src/kiro_crew/crew_log/retention_joins.py:206 -- Removing a chain root replays retained history after its carrier

if not before:
return False
Trigger: A slot has succession chains a1 → a2 and b1, with header timestamps ordered 10, 30, 20 after a backward clock step and a missing predecessor link. b1 carries (a1, a2, b1), while another session key keeps a2 mapped. The sweep permits expired a1 to be removed because it has no predecessor.
Outcome: _succession orders the survivors (b1, a2). Cold slot reads apply b1’s carry and then replay a2, replacing current dashboard values with retired values and repeating history already carried.
Anchor: residual/crash-data-loss-corruption
Fix: Keep a root when its removal changes surviving units’ relative order across chains, rather than checking only inversions within one chain.
Sites: projection_readers._slot_units_for_fold and _read_workers_locked; session_ledger_control._causal_units; late_writes.carrier_after; tree_carry.tree_carries.

[BLOCK-MERGE] b7e8724
[GPT-REVIEWED] b7e8724

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1: the migration call crew_store_carry.py:560 → on_radar_recorded(session_id, data) passes no slot, so hand_to_carrier (late_writes.py:258-260) can't hand the rows on and appends them into the given unit. _all_landed then checks that same unit, so a superseded unit passes verification. Legacy files plus a concurrent reset is rare, but nothing contradicts it, and the loss is silent.
F2: retention_joins.py:206 returns False for a root with no before chain, so ordering across chains is never checked. A backward clock step and a missing predecessor link are unusual, but I couldn't show from this code that the system's own writers can't produce them.

[ADJUDICATION] b7e8724173bb33f105da2a6af27cd90f91437cfe total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] b7e8724173bb33f105da2a6af27cd90f91437cfe

[ADJUDICATION-FENCED] b7e8724173bb33f105da2a6af27cd90f91437cfe fenced=2 flagged=0
UPHOLD-FENCED F1 src/kiro_crew/crew_log/emit_surfaces.py:98 -- With no slot, hand_to_carrier returns False and the legacy rows land in the superseded unit; _all_landed verifies that same unit, so markers commit and retention can silently delete the only copy under a plausible migration-vs-reset race.
UPHOLD-FENCED F2 src/kiro_crew/crew_log/retention_joins.py:206 -- A root with no predecessor is collected without any cross-chain order check, and I could not show from this code that the clock-step plus missing-link combination is unreachable, so silent replay of retired values stands.
[GPT-ADJUDICATED-FENCED] b7e8724173bb33f105da2a6af27cd90f91437cfe

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b7e8724173bb33f105da2a6af27cd90f91437cfe false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b7e8724173bb33f105da2a6af27cd90f91437cfe — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The carry is aimed at the cause: an old slot crew log could never be collected because live slot state lived in it. The visible history losses after collection are a note, not a blocker.

Notes

  • Item 7 — inherited: the decision is recorded only in a docs/decisions/ entry that this diff adds (Decided by: … maintainer). No base RFC covers it. It accepts that "the panel covers only the turns of the crew logs still kept" and that the root card "loses the calls of a collected crew log". Whether that counts as product shape is arguable, so this is a note, not trigger (c).
  • Item 9 — rides along: the sampled-seq durability proof (_unit_last_seq, test_a_log_unreadable_*, test_durability_is_answered_from_this_units_own_growth) is swapped for a lane Receipt. The description gives the reason: "no clock or sampled seq decides where the entry goes or whether it landed". The deleted pins are superseded by the new late-write tests, not recast as "a gap".

What this change ships

Inventory (10 items) — 8 justified

Intent: make a chat's, crewmate's or conductor's old crew logs expire under session.archive_retention_days without changing what the slot shows. FIX (#23082).

  1. Older crew logs of a slot that moved on get closed superseded — justified
  2. Retention sweep collects superseded crew logs beside destroyed ones — justified
  3. The slot's newest crew log carries every slot fold, the work tree and the status (session/carried) — justified
  4. Slot folds replace their state at a carry; the older logs stop affecting reads — justified
  5. Writes that arrive late for a superseded log are re-routed to the slot's newest log on its writer lane — justified
  6. Openers record the session key; ledger carry joins other spellings; aliases file added — justified
  7. Context panel history, tool tally and the board's left-out count shrink once old logs are collected — undeclared in Goal; decision entry added here
  8. The gateway's session-key probe holds a close back while a key still maps the old session — justified
  9. Ledger durability is now answered by a lane Receipt instead of seq sampling — rides along
  10. Unlisted units are ordered by slot succession, not the header clock (core + Issue Radar) — justified

[FIRST-PRINCIPLES-REVIEWED] b7e8724

Description read

[DESCRIPTION-READ] 7943d0916f24b8416db74f6aba2a40f42efa317924559e00058f8acc06d1efde

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b7e8724173bb33f105da2a6af27cd90f91437cfe — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b7e8724

Verdict parsed from the review's SHA-scoped output markers for commit b7e8724173bb33f105da2a6af27cd90f91437cfe.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b7e8724173bb33f105da2a6af27cd90f91437cfe false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 10, 2026 01:47
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 10, 2026
@bolichen97
bolichen97 force-pushed the fix/crew-log-units-never-collected branch from c406128 to daa4466 Compare November 10, 2026 16:59
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • A predecessor finishing during the fold loses its final records span=2117439f419d
    self-added: yes
    mechanism: tail check of every unit the carry folds read, with a bounded refold

fixed in daa4466: a live turn of this process on an older crew log now stands the carry down before anything is read, and the tail of every crew log the folds read is taken first (emit_session._unit_tail: segment identity, size and mtime, over projection_readers.carried_fold_units: the slot's own, the tree's and the ledger join's other spellings).
The writer asks each tail again; one that moved makes the carry fold again, at most _CARRY_REFOLDS = 2 times, then it stands down with nothing written and nothing closed. _close_superseded also keeps a crew log whose tail moved after the carry checked it.
test_a_turn_the_older_unit_ends_while_the_carry_folds_is_not_lost fails on the c406128 code and passes here; five more tests pin a record, a joined spelling, the bounded refold and an entry that lands before the close.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • A later carry permanently duplicates history after a backward clock step span=7db6df3cde56
    self-added: yes
    mechanism: carry folded in its reader's own order, never the header clock's

fixed in daa4466: carried_fold_states no longer moves the new crew log to the end; it folds in the reader's own order and carries nothing unless that order already ends with the new crew log.
No carried fold reads header-clock order any more: the generic folds and workers read units_in_succession, and session_ledger_control._causal_units places a crew log the order file does not name by succession, after the recorded crew logs before it, so a carry keeps its place after what it holds in later reads and carries (radar, ledger, work, panel).
test_a_later_carry_after_a_backward_clock_step_reads_each_turn_once and the radar twin fail on the c406128 code and pass here, with per-fold and placement tests beside them.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 10, 2026
@bolichen97
bolichen97 force-pushed the fix/crew-log-units-never-collected branch from daa4466 to 95f9bbf Compare November 10, 2026 20:29
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 10, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider raised 1 possible problem and estimates a 60% chance that this change introduces a defect; a person should check it.

The carry, close and sweep design is careful in most places. The superseded close is withheld when a live turn still writes the crew log, when it gained an entry after the carry read it, when its header names another slot, when a session-map key maps it or the map can't be read, and when it's already closed for good. Retention still reads only the crew log tree, so the trust boundary holds. One gap remains, though: once a carry lands, spawns written into the carrying crew log (late ones handed on by this PR, or ordinary new ones) can push a carried spawn past the 1,024-per-crew-log read cap. The older crew logs that held that edge are already closed superseded, so retention deletes them and the creating edge is lost for good. The same overflow marks the session-tree read as faulted (cut), which main never does on this path. This breaks the PR's own claim that the older crew logs keep every creating edge. It is the GPT lane's blocking finding from the previous head, and this head's source is unchanged there. On top of that, almost all of CI (backend, Windows, E2E) and both general AI review lanes were still pending on this head, on a 9,777-line change that has turned up real b

Things to check (possible problems the decider raised; at this confidence many turn out to be fine):

  • Round 21 (F2) claims that a tree carry which would leave a spawn past the per-unit bound carries and closes nothing, so the older crew logs keep every creating edge. That only holds at the moment of the carry. Spawns written into the carrier afterwards, whether a late spawn handed on by this PR's ow (src/kiro_crew/crew_log/tree_carry.py, medium)

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit b7e8724173 · 9777 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on 95f9bbf (data loss). Upholding GPT's finding (#34264 (comment)): src/kiro_crew/crew_log/projection_readers.py:711 folds the radar carry by succession, while its reader folds the recorded unit order (crew_store_ledger.py:477 -> crew_store_unit_order.py:154; session_ledger_control._causal_units:867). A late old-session write across a reset records u2,u1, but the carry folds u1,u2,u3, so an issue the late u1 update resolved goes back to implementing; retention then deletes the older units needed to rebuild it, so it can't heal. Please fold the carry in the recorded unit order (session_ledger_control.py:844-892), and when that order can't be read, skip the carry and keep the older units. A test with the u2,u1 recorded order would pin it. The docs/decisions Evidence at line 119 still needs a link.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 10, 2026
@bolichen97
bolichen97 force-pushed the fix/crew-log-units-never-collected branch from 95f9bbf to bf7e6a4 Compare November 11, 2026 00:33
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 11, 2026
@bolichen97
bolichen97 force-pushed the fix/crew-log-units-never-collected branch from e4c88d0 to 8a4a567 Compare November 11, 2026 23:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 11, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@bolichen97
bolichen97 force-pushed the fix/crew-log-units-never-collected branch from 8a4a567 to 4e948f4 Compare November 12, 2026 04:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Late work-board writes replay history already carried forward span=2117439f419d
    self-added: yes
    mechanism: one slot-lane hand-over (late_writes.hand_to_carrier) for every writer that publishes a carried fold's unit order
    resolution: narrowed -- every writer that publishes a carried fold's unit order now lands through the one slot-lane hand-over round 20 added (late_writes.hand_to_carrier), no new mechanism

fixed in 4e948f4: the board copy of a worker report (emit_surfaces.on_work_recorded) and the board relay (work_relay._queue_run) pick their unit inside the writer job, on the board slot's lane, through late_writes.hand_to_carrier, the hand-over the ledgers use. A unit a later unit carried past (any carried fold the entry feeds, late_writes.folds_fed_by, or a superseded close) hands the job to the slot's newest unit, which asks again, and the work order is published for the unit the entry landed in, after the append.
The crew panel publish (on_panel_published), whose route resolves its unit the same way, lands through the same hand-over (class sweep). Only the job that appended, found no log, or was permanently dropped first releases the waiter; a waiter that gave up marks the entry abandoned, so a handed-on job appends nothing.
test_crew_log_carry_late_board_writes.py (report, relay, panel and the abandoned waiter, each with the older unit closed and still mapped): 8 passed at the fix and 8 failed with the src/ hunks reverted (report 3 entries for 2, relay 2 for 1, panel 3 publishes for 2).

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Combining spawn histories permanently discards child lineage span=182bfe589b57
    self-added: yes
    mechanism: a tree carry that a reader's per-unit spawn bound would cut stands the whole carry down
    resolution: narrowed -- a tree carry that a reader's per-unit spawn bound would cut now carries and closes nothing, so the older units stay and keep every edge they held

fixed in 4e948f4: tree_carry.tree_carries answers None when an older unit's spawn scan hit the cap (hit_cap, which cut implies), or when the spawns the new unit holds plus the carried ones pass SPAWNED_PER_UNIT_CAP. The [-SPAWNED_PER_UNIT_CAP:] slice is gone. None makes _write_carries carry nothing and close nothing, so no older unit that holds a child's only creating edge goes to retention.
test_crew_log_tree_carry_cap.py: 4 passed at the fix and 4 failed with the src/ hunks reverted ("176 children are carried nowhere, yet ['cap-c1', 'cap-c2'] were closed for retention"; after the sweep and a restart spawned_creator('kid-0000') is ''). Mutations: removing the hit_cap stand-down fails 1 test, removing the held-plus-carried bound fails 2.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 2) on 4e948f4 is red because main is red, not because of this PR. test/test_cloud_launch_customize.py::TestAmiArchitectureCheck::test_matching_architecture_proceeds_to_deploy also fails on a clean main 4dd046d worktree (DevSpace, Linux, Python 3.12, pytest -n0: 1 failed, exit 1). The cause is #35106's fake run_aws meeting #35010's stack lookup after a failed deploy (wizard._name_stack_left_behind -> ec2.find_stack). The fix is open as #35746, and this PR rebases onto it once it merges.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@bolichen97
bolichen97 force-pushed the fix/crew-log-units-never-collected branch from 4e948f4 to 259e89d Compare November 12, 2026 05:40
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 12, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Late dashboard writes are acknowledged but lost behind the carry span=2117439f419d
    self-added: yes
    mechanism: every dashboard write lands through the one slot-lane hand-over (late_writes.hand_to_carrier) the ledgers, board, relay and panel use
    resolution: narrowed -- _append_dashboard decides its unit inside the writer job on the slot's lane, no new landing mechanism

fixed in 533d7b6: emit_surfaces._append_dashboard, which the agentic and custom writes, their refusals and failed runs and the stale wake share, now asks late_writes.hand_to_carrier in its writer job and hands the entry to the slot's newest unit once that unit carried a fold the entry feeds or the named unit is closed superseded. The crewmate feed's on_goal_recorded and on_waiting_changed (the carried goal and waiting folds) go through the same helper (class sweep).
The answer is fixed once (_Landing): by the job that appended, a refusal, or the first job's drop or ceiling rejection, never by a job that handed the entry on; a waiter that gave up abandons it, so a handed-on job appends nothing. The feed, which calls from the serving loop, reads a closed unit's header on the writer and queues itself again on the slot's lane (late_writes.requeue_on_slot_lane).
test/test_crew_log_carry_late_dashboard_writes.py: 31 passed at this head, 29 failed with the src/ hunks reverted (custom first not second, agentic 1 not 2, goal ship v1 not ship v2, waiting the old list, nothing handed to the carrier).
Recurring span 2117439f419d, hit 3 (c406128, 8a4a567, 259e89d): each hit named one more writer that resolves a unit before its job runs. This round adds a syntax scan over emit_surfaces that fails when any append of an entry type feeding a slot-keyed carried fold skips the hand-over, so a writer added later is caught by the test, not by review.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes on 533d7b6. Thanks for the rework. The late dashboard write is fixed. _append_dashboard now asks late_writes.hand_to_carrier inside its writer job, and when that hand-over is turned off, 28 of the 31 cases in test/test_crew_log_carry_late_dashboard_writes.py go red. The radar carry still folds a crew's units in the order the crew recorded them. One point from our last review is still open, and on this head it is worse than a doubled list: an acknowledged ledger update is lost. After a gateway restart, the writer cannot tell that a newer log carried the slot when the older log is still mapped, because it learns of a carry only from an in-memory cache or from a superseded close (src/kiro_crew/crew_log/late_writes.py:136-176). So a ledger update that names the older log lands there, and the older log is then folded after its carrier. I reproduced it with the PR's own carry fixture, using emit.reset_caches() in place of the restart. Once the late update lands, the slot's phase goes back to the value the carry replaced, and the earlier events and tried approaches are listed twice. The next update that the newer log records reorders the logs again, and the slot then drops the late update, which had returned durable=True. Without the restart, the same update goes to the newer log and is kept. Please recover the carrier from a later log's durable session/carried entries when the cache cannot answer, and add a test that resets the caches between the carry and the late write. Separately, the GPT 6.1 lane blocks this head on a new finding in tree_carry.py; please answer that one on the lane's comment. Smaller: the decision entry's Evidence line still links nothing. Decisive lines: src/kiro_crew/crew_log/late_writes.py:136-176 (carrier_after, ours, unchanged since 259e89d); src/kiro_crew/session_ledger.py:539 and :578-583 (the writer that asks it)

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Carried tree decisions override a later acknowledged handoff span=182bfe589b57
    self-added: yes
    mechanism: tree mutations decide their unit on the slot's writer lane through the one hand-over the ledger, board, panel and dashboard writes already use
    resolution: narrowed -- adoption, release and spawn writes go through late_writes.hand_to_carrier under late_writes.TREE_KEY; no new mechanism

fixed in 2a8f65f: on_session_adopted, on_session_released and on_session_spawned now decide their unit inside the writer job on the slot's lane (emit_session._tree_lands_in) and hand themselves to the slot's newest unit once a carry landed there, so a decision filed after the tree carry lands beside the carried copy and wins by seq.
Every carry notes the tree (late_writes.TREE_KEY) whatever it copied of it, and a carry an earlier process wrote is read off the later unit's log (late_writes._read_carries), so a restart between the carry and the write changes nothing.
on_settled is told once, by the job that appended, found no log or was refused (emit_tree._TreeSettle.handed_on), so tree_moved reports the unit the entry landed in.
The handoff (session_handoff._move_tree_edge) captures the target's slot object before its first wait, re-reads the target id after the last one with nothing suspending before the append is queued, and moves no edge when the slot was replaced.
Pinned by test/test_crew_log_carry_late_tree_writes.py and three tests in test/test_session_handoff.py; on a DevSpace with pytest -n0, 21 passed at this head and 20 failed with this round's src hunks reverted.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Agent reply from this PR's babysit loop, point by point on the review of 533d7b6. The fixes are in 2a8f65f, rebased on main ff1a93a. The change request stands until you re-review.

  1. A late ledger update after a restart lands in the older crew log (late_writes.carrier_after). Fixed the way you asked. When this process's notes and the superseded close cannot answer, carrier_after now reads each later crew log for a durable carry (late_writes._read_carries: a session/carried for the slot that names one of the writer's folds). Each later crew log is read once per process: oldest segment first, records without a carry marker skipped unparsed, and the read stops at the end of the carry's one group commit. What it finds is remembered, so the next write opens no log, and the slot's newest crew log (the ordinary writer) never reads one. A failed read proves nothing: that write goes where it always went, and the next one reads again. test/test_crew_log_carry_across_restart.py (new) calls emit.reset_caches() between the carry and the late write, as in your repro. The late update lands in the carrier, the phase stays build (the carried one), each event and tried approach is listed once, durable is true, and the next update keeps the late one. A dashboard value after a restart takes the same path. At 533d7b6 the restart case fails ('late' lands in the older crew log); at 2a8f65f it passes.
  2. The GPT 6.1 finding on tree_carry.py:92. Answered on the lane's comment: fix: carry slot state forward so superseded crew logs expire #34264 (comment). Adoptions, releases and spawns now pick their crew log on the slot's writer lane and hand over to the carrier like the ledger writes, and the handoff reads its target again after its waits.
  3. The decision entry's Evidence line linked nothing. It now links fix: carry slot state forward so superseded crew logs expire #34264.

Checked on a DevSpace (Linux, Python 3.12, pytest -n0): the new and changed tests, 21 passed at this head; with this round's src/ hunks reverted, 20 failed and 1 passed (the same-process ledger case, which already worked). The member scenario (a real gateway: reload, reset-conversation, restart, then the sweep) exits 1 on main df73916 and 0 at this head. CI runs the rest.

crew-log/sessions grew whatever session.archive_retention_days said:
a reset, a discarded conversation, a rewind and a restart whose resume
fails each leave a session unit behind, and the sweep collected a unit
only when its newest close was destroyed. Collecting those units on
age alone would have deleted live slot state, because every slot-keyed
fold reads every unit its slot ran under, and the work tree fold reads
the units of the board at its root.

So a slot's newest unit now carries the slot first. When session/opened
creates a unit for a slot that already has units, the emitter folds each
carried fold over the units its reader folds, in that reader's own order
(off the writer, behind what the predecessor still owed), and writes
every non-empty state into the new unit as session/carried entries in
one group commit. The carried folds are every slot fold, the worktree
tree fold, carried under its root's slot, and the status session fold,
which a gone row reads over the slot to tell closed from lost. A driver
folding a slot's units that meets a carry for its own fold and slot
replaces its state with it, so the older units stop changing any answer;
a read of one unit names no slot and is unchanged. Then each older unit
gets session/closed {superseded} unless a live turn still writes it, it
gained an entry after the carry read it, it is already closed for good,
or the gateway's session-key probe finds a key that maps its id or
cannot say. A unit the slot has already moved past when its fold runs or
its carry is written carries nothing and closes nothing, since the
slot's later unit carries it. The sweep admits superseded beside
destroyed and still reads only the crew log tree.

Every other unit the fold reads must not change before the carry is
written, because the carry replaces what those units gave the folds.
An older unit with a live turn of this process stands the carry down
before anything is read, and the tail of each unit the folds read (its
segments' identity, size and mtime, from stat calls) is taken before
the read and asked again on the writer. A unit whose tail moved gained
an entry the read may have missed, such as a turn that ended in
between, so the carry folds again, at most twice, and then stands down
with nothing written and nothing closed.

A carry is written only where its reader folds the new unit last, and
a carrying unit must stay after the units it carries in every later
read and carry, or those units are laid over it again. So no carried
fold reads the header clock's order any more, which a backward clock
step inverts. The folds that keep a recorded-order file place a unit
the file does not name by the slot's previous_sid succession, right
after the last recorded unit before it there, or first when none is
(_causal_units), and a slot with no order file reads in succession.
Every other carried fold, the crewmate's workers fold among them,
reads the slot's units in succession. The emitter also publishes the
new unit in the ledger, work and panel order files (the work tree
reads the work file). The Issue Radar crew store keeps its own order
file and relies on that placement alone, so a crew's events are not
read twice once the crew moves past a carried unit it never recorded
into.

The work ledger's readers also join the units headed with a caller's own
spelling of the slot (a cron:<id> key, a gone chat's key that a
workflow's workflow-<run> fallback tab runs under, or the raw slack:<ts>
an older build headed a channel conversation with), so the ledger is
carried over that same join. Each opener now records in session/opened
the session key the slot's turns run under (the key the dashboard turn
acquired its session under, a channel conversation's own key), and the
carry joins the units headed with that key's ledger spelling. The new
unit is this build's own, so the first carry after an upgrade needs
nothing an older build wrote. A new unit whose opening states no session
key is not carried, and no older unit of its slot is closed. The carry
also joins a spelling the gateway names the slot from (its filename
fold, or a cron job's cron-<id> tab), found on the unit headers, and any
spelling the slot's aliases control file kept when a ledger write or a
live read joined it. A carry replaces what a joined read folded before
it, and each later carry folds over the carry before it, so a record
keeps reading for a key the slot's first carry joined; a later carry
that joins a key does not bring back what the first one left out. No tab
reaches that case: the dashboard links a tab only while it has none, so
a tab linked when it is created states that key in every opening, and a
tab it links after the tab ran is named from that key, so every carry
joins it from the unit headers, the first one included.

A joined spelling can be a slot that carries and closes its own units: a
gone chat whose reset carried its ledger and closed its older unit
superseded, read by its workflow's fallback tab. A read through the join
applies the reading slot's carries, never the chat's, so until the
reading slot carries its ledger it reads the chat's records from the
chat's units themselves. So a slot's first unit carries the ledger too
when its ledger readers join another spelling's units, and closes
nothing. Where a joined unit may hold its own spelling's carry, the
ledger carry is folded cold with the leading spelling's own carry
applied inside that spelling's units, so it holds the chat's records
however many of the chat's units are left; a unit of a later spelling
that holds a carry of its own stands the carry down. And the sweep keeps
a superseded unit that holds a ledger record while a slot that reads it
through the join holds no ledger carry of its own: a slot one of whose
units states, on its second line or in the newest opening of an older
build's unit this build resumed, a session key whose ledger spelling is
that unit's slot, or the slot the gateway names from a raw caller key
header. It reads that from the crew log tree alone, and a superseded
unit with no ledger record is collected whoever reads it.

So a record written under the key a slot's new unit runs under keeps
reading for a caller under that key after the carry and after retention.
A slot that starts joining a spelling after the sweep collected one of
its units reads that unit's records from its own first carry on; a read
between its first opening and that carry, which folds off the writer,
folds the joined units that are left.

A write of a carried fold's entry into a unit resolved before its job
runs (a ledger record, a worker report's board copy, a relayed board
copy, a crew panel publish, a dashboard value, record, refusal, goal or
waiting snapshot) is decided on the slot's writer lane, where the
successor's carry is committed too. The writer job asks whether a
later unit carried a fold the entry feeds, or the unit is closed
superseded, and hands itself to the slot's newest unit when one did; it
appends and publishes the unit order for the unit the entry landed in,
and a ledger job settles a receipt with that unit and seq. So the carry
lands wholly before or after the write, and no clock or sampled seq
decides where the entry belongs. The carried session-tree entries are
marked carried, so a per-unit count of a unit's children does not add a
creator's children again under its new unit, and the sweep searches a
resumed unit's segments for its newest opening when the tail window
holds none. A tree carry that a reader's per-unit spawn bound would cut
carries nothing and closes nothing, so the older units stay and keep
every creating edge they hold.

The session tree's adoption, release and spawn writes, whose unit a
dashboard verb reads from the session map before it queues, decide
their unit on the same lane and hand themselves to the slot's newest
unit once a carry landed there, since the tree ranks the newer unit's
copied decision first and a spawn filed under a superseded unit would
go to retention with it. A carry an earlier process wrote is read off
the later unit's log, once per unit per process, so a gateway restart
between the carry and a late write changes nothing. The handoff reads
its target's crew log again after its waits, and moves no edge when
the target's slot was replaced in between.

Four things are not carried and are recorded as costs: a worker's
legacy reports, which only the operator's work-board rebuild reads
from the worker's own units, a board's count of units left out past
its 512-unit limit, the context panel's history of past turns (the
usage session fold read over the slot), and the root card's tool
tally, which folds the slot's succession chain and so cannot take a
slot carry across a reset.

Decision: docs/decisions/2026-11-08-a-slots-newest-crew-log-carries-
its-state-so-older-ones-expire.md (option A of the stuck-fix brief).
@iamwhatever

Copy link
Copy Markdown
Contributor

Stale-PR cleanup check (2026-11-17): this PR now conflicts with main, GPT 6.1 Review is still red on b7e8724, and @buluoray's change request from 2026-11-10/11-12 is still open. After several review rounds in the same crew-log carry area, this needs @bolichen97 to rebase and answer the latest GPT finding, and then a re-review from @buluoray. The cleanup will not push to this branch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

crew_log sweep_expired never collects units closed by reset or left unclosed by a restart

4 participants