Skip to content

feat(work-ledger): auto-retire a spent conductor board in place at the stored bound - #18398

Closed
iamwhatever wants to merge 1 commit into
kirodotdev:mainfrom
iamwhatever:feat/work-ledger-auto-retire
Closed

iamwhatever wants to merge 1 commit into
kirodotdev:mainfrom
iamwhatever:feat/work-ledger-auto-retire

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A conductor whose board has run its lifetime create limit keeps working automatically once its items are all finished, instead of blocking until someone runs a manual sweep.

Today a conductor board has a lifetime cap of 256 creates (MAX_STORED_ITEMS_PER_CONDUCTOR), counted by a monotonic created_total. When the board hits the cap, _create_item refuses every further create and tells the operator to run kirocrew ledger-sweep --purge by hand. But the sweep only deletes a board whose items are all terminal AND older than its window — and it silently keeps any board that still has an open item. So the one case that actually blocks you — "board full, I still need to create" — is exactly the case the manual step cannot help with, and the agent stalls waiting on a human action that may do nothing.

Why it matters

A long-running conductor (one that opens one item per ticket and closes it) reaches 256 lifetime creates in normal use. At that point it stops dead and requires an operator to notice, run the sweep, and discover the sweep did nothing because the board still had the create's own would-be slot occupied. This turns a routine capacity rollover into a manual, confusing intervention. Auto-retiring a spent-but-finished board removes that intervention entirely while preserving the two real guards (≤32 live fan-out, ≤256 lifetime history per generation).

Not a goal

  • Does not raise or remove either cap (live fan-out stays 32, lifetime history stays 256 per generation).
  • Does not auto-retire a board that still has an open item — live work is never discarded; that case still refuses, now pointing at closing the open items.
  • Does not change ledger-sweep behavior for finished old boards (whole-board purge on its age window is unchanged).

What changed (motivation → approach → change)

  • Motivation: the manual-sweep exit is a dead end precisely when it is needed (full board + a pending create), so the create path should resolve the common case itself.
  • Approach: at the cap-hit point in _create_item, decide by whether any item is still live. All items terminal → the board has done its job → retire this generation in place and open a fresh one under the same slot, then let the create proceed. Any item still open → refuse, but point at closing those items (which is what makes the board retireable), not at the sweep.
    • In-place rotation (clear item files, reset created_total, mint a new generation, write the header back) was chosen over delete-dir-and-recreate because the directory is named by slot_key; deleting and re-ensure_conductor-ing reintroduces the lock-inode race and Windows handling that purge_conductor spends large comments guarding. Rotation leaves the directory and lock inode untouched.
  • Change: new _retire_and_recreate_locked helper; _create_item calls it under the existing conductor lock when the board is full and all-terminal. The error for the full+open case is reworded to point at closing/abandoning the open items.

Crew-log alignment: the rotation's new generation is stamped onto the next conductor write, and the crew-log fold's _work_step resets the board the moment it sees a conductor entry with a new generation — so cache and fold move to the new generation together. The stuck crew_log_incomplete state only occurs when cache and fold carry different generations, which a purge creates (it cannot delete append-only log entries) but a rotation does not. Rotation and the first new create run inside the same conductor lock a rebuild also needs, so no rebuild can observe the intermediate state.

Backwards compatibility

Compatible: no field, kind, key or validator is added, renamed, removed or tightened. The only observable behavior change is that a create against a full+all-terminal board now succeeds (via in-place rotation) where it previously raised item_store_full; a create against a full board with an open item still raises item_store_full (reworded message, same code and field). Callers that handle item_store_full are unaffected — the error still fires for the only case that cannot be auto-resolved.

Removes nothing: no path, record, row, key or public function is hidden, deleted or renamed; created_total and generation keep their meaning, and a new generation is a fresh board under the same slot (the mechanism crew-log already supports).

Tests

  • test/test_work_ledger.py: a full all-terminal board auto-retires on the next create (new generation, created_total reset, create proceeds); a full board with an open item is refused with a close-hint message that does not mention the sweep. Existing counter-invariant tests updated to keep one open item so they still exercise "removing a file reclaims nothing" without triggering rotation.
  • test/test_work_ledger_projection.py: end-to-end against a real crew log — drive a board to the cap through the routes, trigger auto-retire with one more create, then rebuild and assert it succeeds (no crew_log_incomplete) and reflects the new generation. This is the safety-critical assertion.
  • test/test_work_ledger_tools.py: the route-level full+open case still returns the item_store_full conflict.

test_work_ledger*, test_work_ledger_projection, test_work_ledger_tools and test_ledger_sweep suites pass.

Manual verification

N/A — unit and projection coverage is sufficient; the crew-log fold round-trip is exercised end-to-end in test_work_ledger_projection.py.

Related Issues

Introduced the cap in #13879.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 9, 2026 08:21
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 4e02ed607b9d7c7608aaa814dfbf2f6cf500b4ed touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🔴 BLOCK

Premise-level review of 4e02ed607b9d7c7608aaa814dfbf2f6cf500b4ed via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: BLOCK

Product-shape change without accepted RFC: at the stored bound, a create the model can trigger now irreversibly deletes the board's records. The base RFC and spec say it must refuse until an operator purges.

Not justified as shipped

  • Item 1 — symptom-level: the description says the sweep "is a dead end precisely when it is needed". But the retire only fires on an all-terminal board, which kirocrew ledger-sweep --purge --older-than-days 0 already purges (ledger_sweep.py:649, :540). The case the sweep can't help, an open item, is still refused.
    Subtraction: delete _retire_and_recreate_locked and its call site, and keep only item 3's reworded open-item message.
    Clears when: the auto-retire is removed, or a maintainer records the shape (see Blockers).
  • Item 2 — undeclared: the description never mentions the separate goal-board refusal or its new "fresh conductor" message.
  • Item 4 — undeclared: the description is silent on the deleted pins. It drops the test test_the_stored_bound_refuses_a_create_on_a_board_of_closed_items and the comment "refuses rather than evicts -- nothing in this module deletes a record".
    Subtraction: restore both pins.
    Clears when: the pins are restored, or there is evidence they were wrong.

What this change ships

Inventory (4 items) — 1 justified

Intent: a long-running conductor keeps creating items after its board fills up, with no manual step. This is an ADDITION framed as a fix.

  1. A full board whose items are all finished and that has no goal now deletes its items and starts fresh on the next create — symptom-level
  2. A full board running a goal is refused with a new "fresh conductor" message — undeclared
  3. A full board with an open item is refused with a "close the open items" hint instead of the sweep hint — justified
  4. Pins that recorded "refuse, never delete" are deleted or rewritten — undeclared

Blockers

Product-shape change without accepted RFC: the create path auto-deletes a spent board

  • rfc-conductor-work-ledger.md is partial on the base list. Its text pins the opposite: "a board at it refuses further creates with item_store_full until the finished ledger is purged" (line 306).
  • The kept spec line reads: "the deletion is irreversible, so no model-reachable surface reaches it".
  • The diff gives that deletion to apply_conductor_action, which the model reaches.

Clears when: an RFC covering auto-retire at the stored bound is merged non-draft on the base, followed by a push or rebase, or a maintainer posts /ai-review override first-principles 4e02ed607b9d7c7608aaa814dfbf2f6cf500b4ed: <decision>.

[FIRST-PRINCIPLES-REVIEWED] 4e02ed6

Description read

[DESCRIPTION-READ] 99ad514a200e39a8c59433d3522e2ee8bda912d62f744f5a781c18d5f525c89a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🔴 BLOCK (blocking)

Design-level review of 4e02ed607b9d7c7608aaa814dfbf2f6cf500b4ed via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: BLOCK

The model's routine create now irreversibly deletes a spent board's records, which reverses the spec's "no model-reachable surface reaches" deletion decision. The PR reads that refusal as a gap.

Blockers

An automatic, model-reachable deletion path overturns a recorded decision

  • Cause: the base comment says the create path "refuses rather than evicts -- nothing in this module deletes a record." The spec says "the only cleanup is an explicit operator command" and "the deletion is irreversible, so no model-reachable surface reaches it." The deleted test test_the_stored_bound_refuses_a_create_on_a_board_of_closed_items pinned that refusal and its operator remedy.
  • Mechanism: a conductor agent's create against a full, finished, goal-free board now runs _remove_contents_locked, removing every item record and item_events log. A rebuild only reproduces the new generation.
  • Consequence: with no human involved, 256 items of history disappear from every product surface. Spec lines 74 and 76 are left contradicting the new paragraph inserted between them.
  • Fix: keep the delete-free invariant. Either rotate by archiving the old generation (move it, don't unlink it), or keep refusing and fix the dead-end hint (e.g. let the sweep take a finished board at a zero-day window).

Clears when: the rotation no longer unlinks records (archive or rename), or a maintainer explicitly approves reversing "no model-reachable deletion" and spec lines 74/76 are rewritten to match

Watch

  • The description leaves out the goal-board branch. The diff adds a third case: goal boards are refused with a new "fresh conductor" message. The description's Approach and Backwards-compatibility sections only describe all-terminal and open-item boards.

Clears when: the description accounts for the goal-board refusal and its changed message

[DESIGN-REVIEWED] 4e02ed6

Description read

[DESCRIPTION-READ] a5c59924fb57bf07a6fd70f176223c482f699dacfb290e81b9af4a59b1aa0499

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 4e02ed607b9d7c7608aaa814dfbf2f6cf500b4ed via the fork AI-review pipeline; updated in place on each push.

Review details

Advisory only. Nothing here blocks the merge.

FINDING — src/kiro_crew/work_ledger.py:2258 — live = _retire_and_recreate_locked(slot_key, live) deletes every item record and writes the rotated header before the create can still fail, and neither undo path covers that. (1) When the crew-log append is not confirmed, _refuse_unrecorded → restore_snapshot puts back only what snapshot_for_write took: the old header and the new item's files. The board is left with the OLD generation and created_total at the bound over an empty items/. The crew log still holds the dropped items, and the cache is not flagged dirty. (2) When _commit_item_locked raises OSError, _restore_text(header_path, header_before) puts back the ROTATED header, because header_before was read after the retire. The route returns 503 without restoring anything. A later rebuild then refuses with crew_log_incomplete on the generation guard in _refuse_fold_behind_cache, until another create lands. Either way, a create that failed has still retired the board, which breaks the docstring's promise that "a refused create leaves every file byte-identical". → Fix: in the changed lines, run _require_record_fits before the retire and read header_before before the retire. Covering the item files and the unrecorded-append path also needs code this PR did not touch: the route should call mark_cache_dirty when an undone create had retired the board, or snapshot_for_write should snapshot every item file before a create that may retire.

[OPUS-REVIEWED] 4e02ed6

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — 🔴 changes requested (blocking)

Reviewed 4e02ed607b9d7c7608aaa814dfbf2f6cf500b4ed via the fork AI-review pipeline; updated in place on each push.

1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

BLOCKING -- src/kiro_crew/work_ledger.py:2067 -- Retirement deletes files the create rollback cannot restore

failures = _remove_contents_locked(directory)
Full, all-terminal, goal-free board + crew-log append timeout -> api_work_ledger_record rotates it -> _refuse_unrecorded restores only the snapshotted header and slot key, leaving the old generation’s full header over an empty board without flagging the cache dirty.
Anchor: residual/crash-data-loss-corruption
Fix: Retain the retired generation’s files until the create and crew-log append commit, and restore that complete generation on failure.

[BLOCK-MERGE] 4e02ed6
[GPT-REVIEWED] 4e02ed6

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

No adjudicable findings, so total=0. F1 is fenced, so it keeps blocking whatever I rule; I've flagged it with a draft override rationale for a repository writer.

F1: the deleted item records are all terminal (work_ledger.py:2047, plus the census refusals in patch hunk 140–159). They are exactly the records a successful retirement deletes on purpose. Only the conductor header, slot_key and the newly created item are snapshotted (work_ledger.py:3639-3641), so the undo (handlers/work_ledger.py:1667 → restore_snapshot at work_ledger.py:3693) puts back the old full header over an empty items/ and does not mark the cache dirty. The crew log, which is the record (handlers/work_ledger.py:1027), still holds the old generation, so a rebuild restores it. The next create also cleans this up on its own: created_total is still at the bound and list_work_items finds nothing open, so the board is retired again and the create goes through (patch hunk 233–258). The failure needs the crew-log append to fail on the one create that triggers retirement, which happens once per 32 creates (MAX_STORED_ITEMS_PER_CONDUCTOR).

[ADJUDICATION] 4e02ed607b9d7c7608aaa814dfbf2f6cf500b4ed total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 4e02ed607b9d7c7608aaa814dfbf2f6cf500b4ed

[ADJUDICATION-FENCED] 4e02ed607b9d7c7608aaa814dfbf2f6cf500b4ed fenced=1 flagged=1
FLAG F1 src/kiro_crew/work_ledger.py:2067 -- The records it drops are only terminal items that a successful retirement deletes anyway, the crew log still holds them so a rebuild restores them, and the next create re-retires the item-less full header (patch 244-258) and converges, so the only effect is a temporary empty-board read after a crew-log failure on the one create per 32 that triggers retirement.
[GPT-ADJUDICATED-FENCED] 4e02ed607b9d7c7608aaa814dfbf2f6cf500b4ed

🏷️ Fenced finding(s) machine-flagged as likely edge case

The security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override recorded by a repository writer, who must independently verify a rationale before recording it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop. (This lane's comment deliberately carries no override command.)

  • F1 src/kiro_crew/work_ledger.py:2067 — The records it drops are only terminal items that a successful retirement deletes anyway, the crew log still holds them so a rebuild restores them, and the next create re-retires the item-less full header (patch 244-258) and converges, so the only effect is a temporary empty-board read after a crew-log failure on the one create per 32 that triggers retirement.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 9, 2026
@iamwhatever
iamwhatever force-pushed the feat/work-ledger-auto-retire branch from 96db6d8 to f58d870 Compare October 9, 2026 16:38
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 9, 2026
@iamwhatever
iamwhatever force-pushed the feat/work-ledger-auto-retire branch from f58d870 to 40392de Compare October 9, 2026 16:45
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 9, 2026
…ound

A conductor board bounds its lifetime creates at MAX_STORED_ITEMS_PER_CONDUCTOR
with a monotonic counter, so a full board refused every further create and the
only exit was a manual `kirocrew ledger-sweep --purge` -- which could not even
help, because the sweep keeps any board the operator is still creating against.

Resolve a full board inside _create_item instead: when the next create arrives
and every item is terminal, retire the generation IN PLACE
(_retire_and_recreate_locked) -- drop the item records, reset created_total,
mint a fresh generation, carry goal/depth/parent across -- and let the create
proceed on the new board. In-place rotation (not purge-and-recreate) avoids the
lock-inode race purge handles, and does not create the crew_log_incomplete stuck
state a purge can: the next create is a conductor entry stamped with the new
generation, which the work fold resets the board on, so cache and fold cross
together; a rebuild cannot slip into the gap (same conductor lock).

A full board that still holds an OPEN item is NOT retired (that would drop live
work) -- it is refused with item_store_full, pointing at closing the open items
rather than at the sweep.

Tests: all-terminal full board auto-retires; full board with an open item is
refused with a close hint; an auto-retired board rebuilds clean from a real
crew log (no crew_log_incomplete). Specs and the user doc updated.
@iamwhatever
iamwhatever force-pushed the feat/work-ledger-auto-retire branch from 40392de to 4e02ed6 Compare October 9, 2026 19:18
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Addressed all four review findings in 4e02ed6 — redesigned the retirement rather than point-patching.

F1 (goal-cap bypass) + F4 (goal lost on rebuild): the create path now never auto-retires a board that is running a goal (goal text set, or goal_version > 0). A goal's item_cap is a spend ceiling a person set, so a full goal board is refused with item_store_full, pointing at starting the next goal in a fresh conductor — a human decision, not a silent reset. Only a goal-free queue board (the pipeline/security conductors that never record a goal) auto-retires, so there is no goal accounting to reset and no goal to drop on a later rebuild.

F2 (non-atomic destructive rotation): _retire_and_recreate_locked now reuses the purge's proven machinery instead of a hand-rolled unlink loop — _hold_every_item_lock + census_items + _remove_contents_locked. The fresh header is written only after a clean removal; a partial removal (counted, not swallowed) leaves the OLD full header over whatever survived — a board that still refuses creates and can be retried — never a zeroed header over half-deleted records.

F3 (torn-record census): the terminal check is now census_items (reads the item files, counts a torn/misnamed record as neither open nor closed) under every item lock, not list_work_items (which skips an unreadable record). A board with any open OR unreadable record, or directory damage, refuses the retire with ledger_not_finished.

Tests: added test_a_full_board_running_a_goal_is_refused_not_retired and test_a_full_board_running_a_goal_is_not_retired_through_the_routes; updated the retire/open-item/409 tests to goal-free boards; the end-to-end test_an_auto_retired_board_rebuilds_clean_from_the_crew_log still asserts the rebuild does not latch crew_log_incomplete. 887 work-ledger/projection/tools/sweep/conductor tests pass (the one failure is a pre-existing env assertion that also fails on clean main).

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 9, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Superseded by #18584, which carries the same change as a same-repo PR (this fork PR pulled in the Fork Workflow-Change Guard lanes unnecessarily — I have push access to upstream). #18584 also addresses the four GPT/Opus review findings from this PR: the retire now never touches a goal board (goal cap preserved), reuses the purge's census + item-locks + counted removal, and writes the fresh header only after a clean removal. Closing in favour of #18584.

@iamwhatever iamwhatever closed this Oct 9, 2026
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants