Repository navigation
fix: retry a work-ledger wake whose steered turn dies - #33732
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns. The fix reuses the existing Notes
[DESIGN-REVIEWED] 59b28e9 Description read[DESCRIPTION-READ] 19c734cb0c2f7005ddb69446b35fb6f21024c8e5b3eedd8c0901892b0191e444 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The change fixes the linked defect (#32250) by reusing the What this change shipsInventory (7 items) — 7 justifiedIntent: a worker's report steered into a busy conductor turn is delivered again on the next tick when that turn dies, instead of waiting out the quiet floor. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 59b28e9 Description read[DESCRIPTION-READ] aa4c74352892337aab19f9ff152a27a7be268e611b7db4acdcdb93b25c5b2974 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
buluoray
left a comment
There was a problem hiding this comment.
Approving. A steered work-ledger wake now carries its loop's (id, generation), so a turn that took it and died, or the requeued wake's own turn, charges the loop and earns the one retry. Generation never reaches a row or the durable queue (dashboard/chat_turn/steer_queue.py:376-382, 447-450, 790-794; dashboard/chat_turn/queue_drain.py:1448-1452; dashboard/session_deliver.py:86-96; slack/gateway_runtime/nudge_dashboard_fire.py:779-781). GPT's blocking race is real but needs rare timing, and when it hits, the outcome is the pre-PR behavior (wait for the quiet floor). The adjudication step failed to run, so the block still needs the author's disposition or an override.
15b5ecb to
c3863d5
Compare
|
self-added: yes
[operator: bolichen97] |
c3863d5 to
bcd67c9
Compare
|
self-added: yes
[operator: bolichen97] |
bcd67c9 to
97f5d8e
Compare
A work-ledger wake that lands while the conductor's turn is running is steered into that turn and counted as delivered, so its one retry only fires when the loop's failure streak grows. The turn that carried the wake never charged the loop: a steer the running turn ended before taking drained as its own turn without the loop's identity, and a person's turn that took the steer and then died was not the loop's cycle. Either way the loop read the ledger as unchanged and sat on its quiet floor. The steer now records the loop's config generation next to its automation meta. A turn that takes the steer adopts that loop's cycle for its fault counts (the newest wake, when one echo settles several), and a steer requeued at a turn's end carries the loop's id and generation on its process-local queue entry, so the turn it drains into is charged like the cycle's own. Neither gains wake authority, nothing new is persisted, and a person's edit of a requeued wake's card charges no loop. Closes #32250
97f5d8e to
59b28e9
Compare
|
self-added: yes
[operator: bolichen97] |
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own. The PR makes a steered work-ledger wake count as the loop's cycle in whichever turn carries it. If the running turn takes the steer, it adopts the newest settled wake's (id, generation) as _wake_cycle. If the turn ends before taking it, the requeued entry carries it on the in-memory _recovered_wake key, and the drain passes it on only while the entry still counts as automation or recovery. The generation is never persisted: the transcript row is rebuilt without it, it is removed from the queue meta, and the durable queue writer only emits a fixed list of keys. The earlier GPT blocker about a stale generation is fixed by replacing _wake_cycle outright. The retry baseline is now read before the fire's await, and the settle step does not reset the streak. Every path has a test. Only rare race-timing differences remain. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
buluoray
left a comment
There was a problem hiding this comment.
Approve on 59b28e9. The delta since the earlier APPROVE at 97f5d8e fixes the GPT batch-settle finding. When one echo settles several steered wakes, the turn now carries the last-sent wake, so the charge uses the loop's newest generation. The rest of the PR is unchanged on the same base, and the delta is correct and tested. Decisive lines: src/kiro_crew/dashboard/chat_turn/steer_queue.py:247-255; test/test_recovery_l1_chat_runner.py:1510-1571
Problem / Motivation
Goal: A work-ledger wake steered into a running turn gets the same one retry as a fired wake when the turn that carries it dies before any turn on the slot lands.
When a worker reports while the conductor is mid-turn, the loop steers the
[work-ledger wake]into that running turn and counts it as delivered. The wake's one retry only fires when the loop's failure streak grows, and the turn that carried the wake never grew it: a steer the running turn ended before taking drained as its own turn without the loop's identity, and a person's turn that took the steer and then died was not the loop's cycle. On a provider throttle the next ticks logquiet tick (0 pending)and the worker's report waits for the quiet floor.Why it matters
A conductor driving workers is often busy when they report. Losing the prompt re-delivery means a finished or blocked worker sits unanswered for a whole quiet-floor interval after a throttle, which is the same stall #31640 fixed for the wake's own recovery turn.
Not a goal
(id, generation)fence stay as they are.meta.nudge.What changed (motivation → approach → change)
A conductor that was mid-turn when a worker reported now gets that report delivered again on the next tick when the turn carrying it dies, instead of waiting out the quiet floor. Cause: the loop is charged only when the turn that failed is the loop's own cycle (
_ChatTurn._wake_cycle), and the steer gave its carrying turn no way to be that cycle, because the steer carried the loop's id but not its config generation. The change reuses the carrier #31640 added instead of adding a second mechanism:(id, generation)steering_consumedsettle hands the turn the newest wake it settled (one echo can settle several), which the turn adopts in place of any cycle it carries (the steer names the slot's live loop at its current generation)_recovered_wakekey, and the drain passes it to the wake's own turnThe wake's retry is measured from the loop's failure streak as it stood when the wake went out, read before the fire awaits its delivery (
autonudge_service/firing.py,gate.py): a backend that reports the steer taken before its steer call returns can have charged the carrying turn's death by then.session_deliver.automation()takes an optionalgeneration, which only the steer (nudge_dashboard_fire.py) passes; the steered row is rebuilt from cycle and loop id, and the requeue moves the generation out ofAUTOMATION_STEER_META, so no row or queue file carries it. The drain reads the key only while the entry still is the requeued wake, so a person who edits the card is not charged. The comments that described the carrier or said only the loop's own cycle may spend its budget (chat_runner.py,chat_delivery.py,state_parts/slot_ladder.py,recovery.py,turn_failures.py,turn_locals.py,autonudge_service/timers.py) name the steered carrier;turn_failures.pyreadsturn._wake_cyclewhere it held a copy, because the settle now writes it on the turn; andlearn-cron-dashboard.mdandmonitor-architecture.mddescribe both paths, including that a person's turn that took the wake can bring a stand-down one cycle forward on a loop already one failure short of it.Backwards compatibility
Compatible: every new value is optional and process-local; an automation steer without a generation behaves exactly as before.
Removes nothing: no key, row field, validator or caller is taken away; the requeued entry's
AUTOMATION_STEER_METAkeeps its{"cycle", "loop_id"}shape.Tests
test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_a_turn_that_took_a_steered_wake_charges_its_loop_when_it_dies: a person's turn takes a steered wake, then dies, and the loop is charged under its id and generation.test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_a_loop_cycle_that_took_a_steered_wake_charges_the_right_cycle: a loop's own cycle takes a steered wake of the same loop at a newer generation and is charged under the newer one, and a wake of a loop that replaced it on the slot is charged to that live loop.test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_one_echo_that_settles_two_wakes_charges_the_newer_generation: one echo settles a loop's wake at generation 3 and then its wake at generation 4, and the dying turn is charged under 4. The two wake texts pin their hash, so the walk the settle replaced visits generation 3 first on every run.test_automation_steer_row_role.py::TestRequeuedAutomationSteer::test_the_requeued_wakes_own_turn_is_its_loops_cycle: a real steer, then the turn-end requeue and the drain: the drained turn runs with the loop's wake, and neither the steered row, the drained row nor the queue meta shows the generation.test_automation_steer_row_role.py::TestRequeuedAutomationSteer::test_a_human_edit_clears_automation_origin_and_drains_as_user: an edited wake card drains without the loop's wake.test_autonudge_ledger_quiet_floor.py::test_a_wake_whose_turn_fails_before_its_delivery_returns_is_retried: the loop is charged for the wake's dead turn before its delivery returns, and the next tick still retries the wake.test_work_ledger_wake_delivery.py::TestSteeringAWakeIntoARunningTurn::test_a_busy_conductor_is_steered_rather_than_skippedandtest_session_deliver.py::TestCallersRouteThroughDeliver::test_a_work_ledger_wake_is_delivered_as_automation: the fire's steer carries the loop's generation.Proof: PROVEN test/test_autonudge_ledger_quiet_floor.py::test_a_wake_whose_turn_fails_before_its_delivery_returns_is_retried, test/test_autonudge_refactor_contract.py::TestSurface::test_a_service_member_keeps_its_kind_and_signature[_settle_fire_cycle], test/test_chat_runner_base_surface.py::test_a_moved_name_keeps_its_base_shape_and_owner[steer_queue:_settle_consumed_steers], test/test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_a_turn_that_took_a_steered_wake_charges_its_loop_when_it_dies, test/test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_a_loop_cycle_that_took_a_steered_wake_charges_the_right_cycle[same-loop-newer-generation] (head 59b28e9)
Determinism: the eight tests above (both cases of the parametrized one) ran 20 times at
-n0and 10 times at-n 4 --dist loadgroup --max-worker-restart=2besidetest_steer_requeue.pyandtest_automation_steer_row_role.py, all green, and their seven files ran green in a shuffled order (seed 6953, 895 passed). One earlier-n0attempt hit the suite's 120 s timeout once, at the new test's teardown, while the machine's disk I/O was saturated (an unchanged neighbour's call took 105 s in that run); the 20 runs above are the repeat.Manual verification
N/A — unit coverage sufficient: a live repro needs a provider throttle inside a running conductor turn at the moment a worker reports, and the tests drive the same runner, settle, requeue and drain seams that path takes.
Screenshots / video
N/A — no frontend change.
Related Issues
Closes #32250
Sibling sweep
Swept: git grep -n "_steer_automation" -- src/
Sibling: src/kiro_crew/dashboard/chat_delivery.py -- not affected -- its pops clear a steer that was refused, or already settled or requeued, so no turn carries that wake
Sibling: src/kiro_crew/dashboard/chat_api/turn_stop.py -- not affected -- a hard stop discards the steer with the queue, so no turn carries that wake
Sibling: src/kiro_crew/dashboard/steer_retract.py -- not affected -- a take-back removes the steer before any turn takes it
Swept: git grep -n "session_deliver.automation(" -- src/
Sibling: src/kiro_crew/slack/gateway_runtime/nudge_dashboard_fire.py -- changed here
Swept: git grep -nE "_note_ledger_wake(|_ledger_wake_streak[" -- src/
Sibling: src/kiro_crew/autonudge_service/firing.py -- changed here
Sibling: src/kiro_crew/autonudge_service/gate.py -- changed here
Swept: git grep -nE "_wake_cycle = |[_RECOVERED_WAKE_KEY] = |consumed_wake" -- src/
Sibling: src/kiro_crew/dashboard/chat_turn/steer_queue.py -- changed here
Sibling: src/kiro_crew/dashboard/chat_turn/turn_ledgers.py -- not affected -- it copies the turn's current
_wake_cycleonto the recovery it queuesSibling: src/kiro_crew/dashboard/chat_turn/turn_locals.py -- not affected -- it sets the cycle at turn start, before any steer can arrive
Pattern harvest
Not generalizable: the carrier is specific to the three ways a loop's wake reaches a turn (the fired turn, its recovery, the steer), and all three now carry it.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)