Skip to content

fix: retry a work-ledger wake whose steered turn dies - #33732

Merged
bolichen97 merged 1 commit into
mainfrom
fix/32250-steered-wake-retry
Nov 12, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/32250-steered-wake-retry

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A work-ledger wake steered into a running turn gets the same one retry as a fired wake when the turn that carries it dies before any turn on the slot lands.

When a worker reports while the conductor is mid-turn, the loop steers the [work-ledger wake] into that running turn and counts it as delivered. The wake's one retry only fires when the loop's failure streak grows, and the turn that carried the wake never grew it: a steer the running turn ended before taking drained as its own turn without the loop's identity, and a person's turn that took the steer and then died was not the loop's cycle. On a provider throttle the next ticks log quiet tick (0 pending) and the worker's report waits for the quiet floor.

Why it matters

A conductor driving workers is often busy when they report. Losing the prompt re-delivery means a finished or blocked worker sits unanswered for a whole quiet-floor interval after a throttle, which is the same stall #31640 fixed for the wake's own recovery turn.

Not a goal

  • A running turn that lands without taking the steer still ends the wake's claim. The requeued wake's own turn is then charged if it dies, but earns no immediate retry; the quiet floor delivers. Changing what a landed turn means is separate work.
  • No wake authority for the turn that carries the wake: the structural-terminal and model-refused stops still read the loop's own delivered turn only.
  • No change to the bounds: one retry per wake, the five-failure stand-down and the (id, generation) fence stay as they are.
  • No new persisted field and no change to any row's meta.nudge.

What changed (motivation → approach → change)

A conductor that was mid-turn when a worker reported now gets that report delivered again on the next tick when the turn carrying it dies, instead of waiting out the quiet floor. Cause: the loop is charged only when the turn that failed is the loop's own cycle (_ChatTurn._wake_cycle), and the steer gave its carrying turn no way to be that cycle, because the steer carried the loop's id but not its config generation. The change reuses the carrier #31640 added instead of adding a second mechanism:

Path What carries the loop's (id, generation)
The running turn takes the steer, then dies its steering_consumed settle hands the turn the newest wake it settled (one echo can settle several), which the turn adopts in place of any cycle it carries (the steer names the slot's live loop at its current generation)
The turn ends before taking the steer the requeued entry carries it on the process-local _recovered_wake key, and the drain passes it to the wake's own turn

The wake's retry is measured from the loop's failure streak as it stood when the wake went out, read before the fire awaits its delivery (autonudge_service/firing.py, gate.py): a backend that reports the steer taken before its steer call returns can have charged the carrying turn's death by then.

session_deliver.automation() takes an optional generation, which only the steer (nudge_dashboard_fire.py) passes; the steered row is rebuilt from cycle and loop id, and the requeue moves the generation out of AUTOMATION_STEER_META, so no row or queue file carries it. The drain reads the key only while the entry still is the requeued wake, so a person who edits the card is not charged. The comments that described the carrier or said only the loop's own cycle may spend its budget (chat_runner.py, chat_delivery.py, state_parts/slot_ladder.py, recovery.py, turn_failures.py, turn_locals.py, autonudge_service/timers.py) name the steered carrier; turn_failures.py reads turn._wake_cycle where it held a copy, because the settle now writes it on the turn; and learn-cron-dashboard.md and monitor-architecture.md describe both paths, including that a person's turn that took the wake can bring a stand-down one cycle forward on a loop already one failure short of it.

Backwards compatibility

Compatible: every new value is optional and process-local; an automation steer without a generation behaves exactly as before.
Removes nothing: no key, row field, validator or caller is taken away; the requeued entry's AUTOMATION_STEER_META keeps its {"cycle", "loop_id"} shape.

Tests

  • test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_a_turn_that_took_a_steered_wake_charges_its_loop_when_it_dies: a person's turn takes a steered wake, then dies, and the loop is charged under its id and generation.
  • test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_a_loop_cycle_that_took_a_steered_wake_charges_the_right_cycle: a loop's own cycle takes a steered wake of the same loop at a newer generation and is charged under the newer one, and a wake of a loop that replaced it on the slot is charged to that live loop.
  • test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_one_echo_that_settles_two_wakes_charges_the_newer_generation: one echo settles a loop's wake at generation 3 and then its wake at generation 4, and the dying turn is charged under 4. The two wake texts pin their hash, so the walk the settle replaced visits generation 3 first on every run.
  • test_automation_steer_row_role.py::TestRequeuedAutomationSteer::test_the_requeued_wakes_own_turn_is_its_loops_cycle: a real steer, then the turn-end requeue and the drain: the drained turn runs with the loop's wake, and neither the steered row, the drained row nor the queue meta shows the generation.
  • test_automation_steer_row_role.py::TestRequeuedAutomationSteer::test_a_human_edit_clears_automation_origin_and_drains_as_user: an edited wake card drains without the loop's wake.
  • test_autonudge_ledger_quiet_floor.py::test_a_wake_whose_turn_fails_before_its_delivery_returns_is_retried: the loop is charged for the wake's dead turn before its delivery returns, and the next tick still retries the wake.
  • test_work_ledger_wake_delivery.py::TestSteeringAWakeIntoARunningTurn::test_a_busy_conductor_is_steered_rather_than_skipped and test_session_deliver.py::TestCallersRouteThroughDeliver::test_a_work_ledger_wake_is_delivered_as_automation: the fire's steer carries the loop's generation.

Proof: PROVEN test/test_autonudge_ledger_quiet_floor.py::test_a_wake_whose_turn_fails_before_its_delivery_returns_is_retried, test/test_autonudge_refactor_contract.py::TestSurface::test_a_service_member_keeps_its_kind_and_signature[_settle_fire_cycle], test/test_chat_runner_base_surface.py::test_a_moved_name_keeps_its_base_shape_and_owner[steer_queue:_settle_consumed_steers], test/test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_a_turn_that_took_a_steered_wake_charges_its_loop_when_it_dies, test/test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_a_loop_cycle_that_took_a_steered_wake_charges_the_right_cycle[same-loop-newer-generation] (head 59b28e9)

Determinism: the eight tests above (both cases of the parametrized one) ran 20 times at -n0 and 10 times at -n 4 --dist loadgroup --max-worker-restart=2 beside test_steer_requeue.py and test_automation_steer_row_role.py, all green, and their seven files ran green in a shuffled order (seed 6953, 895 passed). One earlier -n0 attempt hit the suite's 120 s timeout once, at the new test's teardown, while the machine's disk I/O was saturated (an unchanged neighbour's call took 105 s in that run); the 20 runs above are the repeat.

Manual verification

N/A — unit coverage sufficient: a live repro needs a provider throttle inside a running conductor turn at the moment a worker reports, and the tests drive the same runner, settle, requeue and drain seams that path takes.

Screenshots / video

N/A — no frontend change.

Related Issues

Closes #32250

Sibling sweep

Swept: git grep -n "_steer_automation" -- src/
Sibling: src/kiro_crew/dashboard/chat_delivery.py -- not affected -- its pops clear a steer that was refused, or already settled or requeued, so no turn carries that wake
Sibling: src/kiro_crew/dashboard/chat_api/turn_stop.py -- not affected -- a hard stop discards the steer with the queue, so no turn carries that wake
Sibling: src/kiro_crew/dashboard/steer_retract.py -- not affected -- a take-back removes the steer before any turn takes it
Swept: git grep -n "session_deliver.automation(" -- src/
Sibling: src/kiro_crew/slack/gateway_runtime/nudge_dashboard_fire.py -- changed here
Swept: git grep -nE "_note_ledger_wake(|_ledger_wake_streak[" -- src/
Sibling: src/kiro_crew/autonudge_service/firing.py -- changed here
Sibling: src/kiro_crew/autonudge_service/gate.py -- changed here
Swept: git grep -nE "_wake_cycle = |[_RECOVERED_WAKE_KEY] = |consumed_wake" -- src/
Sibling: src/kiro_crew/dashboard/chat_turn/steer_queue.py -- changed here
Sibling: src/kiro_crew/dashboard/chat_turn/turn_ledgers.py -- not affected -- it copies the turn's current _wake_cycle onto the recovery it queues
Sibling: src/kiro_crew/dashboard/chat_turn/turn_locals.py -- not affected -- it sets the cycle at turn start, before any steer can arrive

Pattern harvest

Not generalizable: the carrier is specific to the three ways a loop's wake reaches a turn (the fired turn, its recovery, the steer), and all three now carry it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 9, 2026 05:49
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 59b28e98df3ea1493953d0008f7a532bb7257a5d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 59b28e98df3ea1493953d0008f7a532bb7257a5d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns. The fix reuses the existing _recovered_wake / _wake_cycle carrier rather than adding a second mechanism, and the generation stays process-local.

Notes

  • When a person's turn takes the steered wake and then dies, that failure now counts against the loop's stand-down budget. The spec records this trade-off. If operators report stand-downs they did not cause, this is the first place to look.

[DESIGN-REVIEWED] 59b28e9

Description read

[DESCRIPTION-READ] 19c734cb0c2f7005ddb69446b35fb6f21024c8e5b3eedd8c0901892b0191e444

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 59b28e98df3ea1493953d0008f7a532bb7257a5d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The change fixes the linked defect (#32250) by reusing the _recovered_wake carrier and _wake_cycle from #31640, and every new value it adds is used by one caller and stays inside one process.

What this change ships

Inventory (7 items) — 7 justified

Intent: a worker's report steered into a busy conductor turn is delivered again on the next tick when that turn dies, instead of waiting out the quiet floor. This is a FIX.

  1. A turn that took a loop's steered wake and then died now counts as a failure of that loop's cycle, so the wake gets its one retry — justified
  2. A wake that was requeued before the turn took it drains into a turn that counts as the loop's cycle, and a person's edit to the card removes that — justified
  3. The retry baseline is read before the fire awaits delivery, so a carrying turn that dies early still gets its retry — justified
  4. session_deliver.automation() takes an optional generation; only the steer passes it (1 of 4 automation( callers), and no row or queue file stores it — justified
  5. When one echo settles several steers, the newest wake in send order is kept, replacing set order — justified
  6. A person's turn that took the wake can now count toward the loop's five-failure stand-down (the description says so) — justified
  7. Comments and specs describe the steered carrier — justified

[FIRST-PRINCIPLES-REVIEWED] 59b28e9

Description read

[DESCRIPTION-READ] aa4c74352892337aab19f9ff152a27a7be268e611b7db4acdcdb93b25c5b2974

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 59b28e98df3ea1493953d0008f7a532bb7257a5d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 59b28e9

Verdict parsed from the review's SHA-scoped output markers for commit 59b28e98df3ea1493953d0008f7a532bb7257a5d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 59b28e98df3ea1493953d0008f7a532bb7257a5d false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 59b28e98df3ea1493953d0008f7a532bb7257a5d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 59b28e9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 59b28e98df3ea1493953d0008f7a532bb7257a5d false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 9, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 9, 2026 06:18
buluoray
buluoray previously approved these changes Nov 9, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. A steered work-ledger wake now carries its loop's (id, generation), so a turn that took it and died, or the requeued wake's own turn, charges the loop and earns the one retry. Generation never reaches a row or the durable queue (dashboard/chat_turn/steer_queue.py:376-382, 447-450, 790-794; dashboard/chat_turn/queue_drain.py:1448-1452; dashboard/session_deliver.py:86-96; slack/gateway_runtime/nudge_dashboard_fire.py:779-781). GPT's blocking race is real but needs rare timing, and when it hits, the outcome is the pre-PR behavior (wait for the quiet floor). The adjudication step failed to run, so the block still needs the author's disposition or an override.

@bolichen97

bolichen97 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the wake's retry baseline is the failure streak read before the fire awaits delivery

  • A consumed wake still loses its retry when the turn fails before delivery returns span=aa50c1825c53

fixed in c3863d5: the fire reads the loop's failure streak just before it awaits delivery and passes it to the wake's record, so a carrying turn that dies, and is charged, before the steer call returns still grows the streak past the recorded value and earns the retry.
Test: test_autonudge_ledger_quiet_floor.py::test_a_wake_whose_turn_fails_before_its_delivery_returns_is_retried charges the loop inside the delivery and asserts the next tick retries; it fails on 15b5ecb.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 9, 2026
@bolichen97
bolichen97 force-pushed the fix/32250-steered-wake-retry branch from c3863d5 to bcd67c9 Compare November 10, 2026 18:09
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/chat_turn/steer_queue.py, a merge conflict with this PR, since this head's CI ran (2caa69b -> ed80c1d); CI re-running.

[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a turn that takes a steered wake adopts that wake's (loop id, generation) as its cycle, in place of any cycle it carries

  • A newer steered wake retains the turn's stale cycle identity span=aa50c1825c53

fixed in bcd67c9: a turn that takes a steered wake adopts the wake's (loop id, generation) in place of any cycle it already carries. A slot holds one loop and the steer names it at its current generation, so a loop revised or replaced while its cycle ran is charged under its live identity.
Test: test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_a_loop_cycle_that_took_a_steered_wake_charges_the_right_cycle (same-loop-newer-generation, replacing-loop); under c3863d5's adoption rule both cases charge the stale (loop-ledger, 3).

[operator: bolichen97]

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 10, 2026
@bolichen97
bolichen97 force-pushed the fix/32250-steered-wake-retry branch from bcd67c9 to 97f5d8e Compare November 11, 2026 02:44
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in setup.cfg and a shared test double (#34646, which clears the two main breaks behind this head's red Backend Tests shards: test_session_markdown and the code-channel trust tests) since this head's CI ran (ed80c1d -> 98d5d19); CI re-running.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 11, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 11, 2026
A work-ledger wake that lands while the conductor's turn is running is
steered into that turn and counted as delivered, so its one retry only
fires when the loop's failure streak grows. The turn that carried the
wake never charged the loop: a steer the running turn ended before
taking drained as its own turn without the loop's identity, and a
person's turn that took the steer and then died was not the loop's
cycle. Either way the loop read the ledger as unchanged and sat on its
quiet floor.

The steer now records the loop's config generation next to its
automation meta. A turn that takes the steer adopts that loop's cycle
for its fault counts (the newest wake, when one echo settles several),
and a steer requeued at a turn's end carries the loop's id and
generation on its process-local queue entry, so the turn it drains into
is charged like the cycle's own. Neither gains wake authority, nothing
new is persisted, and a person's edit of a requeued wake's card charges
no loop.

Closes #32250
@bolichen97
bolichen97 force-pushed the fix/32250-steered-wake-retry branch from 97f5d8e to 59b28e9 Compare November 11, 2026 13:14
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a settle that takes several steered wakes in one echo hands the turn the last one sent

  • Batch settlement discards the newer wake identity span=aa50c1825c53

fixed in 59b28e9: the settle walks the steers one echo settled in send order and keeps the last steered wake, so a loop's wake at generation 3 then 4 leaves the turn carrying 4, the rule a later echo already applies.
Test: test_recovery_l1_chat_runner.py::TestTheRunnerCallSitesChargeTheFailedCycle::test_one_echo_that_settles_two_wakes_charges_the_newer_generation pins both texts' hash so the old set walk meets generation 3 first; it charges 3 on 97f5d8e and 4 on the fix.
span aa50c1825c53, 3rd fix: removing or narrowing the pick drops the goal's running-turn path, so one patch with every branch checked (no wake, no generation, one, same generation, 3 then 4, a replacing loop, one left pending, two echoes).
This verdict was produced in CI but not published (the job hit a rate limit while posting), so it is answered here.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 11, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own.

The PR makes a steered work-ledger wake count as the loop's cycle in whichever turn carries it. If the running turn takes the steer, it adopts the newest settled wake's (id, generation) as _wake_cycle. If the turn ends before taking it, the requeued entry carries it on the in-memory _recovered_wake key, and the drain passes it on only while the entry still counts as automation or recovery. The generation is never persisted: the transcript row is rebuilt without it, it is removed from the queue meta, and the durable queue writer only emits a fixed list of keys. The earlier GPT blocker about a stale generation is fixed by replacing _wake_cycle outright. The retry baseline is now read before the fire's await, and the settle step does not reset the streak. Every path has a test. Only rare race-timing differences remain.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 59b28e98df · 442 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on 59b28e9. The delta since the earlier APPROVE at 97f5d8e fixes the GPT batch-settle finding. When one echo settles several steered wakes, the turn now carries the last-sent wake, so the charge uses the loop's newest generation. The rest of the PR is unchanged on the same base, and the delta is correct and tested. Decisive lines: src/kiro_crew/dashboard/chat_turn/steer_queue.py:247-255; test/test_recovery_l1_chat_runner.py:1510-1571

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@bolichen97
bolichen97 merged commit 72fc530 into main Nov 12, 2026
154 of 156 checks passed
@bolichen97
bolichen97 deleted the fix/32250-steered-wake-retry branch November 12, 2026 18:41
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A work-ledger wake steered into a running turn is not retried when that turn dies

3 participants