Skip to content

fix(cron): drop the oldest queued entry for a script cron too - #23117

Merged
KiroCrewCaptain merged 2 commits into
kirodotdev:mainfrom
leonlaiyc:claude/script-cron-queue-evict
Oct 23, 2026
Merged

KiroCrewCaptain merged 2 commits into
kirodotdev:mainfrom
leonlaiyc:claude/script-cron-queue-evict

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A script cron that reports to a busy dashboard session keeps that session's live queue within its cap, as a cron's send_message already does.

A script cron's result that reaches a dashboard session while a turn runs there is queued behind the turn with no bound. A cron's send_message to its origin session drops the oldest entry once the live queue holds MAX_LIVE_QUEUE_ENTRIES (50). The script-cron path does not, so it keeps appending (#23116).

Why it matters

A script cron that fires faster than its notification turns run (for example, every minute against a session busy with a long task) adds a queued card and a future turn on every fire, for as long as the session stays busy. The queue, the cards in the tab and the backlog of turns grow with no limit. The cap that is meant to stop this, documented as shared by every producer, is skipped by one of the two cron senders.

Not a goal

What changed (motivation → approach → change)

  • Symptom: the live queue of a busy session grows past 50 entries when a script cron reports into it.
  • Root cause: _deliver_script_result (slack/gateway_runtime/cron_exec.py) calls session_deliver.deliver(..., session_deliver.cron(label)) without evict_oldest=True. So _deliver_running queues the notification without checking the cap. The Design Review on refactor(dashboard): route a cron's send_message through session_deliver.deliver #23052 counted it: 1 of 7 deliver( callers sets the flag, and the script-cron caller "takes the default, so a full live queue drops the oldest entry for one cron sender and not the other." The gap is older than refactor(dashboard): route a cron's send_message through session_deliver.deliver #23052. The script path's earlier slot.running branch also called queue_cron_notification without a cap check.
  • Change: the script-cron caller passes evict_oldest=True. monitor-architecture.md now says both cron senders evict.
  • Open dashboards (GPT review on 6993ec2): the shared eviction arm in dashboard/session_deliver.py:_deliver_running removed the entry and its card on the server but sent no frame. The dashboard draws queued cards from WebSocket frames, so an open tab kept every evicted card until reload. The arm now broadcasts queue_cancel with the slot and the evicted entry's id, the frame a person's cancel (chat_api/prompt_queue.py) and a rewind (chat_rewind.py) already send. The arm is shared, so a cron's send_message eviction, which already existed, sends the frame too. injected-messages.md says so.
  • Only a cron notification gives way (Opus review on 6b77aa9): the arm took slot.queue_pop(0), the oldest entry of any kind. While sub-agents run, a person's prompt is held at the head and system entries drain past it, so a fast script cron could push that prompt out, and the new queue_cancel would take its card down as well. The arm now evicts the oldest queued entry of kind cron_notification that no person edited. An edit sets _directive_user_origin (or _directive_channel_origin) but keeps the cron kind, and the words are the editor's (GPT review on b738971). When nothing may give way, the arm writes nothing, logs a warning and returns QUEUE_FULL.
  • Both cron callers read QUEUE_FULL (Opus review on b738971): before, neither checked the result, so a refused report still counted as delivered. _deliver_script_result now sends it as a dashboard notification, as it already does when the session is gone, and the job counts as delivered only through that. api_send_message leaves sent_session False, so the message takes _deliver_send_message_fallback and the audit row does not record a session delivery that never happened. The docstring, injected-messages.md and monitor-architecture.md say so.

Backwards compatibility

Compatible: below the cap nothing changes. At the cap, a script cron's oldest queued report and its card are dropped instead of the queue growing, the same thing a cron's send_message already does at that cap. Only a queued cron notification is evicted. A person's prompt, an app message or a peer send is never dropped to make room, and a cron's send_message, which could evict any entry until now, gets the same rule. A cron card a person edited is never evicted either. On a full queue with nothing that may give way, the new report leaves the session: a script cron's result becomes a dashboard notification, where before it was appended past the cap, and a cron's send_message takes its usual fallback, where before it evicted the head entry.

Reader: src/kiro_crew/dashboard/chat_runner.py:_start_next_queued_turn -- entry: cron -- it drains whatever entries remain; an evicted entry is removed from slot._queue and its card from slot.messages together (_remove_queued_by_id), so no card is left pointing at a missing entry (test_script_cron_on_a_full_live_queue_drops_the_oldest_entry_and_card).

Reader: website/src/hooks/useWebSocket.ts queue_cancel -- entry: the evicted card -- cancelQueuedMessage removes the card whose meta.queueId the frame names, the same handling a person's cancel already gets.

Tests

Proof: PROVEN test/test_cron_queued_card_retires.py::test_script_cron_on_a_full_live_queue_drops_the_oldest_entry_and_card, test/test_cron_queued_card_retires.py::test_a_full_live_queue_drops_a_cron_report_before_a_persons_prompt, test/test_cron_queued_card_retires.py::test_a_live_queue_full_of_other_entries_drops_the_new_cron_report, test/test_cron_queued_card_retires.py::test_a_cron_card_a_person_edited_is_not_evicted, test/test_cron_queued_card_retires.py::test_a_script_cron_result_the_full_queue_refuses_goes_out_as_a_notification (head 201216e)

test/test_cron_queued_card_retires.py::test_script_cron_on_a_full_live_queue_drops_the_oldest_entry_and_card drives the real GatewayOrchestrator._deliver_script_result into a slot with a running turn 51 times. It asserts the live queue holds reports 1–50, and that the queued cards are exactly those 50, so the oldest entry and its card left together. It also asserts exactly one queue_cancel frame, naming the slot and the evicted entry's id.

  • test_a_full_live_queue_drops_a_cron_report_before_a_persons_prompt: a person's prompt at the head of a full queue stays. The oldest cron report is evicted instead, and the queue_cancel frame names that report.
  • test_a_live_queue_full_of_other_entries_drops_the_new_cron_report: with 50 queued prompts and no cron entry, every prompt stays, no card is drawn and no frame is sent.
  • test_a_cron_card_a_person_edited_is_not_evicted: with the oldest of 50 reports edited through queue_edit_by_id, the next oldest report is evicted instead.
  • test_a_script_cron_result_the_full_queue_refuses_goes_out_as_a_notification and test_a_send_message_the_full_queue_refuses_takes_the_fallback: on a queue of 50 prompts, the script result reaches notify and the send_message reaches _deliver_send_message_fallback.

Red on main: the queue holds 51 entries and keeps report 0 (Left contains one more item: 'report 50'). Red without the broadcast (checked locally): AssertionError: assert [] == [{'slot': ..., 'queue_id': ...}]. Red with 6b77aa9's any-kind eviction: both new tests fail, because the head prompt is the one evicted. Red with b738971's source: all three tests added this round fail (assert [] == [('cron', 'watch report')], assert [] == [None], and the edited card is the one evicted).

Local results on Windows, Python 3.13:

  • test_cron_queued_card_retires.py, test_cron_origin_injection.py, test_session_deliver.py, test_slack_gateway_cron_exec_coverage.py, test_cron_locking_regression.py, test_slack_gateway_composition_contract.py, test_chat_turn_timeout_consistency.py and test_flake_pattern_ratchet.py: 481 passed. On the current head, every test file that references session_deliver, evict_oldest, queue_cron_notification or _deliver_script_result (15 files), plus test_flake_pattern_ratchet.py: 769 passed. On 201216e, the same set plus every file that references api_send_message: 1142 passed, 2 failed. The two failures are test_dashboard_file_io.py's test_read_serves_reserved_characters[is it done?.md] (Windows forbids ? in a file name) and test_read_announces_a_lossy_decode_in_a_header (CRLF on read). Both are in a file this PR does not touch.
  • Gates: black, isort, flake8 on session_deliver.py and the test, mypy on cron_exec.py, session_deliver.py and handlers/messaging.py, check_sync_io_in_async.py, comment-history, brand name, and docs_lint.py are all clean.

Manual verification

N/A — unit coverage sufficient: the test runs the real delivery path end to end against a real slot and queue. Only the scheduler's timing is not exercised, and this PR does not change it.

Related Issues

Fixes #23116

Follow-up to #23052 (Design Review, Notes, Item 3).

Sibling sweep

Swept: git grep -n "_remove_queued_by_id(" -- src (every server-side removal of a queued card)
Sibling: src/kiro_crew/dashboard/session_deliver.py:215 -- changed here
Sibling: src/kiro_crew/dashboard/chat_api/prompt_queue.py:62 -- not affected -- broadcasts queue_cancel after the removal
Sibling: src/kiro_crew/dashboard/chat_turn/queue_drain.py:779 -- not affected -- broadcasts queue_pop just before the removal
Sibling: src/kiro_crew/dashboard/chat_turn/recovery.py:467 -- not affected -- broadcasts queue_pop when the removal hits
Sibling: src/kiro_crew/dashboard/chat_turn/steer_queue.py:589 -- not affected -- broadcasts queue_pop right after the removal
Sibling: src/kiro_crew/slack/gateway_runtime/cron_exec.py:471 -- changed here (the missing evict_oldest)

Pattern harvest

Rule candidate: review-prompt
Pattern: a server-side removal of a queued card (_remove_queued_by_id) with no queue_cancel/queue_pop frame, which leaves open dashboards showing the card

The missing cap does not generalize on its own: its class is "a producer of a capped shared queue that skips the cap". After this change both cron callers of session_deliver.deliver pass the flag, the app caller refuses at the cap itself, and the peer and automation origins never reach the cron queue arm. The structural fix, eviction as the cron origin's default with no flag, belongs to the #22747 delivery series.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

@leonlaiyc
leonlaiyc requested review from a team and KiroCrewCaptain as code owners October 22, 2026 15:14
@leonlaiyc
leonlaiyc requested a review from CrysisDeu October 22, 2026 15:14
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention fork Pull request from a fork (external contributor) labels Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 201216e4588fae8bc47be6aae968f03da9430bb2 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 201216e4588fae8bc47be6aae968f03da9430bb2 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix sends the script-cron caller through the cap that send_message already uses, and both cron callers now treat QUEUE_FULL as not delivered and fall back.

Notes

  • When the queue is full, a fast script cron can evict a queued report from a different, slower cron job on the same slot, because the eviction picks the oldest cron notification of any job. If that report matters, prefer evicting the same job's oldest report first; this can be a follow-up.
  • The new queue_cancel frame from eviction carries no content, while a person's cancel (prompt_queue.py:81) does. Rewind (chat_rewind.py:749) already sends it without one, so this works today. Any app-SDK listener that reads content from this frame should handle it being missing.

[DESIGN-REVIEWED] 201216e

Description read

[DESCRIPTION-READ] 83fe4df912c2efc7994225bee56e0d5fdd63d964f3fda8c05b835dbd4037ff7e

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 201216e4588fae8bc47be6aae968f03da9430bb2 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 201216e

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 201216e4588fae8bc47be6aae968f03da9430bb2 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

Notes:

  • When the queue is full and api_send_message takes the fallback, the bell still adds "(session closed — delivered as notification)", but the session is open and only its queue is full.

[OPUS-REVIEWED] 201216e

A script cron's result delivered to a busy dashboard session appended
to the live queue with no cap, while a cron's send_message drops the
oldest entry at MAX_LIVE_QUEUE_ENTRIES. A script firing faster than its
notification turns ran grew the queue for as long as the session
stayed busy. Pass evict_oldest so both cron senders share one rule.

The shared eviction arm removed the entry and its card server-side but
sent no frame, so an open dashboard kept the evicted card until reload.
Broadcast queue_cancel for it, as a person's cancel and a rewind do.

Fixes kirodotdev#23116

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leonlaiyc
leonlaiyc force-pushed the claude/script-cron-queue-evict branch from 6993ec2 to 6b77aa9 Compare October 23, 2026 07:56
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 201216e4588fae8bc47be6aae968f03da9430bb2 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The cap fix is the cause-level change. The narrower eviction rule contradicts the frozen ## Not a goal, but the PR's own change exposes a person's prompt to eviction, so fixing that is in scope.

Notes

  • Item 3 — oversized: ## Not a goal says "Changing the overflow rule… This PR only makes the script path use it", but the hunk replaces slot.queue_pop(0) with the kind-filtered next(...) scan and a return QUEUE_FULL refusal, which changes the rule for send_message too. It is justified because, with this PR, a script cron can evict a held prompt. A human may want the Goal amended to say so.

What this change ships

Inventory (5 items) — 4 justified

Intent: a busy dashboard session's queue stays within its 50-entry cap when a script cron reports into it. This is a FIX.

  1. A script cron reporting into a busy session drops the oldest queued report at 50 entries — justified
  2. An open dashboard tab removes the evicted card immediately, not on reload (queue_cancel frame, also for send_message) — justified
  3. At the cap, only an unedited cron card is evicted; a person's prompt, an app message, a peer send or an edited card is never dropped (this changes send_message too) — rides along; contradicts Not a goal
  4. A script cron result the full queue refuses shows up as a dashboard notification — justified
  5. A send_message the full queue refuses takes the existing fallback and is no longer recorded as delivered to the session — justified

[FIRST-PRINCIPLES-REVIEWED] 201216e

Description read

[DESCRIPTION-READ] 5920ae62030fe4987d37e4c1d60ae56d92bd42e971be28db257cf3e2b479f403

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026
The shared eviction arm took the oldest entry of any kind. With
sub-agents running, a person's prompt is held at the queue head while
system entries drain past it, so a script cron firing faster than its
turns could push that prompt out, and the queue_cancel frame then took
its card down too. Evict the oldest queued cron notification instead,
skipping one a person edited, since its words are theirs though it
keeps the cron kind.

When nothing may give way the arm writes nothing and returns
QUEUE_FULL. Both cron callers now read that: a script cron's result
goes out as a dashboard notification, as when the session is gone, and
a cron's send_message takes its usual fallback instead of reporting
the session delivery as done.

Refs kirodotdev#23116

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leonlaiyc
leonlaiyc force-pushed the claude/script-cron-queue-evict branch from b738971 to 201216e Compare October 23, 2026 09:54
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 23, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 23, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 23, 2026 12:03

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (6 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: a script cron's result on a full live queue now evicts only the oldest unedited cron notification (with a queue_cancel frame), never a person's prompt, and falls back to a notification when nothing may give way; covered by new tests. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/injected-messages.md, docs/system-specs/modules/monitor-architecture.md.

@KiroCrewCaptain
KiroCrewCaptain merged commit 4eb2984 into kirodotdev:main Oct 23, 2026
105 of 113 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A script cron's result grows a busy session's live queue past its cap

2 participants