Skip to content

feat(doctor): name task rows that are not terminal, unleased and long past due - #22270

Merged
iamwhatever merged 1 commit into
mainfrom
feat/22247-doctor-stranded-taskq-rows
Oct 20, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
feat/22247-doctor-stranded-taskq-rows

Conversation

@cruisercohen

@cruisercohen cruisercohen commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: kirocrew doctor names the task-queue rows that nothing is picking up, so a user can find the row that keeps a chat slot answering 409 without opening the task store by hand.

The doctor's task-store section prints the pending and active counts and the oldest wait. A row that is not terminal, holds no lease and fell due hours ago shows up there only as one more pending row. That is the shape a lost terminal write leaves behind (a stopped sub-agent parked in retry_wait with no owner, #17158), and it keeps its parent slot answering 409 slot_subagents_running on every guarded action. Today finding it means reading tasks.db by hand.

Why it matters

The user sees a slot that refuses Continue, Resume and every switch, and doctor, the tool they run to find out why, says nothing about the row that causes it.

Not a goal

What changed (motivation → approach → change)

Gap: no read answers "which rows has nothing claimed long after they fell due". Approach: one SQL read on the store, and the doctor's task-store section renders it.

  • TaskStore.overdue_unleased_rows(overdue_secs) (taskq/store_reads.py, bound in taskq/store.py): every row whose state is not terminal, whose lease_owner is empty, and whose next_run_at is set and at or before now - overdue_secs, oldest due first. A row never deferred (next_run_at NULL) is not counted, and a row that holds a lease (a live owner, or waiting_infra, which keeps its lease) is not either.
  • doctor_checks/workload.py: _doctor_task_store calls it with _STRANDED_OVERDUE_SECS = 3600 on the same diagnostic=True store it already opens, and _report_stranded_rows names the first five rows by id, state, session key and hours overdue (+N more for the rest). Each value read off disk goes through safe_terminal_line. A read the store refuses prints stranded rows: not checked (...) and adds no issue, and the counts above it still print.
    • With a live gateway holding the home's lock, or when the lock probe cannot tell (the pair the Memory Pressure section composes: cli_doctor._read_gateway_pid and resources._gateway_lock_indeterminate), the rows are a warning that says why such a row keeps its parent slot answering 409, plus one doctor issue. A probe that raises keeps the warning too.
    • With no gateway running they are advisory and record no issue: a stopped gateway leaves every deferred row past due, and the next start claims them, so the lines say to run doctor again once the gateway has been up a few minutes.
  • docs/system-specs/modules/taskq.md describes the read and the doctor lines.

One hour leaves room for a due row that waits behind a full execution cap; a stranded row stays due for as long as nothing settles it.

Relation to #22252 (which addresses #22244): that PR settles, at boot, an unleased retry_wait row whose tombstone proves it ended. Every row it settles is one this check names, so after a restart this check reports only what the reconcile could not prove, plus the other non-terminal states. The two touch different files (taskq/reconcile.py there; taskq/store_reads.py, taskq/store.py and doctor_checks/workload.py here), so this is based on main.

Backwards compatibility

Compatible: doctor gains one line block, printed only when such a row exists, and one issue only when a gateway runs (or may be running). A healthy store prints exactly what it printed before.

Removes nothing: TaskStore gains a read; no key, field, function or validator is removed or tightened.

Tests

Failing first on origin/main de80cdc, passing with the change (test/test_doctor_stranded_taskq_rows.py):

  • test_the_read_returns_only_unleased_non_terminal_rows_long_past_due: a row claimed, started and released to retry_wait six hours ago is returned; a row due ten minutes ago, a queued row never deferred, a leased waiting_infra row and a terminal row with an old due time are not (red on main: no such read).
  • test_doctor_names_a_stranded_row_and_leaves_the_store_untouched: doctor prints the row's id, retry_wait, its session key and overdue 6.0h, records the issue, and every row's state, generation, lease, due time and event count are unchanged afterwards (red on main: no stranded line).
  • test_doctor_caps_the_named_rows_and_counts_the_rest: seven stranded rows name the five most overdue and print +2 more (red on main).
  • test_doctor_prints_no_stranded_warning_for_a_healthy_store: a queued row, a row due a minute ago and a cancelled row print no warning and no issue.
  • test_a_refused_stranded_read_is_reported_unread_and_the_counts_still_print.
  • test_with_no_gateway_running_the_rows_are_advisory_and_no_issue, test_an_indeterminate_gateway_probe_keeps_the_warning and test_a_failed_gateway_probe_keeps_the_warning pin the gateway readings; the other doctor tests pin the probe to "running".

test_store_taskq_bindings.py gains the new member's inventory row; test_cli_doctor_refactor_sections.py::test_the_task_store_rows_and_warnings, which stubs every store read the section makes, stubs this one too.

Run locally: all test/test_taskq*.py, test/test_cli_doctor*.py and test/test_doctor*.py files plus the Static Ratchet Gate's files (1756 passed). black, isort, flake8 and mypy on the touched files; the comment-history, brand, feature-map, harness-parity, memory-store-seam, loop-bound-locks, testpaths-coverage and docs-lint gates.

Manual verification

N/A: unit coverage sufficient. The doctor tests run _doctor_task_store against a real tasks.db under a temporary KIROCREW_HOME, built through the real store's claim and transition writes.

Related Issues

Fixes #22247

Pattern harvest

Not generalizable: one diagnostic read over one store; no rule other code could break the same way.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 7ad84e23cf09cb4800a32b69e1c9b05cd96ae100 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7ad84e23cf09cb4800a32b69e1c9b05cd96ae100 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check before merge: a queued row that was never deferred has next_run_at NULL, so this check skips it, even if it is stuck and holding its parent slot at 409.

[DESIGN-REVIEWED] 7ad84e2

Description read

[DESCRIPTION-READ] 4c9973ffb47ec02ab375fd56ce6c744e3c6ff79e40932f315b8e44bf0650e7ef

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 7ad84e23cf09cb4800a32b69e1c9b05cd96ae100 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that the one-hour bound cannot flag a busy gateway's real backlog: a row stuck behind a full cap for over an hour is recorded as an issue.

What this change ships

Inventory (6 items) — 6 justified

Intent: let a user find, using kirocrew doctor, the task row nobody is picking up that keeps a chat slot answering 409. ADDITION (a read-only diagnostic).

  1. Doctor's task-store section now names unleased, non-terminal rows that fell due more than an hour ago, showing id, state, session and hours overdue, five at most plus +N more — justified
  2. While a gateway runs, or the lock probe can't tell or fails, those rows print as a warning with one doctor issue — justified
  3. With no gateway running, the same rows print as advisory with no issue and a "run doctor again" hint — justified
  4. When the store refuses the read, doctor prints stranded rows: not checked (...) and the counts above still print — justified
  5. New TaskStore.overdue_unleased_rows read, with one consumer (doctor_checks/workload.py:141) — justified
  6. The taskq spec describes the read and the doctor lines — justified

[FIRST-PRINCIPLES-REVIEWED] 7ad84e2

Description read

[DESCRIPTION-READ] 9c270d26ba4c5f646b073e42f6fb193ec4c5c4e05f5ce0f4f10fc310f4c6fd81

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7ad84e23cf09cb4800a32b69e1c9b05cd96ae100 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7ad84e2

Verdict parsed from the review's SHA-scoped output markers for commit 7ad84e23cf09cb4800a32b69e1c9b05cd96ae100.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7ad84e23cf09cb4800a32b69e1c9b05cd96ae100: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7ad84e23cf09cb4800a32b69e1c9b05cd96ae100 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7ad84e2

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7ad84e23cf09cb4800a32b69e1c9b05cd96ae100: <one-sentence reason>

@cruisercohen

Copy link
Copy Markdown
Contributor Author

Disposition of the advisory finding the Design Review and First Principles Review both raised on 0cba345 (a stopped gateway makes every deferred row more than an hour past due read as stranded): fixed in 9ff59fd. Doctor now reads the gateway lock holder, the oracle the Memory Pressure section uses. With no gateway running the rows are listed as advisory, no issue is recorded, and the lines say the next start claims due rows, so run doctor again once the gateway has been up a few minutes. A running gateway, or a lock probe that cannot tell, keeps the warning and the issue. Tests: test_with_no_gateway_running_the_rows_are_advisory_and_no_issue, test_an_indeterminate_gateway_probe_keeps_the_warning, test_the_gateway_probe_reads_the_lock_holder.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@cruisercohen
cruisercohen force-pushed the feat/22247-doctor-stranded-taskq-rows branch from 9ff59fd to 7ad84e2 Compare October 19, 2026 20:10
@cruisercohen

Copy link
Copy Markdown
Contributor Author

Dispositions for the First Principles Review on 9ff59fd; addressed in 7ad84e2 (squashed to one commit, per the two-commit gate).

Blocker and item 5 (a third spelling of the gateway-lock probe): fixed. _gateway_holds_its_lock is deleted. _report_stranded_rows now reads the pair the Memory Pressure section composes, cli_doctor._read_gateway_pid() is None and not resources._gateway_lock_indeterminate(). A probe that raises keeps the warning, as that section prints its own "probe failed" line rather than "not running".

Item 4 (the no-gateway branch undeclared): the description was updated after that review read it; "What changed" now states both branches.

Item 2 (overdue_unleased_rows(overdue_secs) takes a bound its one caller always passes): kept as a parameter. The bound is doctor reporting policy, and the store keeps none of its callers policy: the sibling read deferred_longer_than(kind, bound) takes its bound from its caller the same way. Folding 3600 into the store would put a doctor threshold in taskq.

Semgrep mock-pid-allocatable on the test double: fixed, the running-gateway pin now uses UNALLOCATABLE_PID.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@cruisercohen

Copy link
Copy Markdown
Contributor Author

Dispositions for the advisory notes on 7ad84e2 (both lanes PASS):

Design Review (a queued row never deferred has next_run_at NULL, so the check skips it): kept, by the criterion #22247 sets ("long past next_run_at"). A never-deferred queued row is claimable at once, so a running gateway dispatches it on its next refill; measuring it from created_at instead would report ordinary backlog waiting behind a full cap. A queued row that never moves with no due time would be a dispatcher defect of a different shape, outside this report.

First Principles Review (the one-hour bound can name a busy gateway real backlog): accepted as the intended reading. A row still unclaimed an hour after it fell due, with a gateway running, is worth a look even when a full cap is the cause, and each line names the row state so the user can tell a backlog from a stopped run. The bound is one constant (_STRANDED_OVERDUE_SECS) if it proves noisy.

@cruisercohen
cruisercohen marked this pull request as ready for review October 19, 2026 21:00
@cruisercohen
cruisercohen requested review from a team and KiroCrewCaptain as code owners October 19, 2026 21:00
@cruisercohen
cruisercohen enabled auto-merge (squash) October 19, 2026 21:00
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at 7ad84e2.

The new read is one parameterized, read-only query on the store doctor already opens with diagnostic=True (store_reads.py L435-L459). If the read is refused, doctor prints not checked for this check alone and the counts still print. Every value read from disk goes through safe_terminal_line. A gateway probe that fails or cannot tell keeps the warning, and a stopped gateway gets an advisory with no issue (workload.py L170-L222). The Design and First Principles checks on this head have sound replies, the out-of-scope items are tracked in #22252 and #22248, and every check is green.

— crew-pr-reviewer (r113)

@iamwhatever
iamwhatever merged commit a3c95a3 into main Oct 20, 2026
109 of 117 checks passed
@iamwhatever
iamwhatever deleted the feat/22247-doctor-stranded-taskq-rows branch October 20, 2026 01:18
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kirocrew doctor: flag task-queue rows that are not terminal, hold no lease and are long past next_run_at

2 participants