Repository navigation
feat(doctor): name task rows that are not terminal, unleased and long past due - #22270
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check before merge: a [DESIGN-REVIEWED] 7ad84e2 Description read[DESCRIPTION-READ] 4c9973ffb47ec02ab375fd56ce6c744e3c6ff79e40932f315b8e44bf0650e7ef That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that the one-hour bound cannot flag a busy gateway's real backlog: a row stuck behind a full cap for over an hour is recorded as an issue. What this change shipsInventory (6 items) — 6 justifiedIntent: let a user find, using
[FIRST-PRINCIPLES-REVIEWED] 7ad84e2 Description read[DESCRIPTION-READ] 9c270d26ba4c5f646b073e42f6fb193ec4c5c4e05f5ce0f4f10fc310f4c6fd81 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Disposition of the advisory finding the Design Review and First Principles Review both raised on 0cba345 (a stopped gateway makes every deferred row more than an hour past due read as stranded): fixed in 9ff59fd. Doctor now reads the gateway lock holder, the oracle the Memory Pressure section uses. With no gateway running the rows are listed as advisory, no issue is recorded, and the lines say the next start claims due rows, so run doctor again once the gateway has been up a few minutes. A running gateway, or a lock probe that cannot tell, keeps the warning and the issue. Tests: |
9ff59fd to
7ad84e2
Compare
|
Dispositions for the First Principles Review on 9ff59fd; addressed in 7ad84e2 (squashed to one commit, per the two-commit gate). Blocker and item 5 (a third spelling of the gateway-lock probe): fixed. Item 4 (the no-gateway branch undeclared): the description was updated after that review read it; "What changed" now states both branches. Item 2 ( Semgrep |
|
Dispositions for the advisory notes on 7ad84e2 (both lanes PASS): Design Review (a First Principles Review (the one-hour bound can name a busy gateway real backlog): accepted as the intended reading. A row still unclaimed an hour after it fell due, with a gateway running, is worth a look even when a full cap is the cause, and each line names the row state so the user can tell a backlog from a stopped run. The bound is one constant ( |
iamwhatever
left a comment
There was a problem hiding this comment.
APPROVE at 7ad84e2.
The new read is one parameterized, read-only query on the store doctor already opens with diagnostic=True (store_reads.py L435-L459). If the read is refused, doctor prints not checked for this check alone and the counts still print. Every value read from disk goes through safe_terminal_line. A gateway probe that fails or cannot tell keeps the warning, and a stopped gateway gets an advisory with no issue (workload.py L170-L222). The Design and First Principles checks on this head have sound replies, the out-of-scope items are tracked in #22252 and #22248, and every check is green.
— crew-pr-reviewer (r113)
Problem / Motivation
Goal:
kirocrew doctornames the task-queue rows that nothing is picking up, so a user can find the row that keeps a chat slot answering 409 without opening the task store by hand.The doctor's task-store section prints the pending and active counts and the oldest wait. A row that is not terminal, holds no lease and fell due hours ago shows up there only as one more pending row. That is the shape a lost terminal write leaves behind (a stopped sub-agent parked in
retry_waitwith no owner, #17158), and it keeps its parent slot answering409 slot_subagents_runningon every guarded action. Today finding it means readingtasks.dbby hand.Why it matters
The user sees a slot that refuses Continue, Resume and every switch, and doctor, the tool they run to find out why, says nothing about the row that causes it.
Not a goal
retry_waitcase).What changed (motivation → approach → change)
Gap: no read answers "which rows has nothing claimed long after they fell due". Approach: one SQL read on the store, and the doctor's task-store section renders it.
TaskStore.overdue_unleased_rows(overdue_secs)(taskq/store_reads.py, bound intaskq/store.py): every row whose state is not terminal, whoselease_owneris empty, and whosenext_run_atis set and at or beforenow - overdue_secs, oldest due first. A row never deferred (next_run_atNULL) is not counted, and a row that holds a lease (a live owner, orwaiting_infra, which keeps its lease) is not either.doctor_checks/workload.py:_doctor_task_storecalls it with_STRANDED_OVERDUE_SECS = 3600on the samediagnostic=Truestore it already opens, and_report_stranded_rowsnames the first five rows by id, state, session key and hours overdue (+N morefor the rest). Each value read off disk goes throughsafe_terminal_line. A read the store refuses printsstranded rows: not checked (...)and adds no issue, and the counts above it still print.cli_doctor._read_gateway_pidandresources._gateway_lock_indeterminate), the rows are a warning that says why such a row keeps its parent slot answering 409, plus one doctor issue. A probe that raises keeps the warning too.docs/system-specs/modules/taskq.mddescribes the read and the doctor lines.One hour leaves room for a due row that waits behind a full execution cap; a stranded row stays due for as long as nothing settles it.
Relation to #22252 (which addresses #22244): that PR settles, at boot, an unleased
retry_waitrow whose tombstone proves it ended. Every row it settles is one this check names, so after a restart this check reports only what the reconcile could not prove, plus the other non-terminal states. The two touch different files (taskq/reconcile.pythere;taskq/store_reads.py,taskq/store.pyanddoctor_checks/workload.pyhere), so this is based on main.Backwards compatibility
Compatible: doctor gains one line block, printed only when such a row exists, and one issue only when a gateway runs (or may be running). A healthy store prints exactly what it printed before.
Removes nothing:
TaskStoregains a read; no key, field, function or validator is removed or tightened.Tests
Failing first on origin/main de80cdc, passing with the change (
test/test_doctor_stranded_taskq_rows.py):test_the_read_returns_only_unleased_non_terminal_rows_long_past_due: a row claimed, started and released toretry_waitsix hours ago is returned; a row due ten minutes ago, a queued row never deferred, a leasedwaiting_infrarow and a terminal row with an old due time are not (red on main: no such read).test_doctor_names_a_stranded_row_and_leaves_the_store_untouched: doctor prints the row's id,retry_wait, its session key andoverdue 6.0h, records the issue, and every row's state, generation, lease, due time and event count are unchanged afterwards (red on main: no stranded line).test_doctor_caps_the_named_rows_and_counts_the_rest: seven stranded rows name the five most overdue and print+2 more(red on main).test_doctor_prints_no_stranded_warning_for_a_healthy_store: a queued row, a row due a minute ago and a cancelled row print no warning and no issue.test_a_refused_stranded_read_is_reported_unread_and_the_counts_still_print.test_with_no_gateway_running_the_rows_are_advisory_and_no_issue,test_an_indeterminate_gateway_probe_keeps_the_warningandtest_a_failed_gateway_probe_keeps_the_warningpin the gateway readings; the other doctor tests pin the probe to "running".test_store_taskq_bindings.pygains the new member's inventory row;test_cli_doctor_refactor_sections.py::test_the_task_store_rows_and_warnings, which stubs every store read the section makes, stubs this one too.Run locally: all
test/test_taskq*.py,test/test_cli_doctor*.pyandtest/test_doctor*.pyfiles plus the Static Ratchet Gate's files (1756 passed). black, isort, flake8 and mypy on the touched files; the comment-history, brand, feature-map, harness-parity, memory-store-seam, loop-bound-locks, testpaths-coverage and docs-lint gates.Manual verification
N/A: unit coverage sufficient. The doctor tests run
_doctor_task_storeagainst a realtasks.dbunder a temporaryKIROCREW_HOME, built through the real store's claim and transition writes.Related Issues
Fixes #22247
Pattern harvest
Not generalizable: one diagnostic read over one store; no rule other code could break the same way.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.