Skip to content

fix(taskq): take a memory park's span from one clock read - #17217

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/17106-durable-sweep-planned-span
Oct 7, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/17106-durable-sweep-planned-span

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #17106.

Problem / Motivation

Goal: The durable memory-wait sweep ends a parked row on the re-check that reaches agent.subagent_queue_max_wait_secs, not one admit wait later.

TaskStore.deferred_longer_than counts each deferred event's park from its ts to its until. The bridge computed until = store.now() + wait from its own clock read, and TaskStore.defer then read the clock again for the event's ts, so on a clock that moves between reads every park counted a little under its admit wait. Three 30 s parks summed just under a 90 s bound, the re-check that should end the row kept it, and it waited one more admit wait. #17106 fixed the same off-by-one in the gate's in-memory ledger for starts with no durable row; its First Principles review pointed at this sibling.

Why it matters

The bound is how a sub-agent stuck behind a short host gets a terminal answer. Ending it a whole admit wait late is small, but the durable path and the in-memory path should agree on when a wait has run out.

Not a goal

  • Changing what counts as parked time (lapsed deferrals and slot waits stay uncounted).
  • Changing when a parked row becomes eligible again.

What changed (motivation → approach → change)

Surface Before After
TaskStore.defer defer(task_id, until, *, reason); the event's ts is a later clock read defer(task_id, *, wait, reason) computes until = ts + wait from the one read it also stamps the event with
callers (taskq_defer, taskq_defer_posted, the admission-path deferral, RunnerAdmission.admit) now() + wait wait=wait

The sweep's float sum is unchanged. agent.admit_wait_secs and agent.subagent_queue_max_wait_secs are read as whole seconds (_safe_int in config/loader.py), and a whole-second wait lands on the float grid of an epoch-sized ts, so with one clock read (ts + wait) - ts is exactly wait and three 30 s parks sum to exactly 90 s. _append_event_in_tx takes an optional ts so the event carries the same read as until.

Backwards compatibility

Compatible: TaskStore is internal and every caller moves to wait= here; events already in a store are read the same way. A row parked by older code still counts each of those parks a few microseconds short, so its first sweep after the upgrade can end it one admit wait late, once.

Reader: src/kiro_crew/taskq/store.py:deferred_longer_than (unchanged) -- entry: subagent admission pump -- events written by older code are counted the same way; test_deferred_longer_than_counts_the_time_a_row_was_parked and test_deferred_longer_than_bounds_every_state_defer_parks still pass.

Tests

test/test_taskq_store.py:

  • test_a_bound_of_whole_admit_waits_ends_the_row_on_the_recheck_that_reaches_it: a clock that moves 1 µs on every read, a row re-checked at the end of each of three parks, and a sweep in the third re-check's own clock tick, which must return the row for a bound of three waits. Two cases: 30 s waits from start 100000.123 and from an epoch-sized start. Both starts sit far enough inside their power-of-two range that ts + 30 stays in the same binade, which is the condition the exact span needs; a start just below a power of two (such as 1000.123, where ts + 30 crosses 1024) would pass or fail on its bits alone. With the calls written the old way (store.now() + 30) against origin/main's store, both fail (assert [] == ['w']).

test/test_overload_integration_glue.py::test_admitting_id_is_held_until_the_posted_defer_landed now checks the posted deferral passes the admit wait and that the row's next_run_at equals the deferred event's ts plus that wait, which pins the single clock read end to end. The tests that passed an absolute until now pass the equivalent wait.

Run on base 397f4be: test_taskq_store.py, test_taskq_claim_fencing.py, test_taskq_runner_adapter.py, test_subagent_queue_max_wait.py, test_subagent_queued_visibility.py, test_fairness_lanes.py, test_overload_integration_glue.py, test_queue_depth_reconcile.py, test_published_queue_depth.py, test_admission_gate.py: 650 passed. black (--target-version py310), isort, flake8 and mypy on the changed files; docs_lint.py and check_comment_history.py against origin/main pass.

Manual verification

N/A -- unit coverage sufficient: the tests drive the real store on a moving clock.

Related Issues

Closes #17216

Pattern harvest

Rule candidate: review-prompt
Pattern: a duration measured between two separate clock reads (the caller's now() + wait and the callee's own now()), so a span that ran to its planned end is counted short of the plan.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested a review from a team as a code owner October 6, 2026 03:27
@jeeshofone
jeeshofone requested a review from bolichen97 October 6, 2026 03:27
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 6, 2026
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of eccb63aa5602ba6bf0adf23c6856dc2b4f3fc9c5 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The exact-span promise only holds for whole-second waits on an epoch clock. Check that no caller passes a fractional wait and relies on the bound landing exactly.

What this change ships

Inventory (4 items) — 4 justified

Intent: make a parked sub-agent get its terminal answer on the re-check that reaches the max-wait bound, not one admit wait later. FIX (#17216).

  1. A row parked by memory pressure now ends on the re-check that reaches subagent_queue_max_wait_secs, not one admit wait later. — justified
  2. TaskStore.defer takes a relative wait= instead of an absolute until. All 4 production callers move with it (grepped \.defer\b in src/), so no second spelling is left. — justified
  3. _append_event_in_tx takes an optional ts= so the deferred event carries the same clock read as until. Its one consumer is defer, and the change needs it because the base re-reads now() inside the append. — justified
  4. The taskq spec's Deferral paragraph now describes defer(task_id, *, wait, reason), edited in the same commit. — justified

[FIRST-PRINCIPLES-REVIEWED] eccb63a

Description read

[DESCRIPTION-READ] 4e2f57c8014d8feb3e184d720031dc25ba52015e936b01fc740ae5cfdd0b089e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of eccb63aa5602ba6bf0adf23c6856dc2b4f3fc9c5 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm that taskq_admit_wait_secs() always returns whole seconds in production. The exact-span guarantee in the new defer docstring holds only for whole-second waits.

[DESIGN-REVIEWED] eccb63a

Description read

[DESCRIPTION-READ] c0596824bde877325209b9ab7d66f94043f14bad2a4d3cacc382819b7b98be7c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed eccb63aa5602ba6bf0adf23c6856dc2b4f3fc9c5 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] eccb63a

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed eccb63aa5602ba6bf0adf23c6856dc2b4f3fc9c5 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] eccb63a

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 6, 2026
@jeeshofone jeeshofone changed the title fix(taskq): count a full memory park as its planned wait in the sweep fix(taskq): take a memory park's span from one clock read Oct 6, 2026
@jeeshofone
jeeshofone force-pushed the fix/17106-durable-sweep-planned-span branch from a755c4c to e15b414 Compare October 6, 2026 04:26
@jeeshofone

Copy link
Copy Markdown
Contributor Author

First Principles BLOCK on a755c4c (items 2-4, "the count still rests on two clock reads"): upheld, fixed in e15b414.

  • TaskStore.defer(task_id, wait=..., reason=...) now computes until = ts + wait from its one ts = self.now(), and the deferred event is stamped with that same ts (_append_event_in_tx takes an optional ts; before, it read the clock a third time).
  • The persisted "wait" event key, the three-tuple branch in deferred_longer_than and _PARKED_PREFILTER_SLACK_SECS are gone; the prefilter is as on main.
  • There were three bridge callers, not two: taskq_defer_posted passes store.defer by reference to _post_store_write, so a \.defer\( grep misses it. All three pass wait=wait, and test_admitting_id_is_held_until_the_posted_defer_landed checks the posted one does.
  • defer(task_id, until, ...) with an absolute time still works, for tests and callers that hold one; passing both or neither raises.
  • The sum is whole microseconds, not nanoseconds: at an epoch-sized ts the float grid is about 0.24 µs, so (ts + 30) - ts can read 30 s minus a fraction of a microsecond, which a nanosecond sum keeps and a microsecond sum absorbs.

test_a_bound_of_whole_admit_waits_ends_the_row_on_the_recheck_that_reaches_it (a clock that moves 1 µs per read, two start values) fails on origin/main's store when the calls are written the old way (store.now() + 30), and passes here.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 6, 2026
@jeeshofone
jeeshofone force-pushed the fix/17106-durable-sweep-planned-span branch from e15b414 to 67d43a2 Compare October 6, 2026 05:54
@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • Item 3 -- until has zero non-test consumers (reviewed e15b414)

    fixed in 67d43a2: defer is now defer(task_id, *, wait, reason). The until parameter, the both-or-neither ValueError, the assert until is not None and test_defer_takes_exactly_one_of_until_and_wait are gone. The four production callers (three in taskq_bridge.py, one in taskq/adapters/runner.py) already passed wait=. The tests that passed an absolute until (test_taskq_store.py, test_taskq_claim_fencing.py, test_fairness_lanes.py, test_subagent_queued_visibility.py, test_queue_depth_reconcile.py, test_subagent_queue_max_wait.py, and the defer fake in test_overload_integration_glue.py) now pass the equivalent wait; a past-due park is a negative wait. 911 passed across those suites and the other taskq and admission tests.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 6, 2026
@jeeshofone
jeeshofone force-pushed the fix/17106-durable-sweep-planned-span branch from 67d43a2 to 5b972ec Compare October 6, 2026 07:20
@jeeshofone

Copy link
Copy Markdown
Contributor Author

First Principles "Check that the epoch-sized start case fails with the one-read fix alone" (reviewed 67d43a2): checked, and it did NOT fail, so the test did not show the microsecond sum was needed. Fixed in 5b972ec.

  • Why it passed: at an epoch-sized ts the float grid is 2^-22 s, and 30 s lands on that grid, so ts + 30 is exact and a 30 s park sums to exactly 30 s in floats. The rounding only appears when the wait is off the grid.
  • test_a_bound_of_whole_admit_waits_ends_the_row_on_the_recheck_that_reaches_it now also runs 7.3 s waits from an epoch-sized start against a 21.9 s bound, and the sweep reads the third re-check's own clock tick (so no trailing microsecond of clock motion pads the sum). With deferred_longer_than put back to the float sum and the one-read defer kept, that case fails; with the microsecond sum it passes. The two 30 s cases still fail against origin/main's store with the calls written the old way.
  • The PR description's rationale said ts + 30 rounds; it now says the rounding hits waits that are off the float grid, such as 7.3 s.

Also in this head: rebased onto origin/main e1999e4 to clear the merge conflict. #17017 moved the runner's pressure deferral into RunnerAdmission._defer_for_pressure; that call now passes wait=self._admit_wait, so every store.defer caller still passes a wait and none computes now() + wait. 539 passed across the eight suites the diff touches; black, isort, flake8, mypy, docs_lint.py and the comment-history gate pass.

@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • Item 2: the microsecond sum guards fractional waits that config rejects (reviewed b5dc2db)

    Upheld, subtracted in 42e8d87. The earlier reply that kept the sum is withdrawn. admit_wait_secs and subagent_queue_max_wait_secs are read with _safe_int (config/loader.py:3110-3112), and with one clock read (ts + wait) - ts is exact for a whole-second wait, so the one-read defer(..., wait=) fixes the defect on its own.

    • deferred_longer_than is back to main's float sum: bound_us, total_us and the round(... * 1e6) are gone, and store.py now differs from main only in defer and the optional ts on _append_event_in_tx.
    • The spec sentence about summing in microseconds is gone from docs/system-specs/modules/taskq.md.
    • The 7.3 s case is gone from test_a_bound_of_whole_admit_waits_ends_the_row_on_the_recheck_that_reaches_it. Its two 30 s cases stay, and both still fail against origin/main ea7e91c's store with the calls written the old way (store.now() + 30): assert [] == ['w'].
    • The description and commit message no longer describe a microsecond sum. A caller that ever passes a fractional wait owns that precision question in its own change.

    Also in this head: rebased onto origin/main ea7e91c, which carries test: raise the owner-dm module's source-pin budget to its three reads #17290, so the test_source_pin_budget red on b5dc2db should clear. 650 passed across the ten taskq and admission suites; black, isort, flake8, mypy, docs_lint.py and the comment-history gate pass.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Design note "rows parked before this change still count a few microseconds short per park" (reviewed b5dc2db): agreed, and accepted as is. It costs one admit wait, once, on a row that was already parked when the upgrade landed; every park written after it spans exactly its wait. Rewriting stored events to correct it would be a migration for a one-time 30 s difference. The description's Backwards compatibility section now states this.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 6, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head 42e8d87: every review lane passes (First Principles, Design, Opus 5.5, GPT 6.1, UX, Security Scope). The remaining reds are outside this PR's taskq diff:

I will rebase once #17339 lands.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 6, 2026
@jeeshofone
jeeshofone force-pushed the fix/17106-durable-sweep-planned-span branch from 42e8d87 to aa9cd06 Compare October 6, 2026 15:52
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 6, 2026
@jeeshofone
jeeshofone force-pushed the fix/17106-durable-sweep-planned-span branch from aa9cd06 to 11dec34 Compare October 6, 2026 20:26
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head 11dec34 (rebased onto origin/main ffd9136). A read-through of aa9cd06 found no blocking issue and raised one test-robustness point and three nits. All are in this head:

  • test_a_bound_of_whole_admit_waits_ends_the_row_on_the_recheck_that_reaches_it: the small-clock case now starts at 100000.123. At 1000.123, ts + 30 crosses 1024, where (ts + 30) - ts is no longer exact, so that case passed on its bits. The docstring now states the same-binade condition the exact span needs.
  • TaskStore.defer docstring: "exactly wait" is scoped to whole-second waits on an epoch clock, which is what config supplies.
  • test_admitting_id_is_held_until_the_posted_defer_landed also asserts the row's next_run_at equals the deferred event's ts plus the wait, pinning the single clock read end to end.
  • taskq.md: the Deferral paragraph is rewrapped at 78 columns.

The 10 listed test files on base ffd9136: 650 passed. black gate, isort, flake8, mypy (store.py), docs_lint.py and the comment-history gate pass. This push also re-runs Backend Lint, which was cancelled on aa9cd06 by the repo-wide mypy slowdown.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@jeeshofone
jeeshofone force-pushed the fix/17106-durable-sweep-planned-span branch from 11dec34 to ad2ffa2 Compare October 7, 2026 02:46
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head ad2ffa2 (11dec34 rebased onto origin/main 5b2aee2; git range-diff shows the commit unchanged). Every review lane passed on 11dec34 (Design, First Principles, GPT 6.1, Opus 5.5; UX skipped, no UI surface).

The one red there, Backend Tests (Windows) (4) test/test_stale_asset_watchdog.py::test_an_update_that_stayed_up_keeps_a_managed_gateway_up_without_a_check (AssertionError: assert [] at line 990), is outside this PR's taskq diff and has no tracking issue; a job re-run needs admin rights, so this rebased head is its one retry. The 10 taskq and admission test files pass on this head (650 passed).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 7, 2026
TaskStore.deferred_longer_than ends a memory-parked row once its parked time
reaches agent.subagent_queue_max_wait_secs, counting each park from its
deferred event's ts to its until. The bridge computed until as now() + wait
from its own clock read, and defer then read the clock again for the event's
ts, so every park counted a little under its admit wait. Three 30 s parks
summed under a 90 s bound, and the row waited one more admit wait.

defer(task_id, *, wait, reason) now computes until = ts + wait from the one
read it also stamps the event with, and every caller (the three bridge paths
and the task runner's admission) passes wait. The admit and max waits are whole
seconds, so with one read ts + wait - ts is exactly wait and the sweep's float
sum is unchanged.

Follow-up to kirodotdev#17106.

Closes kirodotdev#17216
@jeeshofone
jeeshofone force-pushed the fix/17106-durable-sweep-planned-span branch from ad2ffa2 to eccb63a Compare October 7, 2026 05:16
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head eccb63a is ad2ffa2 rebased onto origin/main 397f4be. git range-diff shows the commit unchanged. The rebase gives a fresh run to Backend Lint & Type Check (3.12), which was cancelled on ad2ffa2 while mypy was still running at the job ceiling. On this head, the ten taskq and admission test files named in the description pass (650 passed), and black, isort, flake8 and mypy pass on the changed files.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 7, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving (crew-pr-reviewer). defer now takes wait= and stamps both the deadline and the event from one now() read. A parked row therefore ends on the re-check that reaches the configured max wait, not one admit wait later. Every caller is switched over. CI is green and the base is main.

@iamwhatever
iamwhatever enabled auto-merge (squash) October 7, 2026 06:46
@iamwhatever
iamwhatever merged commit 4456354 into kirodotdev:main Oct 7, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Durable memory-wait sweep ends a row one admit wait after a whole-wait bound

2 participants