Skip to content

test(taskq): recover the task store after a real kill of its owner - #19933

Merged
javenciu merged 1 commit into
mainfrom
test/store-conformance-restart-family
Oct 14, 2026
Merged

javenciu merged 1 commit into
mainfrom
test/store-conformance-restart-family

Conversation

@javenciu

@javenciu javenciu commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Prove the task store's restart recovery against a process that is really killed, so a change to claim, lease or reconcile code fails a test when a real kill would leave rows the repair cannot settle.

The task store's restart tests stand in for a crash by closing the store cleanly and reopening the file: _seed_crash_scenario ends in s.close() # crash (test/test_taskq_reconcile.py:64), and the retry test closes the store with the comment "the run takes its gateway down with it" (test/test_taskq_reconcile.py:130). Those tests prove "given these rows, reconcile does X". They do not prove that a killed owner leaves those rows. The crew log already has the stronger shape: test/test_crew_log_real_crash.py:3 notes that every other crash test in the suite writes the damaged file itself, and test/fixtures/crew_log_crash_child.py:9 shows the way out, a child that drives the production writer, announces a failpoint with a terminal marker value, and waits for SIGKILL.

Why it matters

reconcile_on_boot (src/kiro_crew/taskq/reconcile.py) decides what happens to a row whose owner died: requeue it, recover it, park it in unknown_side_effect, settle it done from a result artifact, or fail it once its recoveries are spent. A wrong decision either re-runs a side effect that already happened or drops work. Tests that never kill an owner cannot see a regression in the state a kill really leaves.

Not a goal

  • No change under src/. The tests call existing public functions only.
  • No other store, and no store-neutral protocol. The child has the task store's failpoints only; a shared protocol belongs in the change that adds a second store.
  • No schema-upgrade, partial-write, quarantine or crew-log lease properties.
  • No edit to test/test_taskq_claim_fencing.py or test/test_taskq_reconcile.py. They stay as the in-process tests.

What changed (motivation → approach → change)

The restart tests need a real owner that dies, so the approach is a child process and a real signal, with every other input fixed.

  • test/fixtures/crash_child.py opens the task store, drives its public methods to a named failpoint (claimed, running, result-written, or claim-race), writes the exact value "<store>:<failpoint>" to a marker file in one call, and blocks until it is killed. Its inputs arrive in a declared environment (CRASH_DIR, CRASH_CLOCK, CRASH_TASK, CRASH_CLASS, CRASH_GATE, CRASH_WHO, CRASH_IDS).
  • test/taskq_real_crash/harness.py starts the child (spawn), waits for the marker's exact value through kiro_crew.testing.wait.wait_until (wait_announced), sends SIGKILL (kill), and always reaps (reap). run_to_failpoint_and_kill chains them. race holds several children on a flock until each has its store open, so their claims overlap. declared_env names the child's whole environment and never inherits the runner's. A child that exits before its failpoint fails the test and quotes its own output. The wait ceiling (MARKER_TIMEOUT, 60 s) bounds a failure; a passing run returns as soon as the marker holds its value.
  • The child runs with -B and PYTHONDONTWRITEBYTECODE=1 (spawn, declared_env). It imports the package from the checkout, and a plain interpreter start would write __pycache__ directories beside the repository's sources.
  • test/taskq_real_crash/reopen.py is reopen_store, which opens tasks.db again as the next incarnation. The clock is injected in the parent and the child, so no test reads wall time.
  • test/test_taskq_real_crash.py holds the tests.

Restart properties, each with a real kill and a reopen as a new incarnation:

  • test_kill_between_claim_and_start_requeues: a claimed row that never started goes back to queued and is claimable again (side-effect classes none and unknown).
  • test_kill_mid_run_unknown_class_lands_unknown_side_effect: a running row of class unknown lands in unknown_side_effect and is never claimed again; classes none and idempotent_key go to recovering instead.
  • test_kill_after_result_written_before_ack_settles_done: with a result file on disk and an artifact probe, the row settles done. The same kill with no result file does not.
  • test_third_lost_owner_ends_failed: each kill of the owner spends one recovery; after RECOVERY_MAX_ATTEMPTS recoveries the next lost owner ends the row failed with reconciled: lost_owner_limit, and a settled row stays settled.
  • test_running_row_with_expired_lease_is_not_reclaimable: lease expiry gates claim only; a dead owner's running row waits for reconcile.
  • test_cancelled_row_never_revives_after_restart: a cancel that lands after a kill stays a cancel through the next restart, even when the probe says done.

Two more tests need real processes. test_claim_has_one_winner_across_processes starts three children on one file and twelve rows, and every row is claimed once at generation 1. test_a_child_that_exits_before_its_failpoint_fails_the_test_with_its_output covers an unknown failpoint and an unknown store: an early exit is never read as a kill.

Rejected alternatives: keeping the clean-close stand-in for new restart cases, because it cannot show what a dead owner leaves behind; repeating the in-process claim, fencing and cancel cases here, because they need no kill and test/test_taskq_claim_fencing.py already pins them; a store-neutral protocol with an adapter and a property table, because with one store it only restates the test names.

Backwards compatibility

Compatible: the change adds test files only, so nothing that works today stops working.

Removes nothing: no existing test, fixture, helper or source file is edited or deleted.

Tests

In short: 13 tests in test/test_taskq_real_crash.py; the restart and claim-race tests start a child process that is killed or raced. They pass 20 of 20 runs at -n0, 10 of 10 runs at -n 4 beside the two neighbouring task store test files, and 5 of 5 runs in a shuffled order; the module cannot import without the harness, and each kill test's control fails the assertion it guards.

Red, because these tests cover existing behaviour:

  1. The module with the harness package absent fails at collection: ModuleNotFoundError: No module named 'taskq_real_crash'.
  2. Each kill test has a control. Held to the expectation the real test guards, in a scratch copy that is not committed, every control fails on that assertion:
kill at "running", no result file, held to == done
    AssertionError: assert 'unknown_side_effect' == 'done'
kill at "running", class none, held to == unknown_side_effect
    AssertionError: assert 'recovering' == 'unknown_side_effect'
kill at "running", held to == admitted
    AssertionError: the kill should leave the claim in place
    assert 'running' == 'admitted'
RECOVERY_MAX_ATTEMPTS kills (one short of the limit), held to == failed
    AssertionError: assert (TaskRecord(id='t1', ...
      - failed
      + recovering)
kill, no cancel, probe says done, held to == cancelled
    AssertionError: assert 'done' == 'cancelled'
expired lease on a running row, held to a successful claim
    AssertionError: an expired lease must not let a running row be stolen
    assert None is not None

(The TaskRecord repr is elided.)

In the committed tests the same controls are parametrizations: [running-unknown_side_effect] for the result file, and [none-recovering] and [idempotent_key-recovering] for the replay-safe classes.

Green, on origin/main plus this commit:

  • python -m pytest -n0 test/test_taskq_real_crash.py, 20 runs: 13 passed each (about 4 s of test time and 5 s of wall time; every restart test starts and kills a real process).
  • -n 4 --dist loadgroup with test/test_taskq_claim_fencing.py and test/test_taskq_reconcile.py, 10 runs: 70 passed each.
  • The 13 node ids in a random order at -n0, 5 runs: 13 passed each.
  • Ratchets at -n0, FLAKE_RATCHET_BASE=origin/main: test_wait_discipline_ratchet, test_flake_pattern_ratchet, test_pid_reader_ratchet, test_tmpdir_helpers, test_dir_fd_create_race_ratchet, test_spawn_audit, test_kill_chokepoint_ratchet and test_nodeid_length_guard all pass.
  • Files that enumerate test_*.py across test/ (test_leaf_test_scope.py, test_coverage_omit_contract.py, test_real_adapter_gate.py, test_queue_receipt.py, and the test_every_manager_connect_test_pins_the_port_probe test in test_instances.py) pass with the new module present, and so do scripts/run_scoped_tests.py --test and scripts/leaf_test_scope.py --test.
  • The new module patches no module global; it runs first in one process ahead of test/test_taskrunner_atomic_persistence.py, test/test_taskrunner_execution_capture_async.py, test/test_unattended_slot_guardrails.py and test/test_wheel_apply.py, and all pass.
  • Gates: black, isort, flake8 and mypy on the touched files; check_black_formatting, check_subprocess_encoding, check_comment_history and check_brand_name on the merge base; check_agent_sdk_boundary, check_sync_io_in_async, check_loop_bound_locks, check_testpaths_coverage, check_cwd_relative_repo_reads and check_acp_frame_host_data all exit 0.
  • Bytecode: no .pyc file or __pycache__ directory appeared under src/, test/ or scripts/ during any of the runs above. In a scratch copy with the -B argument and PYTHONDONTWRITEBYTECODE removed from the harness, the same module leaves 10 __pycache__ directories under src/.

Size: 5 files, about 600 added lines, all under test/.

Platform: the real-kill tests need SIGKILL and flock, so the module skips on win32, as the crew log real-crash tests do.

Manual verification

N/A — unit coverage sufficient: the change is test code and the runs above are the evidence.

Screenshots / video

N/A — no user-visible surface.

Related Issues

None closed by this change. It builds on the crew log's real-kill tests (#15898).

Pattern harvest

Not generalizable: this adds coverage and fixes no defect.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — not applicable, no documented behaviour changes
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 13, 2026 20:46
@javenciu
javenciu requested a review from mbriones98 October 13, 2026 20:46
@javenciu

Copy link
Copy Markdown
Contributor Author

Closing and reopening right after opening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683).

@javenciu javenciu closed this Oct 13, 2026
@javenciu javenciu reopened this Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 1a3315cbf588b733b45febef78ee89efbd75df5d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 1a3315cbf588b733b45febef78ee89efbd75df5d ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 1a3315cbf588b733b45febef78ee89efbd75df5d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1a3315c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1a3315cbf588b733b45febef78ee89efbd75df5d: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 1a3315cbf588b733b45febef78ee89efbd75df5d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the test/taskq_real_crash/ package imports in CI the same way overload_fakes does, from the test/ rootdir and with no conftest path tweak.

[DESIGN-REVIEWED] 1a3315c

Description read

[DESCRIPTION-READ] 9060b79a82d6a313c1177cf01739f1279846ed8a8eae8d01a444c55cec0793e7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 1a3315cbf588b733b45febef78ee89efbd75df5d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1a3315c

Verdict parsed from the review's SHA-scoped output markers for commit 1a3315cbf588b733b45febef78ee89efbd75df5d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1a3315cbf588b733b45febef78ee89efbd75df5d: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

fixed in 9483a26: the crash child now starts with -B and PYTHONDONTWRITEBYTECODE=1 in its declared environment (test/store_conformance/harness.py, declared_env and spawn). Witness: on the previous head one restart test left 10 __pycache__ directories under src/; on this head the full suite, run three times, leaves none. 23 passed each run; the wait-discipline and flake-pattern ratchets pass (92).

The task store's restart tests stand in for a crash by closing the store
cleanly and reopening the file, so they show what reconcile does with
rows a test wrote by hand, not what a killed owner leaves behind.

A child process (test/fixtures/crash_child.py) opens the store, drives
its public methods to a named failpoint, announces the failpoint by
writing a terminal marker value in one call, and blocks. The parent
waits for that exact value through kiro_crew.testing.wait.wait_until,
ends the child with SIGKILL, reopens tasks.db as the next incarnation
and runs reconcile_on_boot. Parent and child share one injected clock,
so no test reads wall time. The child runs with -B and
PYTHONDONTWRITEBYTECODE, so it leaves no __pycache__ under src/.

The tests cover a kill between claim and start, a kill mid-run, a kill
after the result is written and before the ack, the recovery limit, a
running row whose lease has expired, and a cancel that survives a
restart. Three child processes also race to claim the same rows. A child
that exits before its failpoint fails the test and quotes its output.
@javenciu
javenciu force-pushed the test/store-conformance-restart-family branch from 9483a26 to 1a3315c Compare October 13, 2026 22:45
@javenciu javenciu changed the title test(taskq): add a real-kill conformance harness and restart family test(taskq): recover the task store after a real kill of its owner Oct 13, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author
  1. The seven in-process copies are deleted: stale_generation_write_rejected_and_logged, terminal_never_regresses, late_completion_after_cancel_ignored, cancel_before_claim_wins, cancel_after_claim_fences_dispatcher, claim_has_one_winner_under_threads and claim_skips_unexpired_lease each repeated a test in test/test_taskq_claim_fencing.py, which stays their only home.
  2. The store-neutral protocol is deleted with its adapter, capability flags, property table and three meta tests. What ships is the real-kill harness, reopen_store, and 13 tests (six restart properties after a SIGKILL, the three-process claim race, and the harness's early-exit check). With no shared protocol left, the title drops "conformance", the package and module are renamed test/taskq_real_crash/ and test/test_taskq_real_crash.py, and the body describes only what ships; the second Not a goal bullet no longer claims a store-neutral harness, and Goal and Why it matters are unchanged. The new head is 1a3315cbf5, one commit on main at af8a5db86e.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=052d626f0dd8

fixed in 1a3315c: the finding is right and the subtraction is taken as written. The seven copies of tests from test/test_taskq_claim_fencing.py are deleted, and test/test_taskq_real_crash.py keeps 13 tests, each needing a killed process, a process race, or the harness's early-exit check.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=9c418dc6d2d8

fixed in 1a3315c: the finding is right and the subtraction is taken as written. protocol.py, TaskqAdapter, the capability flags, STORE_CAPABILITIES, PROPERTIES and the three meta tests are deleted; reopen_store stays in test/taskq_real_crash/reopen.py, and 13 tests remain.

@javenciu

Copy link
Copy Markdown
Contributor Author

Closing and reopening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683).

@javenciu javenciu closed this Oct 13, 2026
@javenciu javenciu reopened this Oct 13, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 13, 2026
@javenciu
javenciu enabled auto-merge (squash) October 14, 2026 01:04

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: adds a real-SIGKILL restart harness and tests for the task store; test files only.

@javenciu
javenciu merged commit f5adcd7 into main Oct 14, 2026
103 of 185 checks passed
@javenciu
javenciu deleted the test/store-conformance-restart-family branch October 14, 2026 01:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants