Repository navigation
test(taskq): recover the task store after a real kill of its owner - #19933
Conversation
|
Closing and reopening right after opening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683). |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the [DESIGN-REVIEWED] 1a3315c Description read[DESCRIPTION-READ] 9060b79a82d6a313c1177cf01739f1279846ed8a8eae8d01a444c55cec0793e7 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
The task store's restart tests stand in for a crash by closing the store cleanly and reopening the file, so they show what reconcile does with rows a test wrote by hand, not what a killed owner leaves behind. A child process (test/fixtures/crash_child.py) opens the store, drives its public methods to a named failpoint, announces the failpoint by writing a terminal marker value in one call, and blocks. The parent waits for that exact value through kiro_crew.testing.wait.wait_until, ends the child with SIGKILL, reopens tasks.db as the next incarnation and runs reconcile_on_boot. Parent and child share one injected clock, so no test reads wall time. The child runs with -B and PYTHONDONTWRITEBYTECODE, so it leaves no __pycache__ under src/. The tests cover a kill between claim and start, a kill mid-run, a kill after the result is written and before the ack, the recovery limit, a running row whose lease has expired, and a cancel that survives a restart. Three child processes also race to claim the same rows. A child that exits before its failpoint fails the test and quotes its output.
9483a26 to
1a3315c
Compare
|
|
span=052d626f0dd8
|
|
span=9c418dc6d2d8
|
|
Closing and reopening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683). |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: adds a real-SIGKILL restart harness and tests for the task store; test files only.
Problem / Motivation
Goal: Prove the task store's restart recovery against a process that is really killed, so a change to claim, lease or reconcile code fails a test when a real kill would leave rows the repair cannot settle.
The task store's restart tests stand in for a crash by closing the store cleanly and reopening the file:
_seed_crash_scenarioends ins.close() # crash(test/test_taskq_reconcile.py:64), and the retry test closes the store with the comment "the run takes its gateway down with it" (test/test_taskq_reconcile.py:130). Those tests prove "given these rows, reconcile does X". They do not prove that a killed owner leaves those rows. The crew log already has the stronger shape:test/test_crew_log_real_crash.py:3notes that every other crash test in the suite writes the damaged file itself, andtest/fixtures/crew_log_crash_child.py:9shows the way out, a child that drives the production writer, announces a failpoint with a terminal marker value, and waits forSIGKILL.Why it matters
reconcile_on_boot(src/kiro_crew/taskq/reconcile.py) decides what happens to a row whose owner died: requeue it, recover it, park it inunknown_side_effect, settle itdonefrom a result artifact, or fail it once its recoveries are spent. A wrong decision either re-runs a side effect that already happened or drops work. Tests that never kill an owner cannot see a regression in the state a kill really leaves.Not a goal
src/. The tests call existing public functions only.test/test_taskq_claim_fencing.pyortest/test_taskq_reconcile.py. They stay as the in-process tests.What changed (motivation → approach → change)
The restart tests need a real owner that dies, so the approach is a child process and a real signal, with every other input fixed.
test/fixtures/crash_child.pyopens the task store, drives its public methods to a named failpoint (claimed,running,result-written, orclaim-race), writes the exact value"<store>:<failpoint>"to a marker file in one call, and blocks until it is killed. Its inputs arrive in a declared environment (CRASH_DIR,CRASH_CLOCK,CRASH_TASK,CRASH_CLASS,CRASH_GATE,CRASH_WHO,CRASH_IDS).test/taskq_real_crash/harness.pystarts the child (spawn), waits for the marker's exact value throughkiro_crew.testing.wait.wait_until(wait_announced), sendsSIGKILL(kill), and always reaps (reap).run_to_failpoint_and_killchains them.raceholds several children on aflockuntil each has its store open, so their claims overlap.declared_envnames the child's whole environment and never inherits the runner's. A child that exits before its failpoint fails the test and quotes its own output. The wait ceiling (MARKER_TIMEOUT, 60 s) bounds a failure; a passing run returns as soon as the marker holds its value.-BandPYTHONDONTWRITEBYTECODE=1(spawn,declared_env). It imports the package from the checkout, and a plain interpreter start would write__pycache__directories beside the repository's sources.test/taskq_real_crash/reopen.pyisreopen_store, which openstasks.dbagain as the next incarnation. The clock is injected in the parent and the child, so no test reads wall time.test/test_taskq_real_crash.pyholds the tests.Restart properties, each with a real kill and a reopen as a new incarnation:
test_kill_between_claim_and_start_requeues: a claimed row that never started goes back toqueuedand is claimable again (side-effect classesnoneandunknown).test_kill_mid_run_unknown_class_lands_unknown_side_effect: a running row of classunknownlands inunknown_side_effectand is never claimed again; classesnoneandidempotent_keygo torecoveringinstead.test_kill_after_result_written_before_ack_settles_done: with a result file on disk and an artifact probe, the row settlesdone. The same kill with no result file does not.test_third_lost_owner_ends_failed: each kill of the owner spends one recovery; afterRECOVERY_MAX_ATTEMPTSrecoveries the next lost owner ends the rowfailedwithreconciled: lost_owner_limit, and a settled row stays settled.test_running_row_with_expired_lease_is_not_reclaimable: lease expiry gatesclaimonly; a dead owner's running row waits for reconcile.test_cancelled_row_never_revives_after_restart: a cancel that lands after a kill stays a cancel through the next restart, even when the probe saysdone.Two more tests need real processes.
test_claim_has_one_winner_across_processesstarts three children on one file and twelve rows, and every row is claimed once at generation 1.test_a_child_that_exits_before_its_failpoint_fails_the_test_with_its_outputcovers an unknown failpoint and an unknown store: an early exit is never read as a kill.Rejected alternatives: keeping the clean-close stand-in for new restart cases, because it cannot show what a dead owner leaves behind; repeating the in-process claim, fencing and cancel cases here, because they need no kill and
test/test_taskq_claim_fencing.pyalready pins them; a store-neutral protocol with an adapter and a property table, because with one store it only restates the test names.Backwards compatibility
Compatible: the change adds test files only, so nothing that works today stops working.
Removes nothing: no existing test, fixture, helper or source file is edited or deleted.
Tests
In short: 13 tests in
test/test_taskq_real_crash.py; the restart and claim-race tests start a child process that is killed or raced. They pass 20 of 20 runs at-n0, 10 of 10 runs at-n 4beside the two neighbouring task store test files, and 5 of 5 runs in a shuffled order; the module cannot import without the harness, and each kill test's control fails the assertion it guards.Red, because these tests cover existing behaviour:
ModuleNotFoundError: No module named 'taskq_real_crash'.(The
TaskRecordrepr is elided.)In the committed tests the same controls are parametrizations:
[running-unknown_side_effect]for the result file, and[none-recovering]and[idempotent_key-recovering]for the replay-safe classes.Green, on
origin/mainplus this commit:python -m pytest -n0 test/test_taskq_real_crash.py, 20 runs: 13 passed each (about 4 s of test time and 5 s of wall time; every restart test starts and kills a real process).-n 4 --dist loadgroupwithtest/test_taskq_claim_fencing.pyandtest/test_taskq_reconcile.py, 10 runs: 70 passed each.-n0, 5 runs: 13 passed each.-n0,FLAKE_RATCHET_BASE=origin/main:test_wait_discipline_ratchet,test_flake_pattern_ratchet,test_pid_reader_ratchet,test_tmpdir_helpers,test_dir_fd_create_race_ratchet,test_spawn_audit,test_kill_chokepoint_ratchetandtest_nodeid_length_guardall pass.test_*.pyacrosstest/(test_leaf_test_scope.py,test_coverage_omit_contract.py,test_real_adapter_gate.py,test_queue_receipt.py, and thetest_every_manager_connect_test_pins_the_port_probetest intest_instances.py) pass with the new module present, and so doscripts/run_scoped_tests.py --testandscripts/leaf_test_scope.py --test.test/test_taskrunner_atomic_persistence.py,test/test_taskrunner_execution_capture_async.py,test/test_unattended_slot_guardrails.pyandtest/test_wheel_apply.py, and all pass.check_black_formatting,check_subprocess_encoding,check_comment_historyandcheck_brand_nameon the merge base;check_agent_sdk_boundary,check_sync_io_in_async,check_loop_bound_locks,check_testpaths_coverage,check_cwd_relative_repo_readsandcheck_acp_frame_host_dataall exit 0..pycfile or__pycache__directory appeared undersrc/,test/orscripts/during any of the runs above. In a scratch copy with the-Bargument andPYTHONDONTWRITEBYTECODEremoved from the harness, the same module leaves 10__pycache__directories undersrc/.Size: 5 files, about 600 added lines, all under
test/.Platform: the real-kill tests need
SIGKILLandflock, so the module skips on win32, as the crew log real-crash tests do.Manual verification
N/A — unit coverage sufficient: the change is test code and the runs above are the evidence.
Screenshots / video
N/A — no user-visible surface.
Related Issues
None closed by this change. It builds on the crew log's real-kill tests (#15898).
Pattern harvest
Not generalizable: this adds coverage and fixes no defect.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.