Repository navigation
Conversation
03eb664 to
d3405d2
Compare
|
🤖 Kiro Crew [operator: bolichen97#bb3ad1ca]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author. Assessment: Merge conflict needs a rebase onto main; backend shards fail deterministically on If you'd prefer I don't touch this PR, add the |
d3405d2 to
d636183
Compare
|
🤖 Kiro Crew [operator: bolichen97#bb3ad1ca] Drive-to-green update: rebased and fixed the three assessed blockers. Head is now 1. Merge conflict → rebased onto
2.
3.
Local gates (all green)
Coverage Gate should clear on its own once the backend shards go green (fail-closed behind them; no Now monitoring CI; will address any review-bot findings next. |
|
🤖 Kiro Crew [operator: bolichen97#bb3ad1ca] PR Hygiene fix (zero-push): the check requires fix-type PRs to carry a |
ec329fc to
b1cfd5e
Compare
|
Rebased onto main Conflicts and resolutions:
Gates run locally on the changed files: black, isort, flake8, and Please review the resolution. A maintainer pushed last, so under the repo's last-push rule a second approver is needed. Reply if anything looks wrong. |
b1cfd5e to
aa95831
Compare
|
Rebased onto current main by a maintainer. Conflicts resolved:
Tests: |
… loop Move caller-controlled spec validation/writes and plan-directory creation/cleanup to worker threads while preserving the current workflow lifecycle and rollback semantics. Retain side-effect workers across cancellation so a late write or mkdir cannot strand a handler-owned spec or plan directory after the request loses ownership. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…ompliant
The off-loop refactor (by Leon, leonlaiyc) folded the two spec-path error
responses into one variable-status site, which the error-code contract
scanner buckets as a new `dynamic_status` finding, and removing the two
prose-only sites made the taskrunner baseline entry stale.
- `_validate_spec_path` now returns a machine-readable failure code
(`invalid_spec_path` / `access_denied`) instead of prose + status, so
the blocking work stays off-loop and the handler owns the responses.
- The handler emits two fully literal, compliant responses:
`{"error": "invalid spec path", "code": "invalid_spec_path"}` (400) and
`{"error": "access denied", "code": "access_denied"}` (403), matching
the existing convention (e.g. handlers/prompts.py).
- Re-snapshotted error-code-baseline.json per the ratchet's documented
procedure: taskrunner missing_code 43 -> 41, a legitimate drop only
(totals 1204 -> 1202; no bucket raised).
Round 2 (rebase + review findings):
- Rebased onto main 6ade333 (baseline regenerated on the new tree).
- Adopted the First Principles finding: /start's cleanup now catches
BaseException (mirroring from_chat's rollback), so a request cancelled
during start_background no longer orphans the created inline spec;
Exception still maps to the same 400 response. New mutation-verified
test covers the cancel-during-start window.
Round 3 (GPT blocking finding, verified against taskrunner.py source):
- start_background registers the run placeholder and then awaits
_workflow_begin outside its rollback try, so a cancel there retains a
run whose spec the round-2 cleanup would unlink. Cleanup is now gated
on ownership: _spec_retained_by_run() checks whether any registered
run references the created spec, and the handler never removes a spec
a run retains (this also covers the pre-existing Exception-path hole
where a _workflow_begin failure retained the run yet the spec was
unlinked). Mutation-verified both directions: gate dropped -> the new
retained-run test fails; BaseException narrowed -> the orphan test
fails.
Co-authored-by: Kiro Crew <noreply@kirodotdev.github.io>
aa95831 to
e0fc36c
Compare
|
Correcting my own earlier push on this branch. An automated rebase I ran ~20 minutes ago pushed Conflicts resolved, and why
One thing I tried and backed out, worth knowing if you touch this again: I briefly added Tests run (
|
bolichen97
left a comment
There was a problem hiding this comment.
Requesting changes: this head drops the WorkflowInitializing -> 503 mapping from POST /api/taskrunner
PR #10573 added a typed 503 branch to the except block in api_taskrunner_start, and this head removes it. The merge base (40ac65f, which contains PR #10573) ends the block like this:
if isinstance(exc, WorkflowInitializing):
return web.json_response({"error": str(exc), "code": exc.code}, status=503)
return web.json_response({"error": str(exc)}, status=400)At this head, src/kiro_crew/dashboard/handlers/taskrunner.py:408-410 has only this:
if not isinstance(exc, Exception):
raise # CancelledError and friends: propagate after cleanup.
return web.json_response({"error": str(exc)}, status=400)What breaks
-
While the gateway is still restoring workflows,
start_backgroundcalls_require_workflow_ready(), which raisesWorkflowInitializing(src/kiro_crew/taskrunner.py:870-875). The code isworkflow_initializing, orworkflow_initialization_failedafter a failed init. POST /api/taskrunner now returns a 400 with nocode. Before this change it returned a retryable 503 with a stable code.docs/system-specs/modules/taskrunner.md:88-91anddocs/system-specs/modules/workflows.md:807document the 503. The PR doesn't update either spec and doesn't list this as a behaviour change. -
Who is affected: the website and the MCP
task_runtool only showerror, so the text users see is the same. The loss is the status andcodecontract, which API clients and anything that retries on 503 depend on. -
Existing tests fail on this head with
assert 400 == 503:test/test_workflows_startup.py::test_dashboard_task_reads_and_cancel_remain_available_during_init(:682)test_task_handler_readiness_errors_are_typed[False-start-start_background]and[False-start_inline-start_background](:876)
They pass on origin/main, and main's copies are unchanged, so merging this would turn main's backend suite red.
This most likely happened when the branch was re-derived from b1cfd5ec on 2026-09-21. That commit does not contain PR #10573, and because the lost lines sit inside the except body, a function-level diff against main would not show them.
Fix: put the branch back after the if not isinstance(exc, Exception): raise line and before the generic 400:
if isinstance(exc, WorkflowInitializing):
return web.json_response({"error": str(exc), "code": exc.code}, status=503)The ownership-gated cleanup above it can stay as it is. After that, rebase and regenerate error-code-baseline.json, which currently conflicts with main.
From a full open-PR audit against origin/main c66f95b (2026-09-25); findings were checked by independent reviewers reading the code. Reply here if this is wrong.
|
Superseded by #14303, an in-repo continuation of this change (the taskrunner spec/plan-directory I/O moved off the gateway loop), with @leonlaiyc credited as co-author on both commits. #14303 is green (PR Readiness passed). Closing this in favor of it; the fork branch is kept. |
Pull request was closed
Problem / Motivation
AUTOSDE.yaml'sno-blocking-call-on-event-looprule (blocking: true,file-patterns: src/kiro_crew/**/*.py) names "large synchronous file IO orfilesystem walks" as calls that must not run on the gateway loop, and closes
with "When in doubt, offload — a leaked worker thread is survivable; a frozen
loop is not." Two task-runner handlers do their filesystem work inline anyway.
api_taskrunner_start:api_taskrunner_from_chat:Five blocking syscalls, all on the loop, and the two riskiest take their input
straight from the request body:
write_textcommits an arbitrary,size-unbounded inline spec, and
resolve()/is_file()run against whateverpath the caller named — a UNC path to a disconnected share resolves at the
mount's timeout, not at RAM speed.
Why it matters
The rule states the consequence directly: "a single blocking call there freezes
every task — the user's chat turn AND the liveness heartbeat — until the
watchdog kills the process and the supervisor respawns into the same condition
(a crash loop). This has caused multiple production wedges."
Starting a task from a spec is an ordinary dashboard action, so this is not an
exotic path. The inline-spec write scales with a body the caller controls, and
the path guard's cost is set by a filesystem the caller points at. The rollback
rmdirandunlinkare individually small, but they sit on the failure path,which is exactly where a wedged volume is most likely to be the reason the
operation failed in the first place.
What changed (motivation → approach → change)
Root cause is simply that the offload was never applied here — the module knows
the rule and cites it by name in
_gate_auto_approve, which already routes itssynchronous SEL flush through
asyncio.to_thread. Each site moves to a worker,grouped so that a sequence costs one hop rather than one hop per call:
_validate_spec_path(raw)— the resolve, the..check, theis_filestat and the
is_sensitive_pathdenylist move into one function returning(validated_path, failure_code)— the handler maps the code to its literalerror response. The validation logic is unchanged; what changes is
that it runs in a worker and the handler receives only a path that passed.
That is the same property the inline comment asked for ("no gap where
spec_path could differ from what was checked"), and grouping strengthens it:
the guard and the value it produces can no longer be separated at all, and no
awaitlands between them._write_inline_spec(work_dir, content)— the name mint, themkdirandthe
write_textin one hop._make_plan_dir(work_dir)— the collision-retry loop stays with themkdirit retries, so N collisions still cost one hop instead of N.unlinkand the rejected-planrmdirare awaited throughasyncio.to_thread; both keep their existingexcept OSErrorbest-efforthandling, and the
from_chatrollback still re-raises the originalValueErrorso the handler's 400 is unchanged.Behaviour deltas (deliberate, driven by CI gates and review):
codefields(
invalid_spec_path/access_denied), per the error-code contract gate andthe existing convention (
handlers/prompts.py);error-code-baseline.jsonre-snapshotted for the legitimate drop (taskrunner
missing_code43 -> 41).created (previously orphaned): the
/startcleanup catchesBaseException(mirroring the
from_chatrollback), so a cancel duringstart_backgroundcleans up the inline spec before the cancellation propagates.
Status codes, response ordering, and best-effort cleanup semantics are
otherwise unchanged.
Tests
New
TestSpecIoRunsOffTheEventLoopintest/test_handlers_taskrunner_coverage.py. Each test spies on the realpathlibmethod (filtered to the file or directory the handler owns, so anunrelated call cannot decide the result), records
threading.get_ident(), andasserts no call landed on the test coroutine's thread — the property holds
however the handler reaches the filesystem, and does not depend on the shape of
the fix:
test_inline_spec_write_runs_off_the_event_loop_thread(write_text, alsoasserts the file's contents so an offload that lost the write fails)
test_spec_path_guard_runs_off_the_event_loop_thread(is_file, driven witha real spec file so the guard passes and the handler returns 200)
test_rejected_start_removes_the_inline_spec_off_the_loop(unlink, withstart_backgroundraising)test_from_chat_plan_dir_mkdir_runs_off_the_event_loop_thread(mkdir)test_from_chat_rollback_rmdir_runs_off_the_event_loop_thread(rmdir, withupdate_planraisingValueError; still asserts the 400)test_cancelled_inline_write_does_not_orphan_spec(cancel during theshielded write; the drained worker's file is removed, no
TASK_*.mdorphan)test_cancelled_start_backgrounding_does_not_orphan_spec(cancel after thewrite, during
start_background; theBaseExceptioncleanup removes thespec before the cancel propagates)
test_cancelled_plan_claim_does_not_orphan_directory(cancel during theplan-directory claim; no
plan_*orphan, run map left empty)Red-before, measured against pristine
origin/mainproduction code(
ad0825392) with the new tests in place — 5 failed / 100 passed, everyfailure the ident comparison, i.e. the defect itself and not a fixture or
environment artifact:
Green-after: 105 passed in that module. Blast radius: 451 passed / 1
skipped across
test_handlers_taskrunner_coverage.py,test_taskrunner.py,test_taskrunner_coverage.py,test_taskrunner_autoapprove.py,test_taskrunner_atomic_persistence.pyandtest_taskrunner_v2_scenarios.py.flake8,isortandmypyare clean on both files. Both are on.github/black-baseline.txtand keep the same pre-existing hunk count beforeand after; the added lines are
black-clean.Manual verification
N/A — unit coverage sufficient: the property is which thread a syscall runs on,
observed directly at the
pathlibcall. Reproducing the stall by hand needs adeliberately wedged volume, which the thread assertion pins without one.
Related Issues
Self-reported while auditing
AUTOSDE.yaml'sno-blocking-call-on-event-looprule against the dashboard handlers. Noseparate issue was filed.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Pattern harvest
Rule candidate: an aiohttp handler that creates a filesystem side effect it owns (temp spec file, plan directory) must (1) run the blocking create off the event loop (
asyncio.to_thread), and (2) treat cancellation as an ownership hazard — shield the worker, drain it to completion onCancelledError, then remove the created artifact before re-raising, or a cancelled request orphans the artifact with no owner left holding its path. Grep candidates:write_text|mkdir|unlink|rmdirdirectly insideasync def api_*handlers.