Skip to content

fix: hold the remaining dropped asyncio tasks until they finish - #26463

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/25953-hold-remaining-tasks
Oct 31, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/25953-hold-remaining-tasks

Conversation

@JLargent42

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Every background task Kiro Crew starts and does not await runs to its end instead of being garbage-collected part way.

After the four area PRs (#25968, #25971, #25984, #25986), ten sites on main still start a task with asyncio.create_task, loop.create_task or asyncio.ensure_future and keep no strong reference to it. The event loop holds only a weak reference to a running task, so a task parked on an await that nothing else reaches can be collected mid-run, and Python logs Task was destroyed but it is pending!.

Why it matters

The work silently never finishes. A collected crew-log flush never runs its done callback, so _flushing stays set and the crew log publisher stops broadcasting until restart. A collected shared-crew cancel hook leaves the child or run it was cancelling still running. A collected SIGTERM retire leaves a remote worker running after it was told to stop. A collected sprites terminal send drops a keystroke or a resize.

Not a goal

  • A shared background-task helper. The area PRs each used a module-level set with add_done_callback(set.discard), and this PR follows the same pattern or joins a set the module already has.
  • Changing what any task does or when it starts.

What changed (motivation → approach → change)

Symptom: an unawaited task can vanish mid-run. Cause: nothing holds the returned task (a bare expression statement, a lambda that returns it, or a local that only gets callbacks). Change: each site keeps its task in a set until it finishes, using the set the module already has where there is one.

site task how it is held
sprites_session/terminal.py (2) stdin and resize sends module set via _send_in_background
subagent_manager/admission/gate.py spawn_batch_started event the manager's track_store_task, as waits.py does for the same event family
shared_crew/admission_entry.py (2) child cancel and run cancel hooks module set via _hold
remote_session/worker.py SIGTERM retire (a signal-handler lambda) Worker._term_tasks, through a named _on_sigterm handler
dashboard/handlers/crew_log.py (2) flush pass and session send CrewLogPublisher._tasks
knowledge/artifact_ingest.py ingest/remove task ArtifactKnowledgeSync._tasks
subagent_manager/terminal.py subagent_injection_failed event the manager's existing _report_tasks

The dropped-task guard in test/test_dashboard_task_refs.py now covers these seven files and two more drop shapes: a lambda whose body is the task call, and a sync function's local task that is used only for add_done_callback. Run over all of src/, the extended guard reports no remaining site.

Backwards compatibility

Compatible: each task does the same work at the same time; it is only held until it finishes.

Removes nothing: no key, row, path or capability is removed; the diff only adds references to tasks.

Tests

  • test/test_dashboard_task_refs.py::test_no_dropped_task_statement[...]: one case per newly guarded file fails if that file drops a task again.
  • test/test_dashboard_task_refs.py::test_the_guard_sees_a_task_dropped_by_a_lambda and ::test_the_guard_sees_a_local_task_given_only_a_callback: the guard catches the two new drop shapes.
  • test/test_runtime_background_task_retention.py: each test parks the task on an await, runs gc.collect(), and checks the owner still holds the task and lets it go once it finishes: test_the_shared_crew_child_cancel_task_outlives_a_collection, test_the_shared_crew_run_cancel_task_outlives_a_collection, test_a_sprites_terminal_send_outlives_a_collection, test_the_remote_worker_sigterm_retire_outlives_a_collection, test_an_artifact_knowledge_sync_task_outlives_a_collection, test_a_crew_log_flush_outlives_a_collection, test_a_crew_log_session_send_outlives_a_collection.
  • test/test_crew_log_routes.py and test/test_shared_crew_admission_failures.py: updated fakes that pinned the old callback count or returned None from a fake create_task.

Determinism: the new gc tests pass 10 of 10 runs at -n 0. The related suites pass locally (5378 tests).

Proof: PROVEN test/test_dashboard_task_refs.py::test_no_dropped_task_statement[src/kiro_crew/dashboard/handlers/crew_log.py], test/test_dashboard_task_refs.py::test_no_dropped_task_statement[src/kiro_crew/knowledge/artifact_ingest.py], test/test_dashboard_task_refs.py::test_no_dropped_task_statement[src/kiro_crew/remote_session/worker.py], test/test_dashboard_task_refs.py::test_no_dropped_task_statement[src/kiro_crew/shared_crew/admission_entry.py], test/test_dashboard_task_refs.py::test_no_dropped_task_statement[src/kiro_crew/sprites_session/terminal.py] (head 66f1c31)

A manual revert of all seven production files fails 14 tests: the five guard cases above, the guard cases for subagent_manager/admission/gate.py and subagent_manager/terminal.py, and all seven gc tests.

Manual verification

N/A — unit coverage sufficient: each site is a two-line hold whose effect the gc tests observe directly.

Related Issues

Closes #25953

Sibling sweep

Swept: AST walk of every src/kiro_crew module with the extended guard in test/test_dashboard_task_refs.py (bare create_task/ensure_future statements, lambda bodies, callback-only locals) on origin/main c21f5a0
Sibling: src/kiro_crew/sprites_session/terminal.py:138 -- changed here
Sibling: src/kiro_crew/subagent_manager/admission/gate.py:1561 -- changed here
Sibling: src/kiro_crew/shared_crew/admission_entry.py:365 -- changed here
Sibling: src/kiro_crew/remote_session/worker.py:262 -- changed here
Sibling: src/kiro_crew/dashboard/handlers/crew_log.py:1406 -- changed here
Sibling: src/kiro_crew/knowledge/artifact_ingest.py:943 -- changed here
Sibling: src/kiro_crew/subagent_manager/terminal.py:1587 -- changed here
Sibling: src/kiro_crew -- not affected -- every other create_task/ensure_future result is awaited, returned, stored on an attribute, or added to a held set; the guard over all of src reports none after this change

Pattern harvest

Rule candidate: lint
Pattern: a create_task/ensure_future result that is dropped (bare statement, lambda body, or a local used only for add_done_callback); now enforced for these files by the AST guard in test/test_dashboard_task_refs.py.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

The event loop keeps only a weak reference to a task, so a task nothing
else holds can be garbage-collected mid-run and its work silently never
finishes. Ten sites still dropped theirs after the four area PRs:

- sprites_session/terminal.py: the stdin and resize sends
- subagent_manager/admission/gate.py: the spawn_batch_started event
- shared_crew/admission_entry.py: the child and run cancel hooks
- remote_session/worker.py: the SIGTERM retire (a signal-handler lambda)
- dashboard/handlers/crew_log.py: the flush pass and the session send
- knowledge/artifact_ingest.py: the ingest/remove task
- subagent_manager/terminal.py: the subagent_injection_failed event

Each now keeps its task in a set until done with
add_done_callback(set.discard), or joins the manager's existing
_report_tasks / track_store_task set where the module already has one.
A collected crew-log flush also left _flushing set, which stopped the
publisher for good.

The dropped-task guard in test_dashboard_task_refs.py now covers these
files and two more drop shapes: a lambda that returns the task, and a
sync function's local that only gets callbacks.

Closes #25953
@JLargent42
JLargent42 requested review from a team and KiroCrewCaptain as code owners October 30, 2026 22:49
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 66f1c31a049bec1327d8f82c2de3ca73f5ed815c touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 66f1c31a049bec1327d8f82c2de3ca73f5ed815c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: each of the ten sites now keeps its task in a set the owner already has or a module set, and the guard over src/ reports no other site.

Notes

  • Two sites, gate.py (spawn_batch_started) and subagent_manager/terminal.py (subagent_injection_failed), now put event tasks in _report_tasks. cancel_all waits on that set, up to a time limit, before the loop closes. Shutdown can now wait on a slow _fire_event handler, though never past that limit. waits.py already puts other events in the same set, so this is consistent.

[DESIGN-REVIEWED] 66f1c31

Description read

[DESCRIPTION-READ] fc5961754e44c3e399be11c006514fb18f458ec8d72ebecb94040bcca35359a4

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 66f1c31a049bec1327d8f82c2de3ca73f5ed815c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item is a direct hold of a task that was being dropped. The provenance is #25953 plus gc tests that fail on base, and no shared helper exists on base that the PR skips.

Notes

  • Item 7 — undeclared: adding the injection-failed event task to _report_tasks also makes it count as pending work in the manager's drain tally (subagent_parts/manager.py:831, extra_tasks = [*self._report_tasks, ...]). The description says only "held". The cost is small and stays inside the Goal.

What this change ships

Inventory (8 items) — 8 justified

Intent: background work Kiro Crew starts runs to the end instead of being garbage-collected part way. FIX.

  1. The crew-log flush and session send can no longer vanish, so the publisher can't get stuck "flushing" until restart — justified
  2. Artifact knowledge ingest and remove tasks run to completion — justified
  3. A SIGTERM retire on a remote worker always finishes, through a named handler instead of a lambda — justified
  4. Shared-crew child and run cancel hooks finish their cancellation — justified
  5. Sprites terminal keystroke and resize sends are no longer dropped — justified
  6. The spawn_batch_started event is held by the manager's existing track_store_task — justified
  7. The subagent_injection_failed event is held in the existing _report_tasks set — justified
  8. The dropped-task AST guard now covers 7 more files and two more drop shapes (a lambda body, and a local task given only callbacks) — justified

[FIRST-PRINCIPLES-REVIEWED] 66f1c31

Description read

[DESCRIPTION-READ] df0b4c8f4dd45b1f0715adb93cd977c25b2ebf8c2b6ce3871c157a01d01013fe

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 66f1c31a049bec1327d8f82c2de3ca73f5ed815c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 66f1c31

Verdict parsed from the review's SHA-scoped output markers for commit 66f1c31a049bec1327d8f82c2de3ca73f5ed815c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 66f1c31a049bec1327d8f82c2de3ca73f5ed815c false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 66f1c31a049bec1327d8f82c2de3ca73f5ed815c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 66f1c31

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 66f1c31a049bec1327d8f82c2de3ca73f5ed815c false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 31, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 31, 2026 02:03

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (11 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: bug fix; background asyncio tasks in the crew-log publisher, artifact ingest, remote worker, shared-crew cancel hooks, sprites terminal and subagent manager are now held in a set until they finish, so the loop's weak reference cannot drop them mid-run, with retention tests.

@KiroCrewCaptain
KiroCrewCaptain merged commit 8b3c698 into main Oct 31, 2026
150 of 151 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/25953-hold-remaining-tasks branch October 31, 2026 02:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 31, 2026
bolichen97 pushed a commit that referenced this pull request Oct 31, 2026
Backport to release/0.11.0, one commit:

- fix(install): download the pinned kas engine in cli installers (#26384)
  (cherry picked from commit 0a49fdb); docs/system-specs/modules/cli.md
  took only this PR's sentence on the `kas install` row.
- fix(electron): never spawn a gateway once the app is quitting (#26445)
  (cherry picked from commit 6a4d4e0), clean.
- fix: hold the remaining dropped asyncio tasks until they finish (#26463)
  (cherry picked from commit 8b3c698); adapted: the five sites in modules
  0.11.0 has (crew_log, artifact_ingest, remote_session/worker,
  subagent_manager admission gate and terminal). shared_crew/admission_entry.py,
  sprites_session/terminal.py and their tests are not on 0.11.0, so neither is
  the dropped-task guard test that lists them. The crew-log publisher test
  gains the held-task assertions, with `_flush` stubbed as on main.
- fix(kas): refuse a file write a fallbackAction deny spec does not cover
  (#26597) (cherry picked from commit 84c2aaa); doc and comment
  conflicts only: kept 0.11.0's wording around the sub-agent rules (#25361
  is not on 0.11.0) and dropped main's "v2 parity" acp-client block for the
  same reason.

Not picked: #26456 needs main-only `security/argv_rules.py` and the s3
download narrowing of #24562 in `deny_gate.py`. The v0.11.1-rc.1 Release
run's test failures are fixed by #26729.

All three version files move from 0.11.1-rc.1 to 0.11.1-rc.2, and the
in-flight [0.11.1] CHANGELOG section gains the four fixes and the
filelock bump of #26536 and the cron label fix of #26729 (#24718), both
already on the branch. #26149 and #25703 are test-only and have no
reader-visible line.
bolichen97 added a commit that referenced this pull request Oct 31, 2026
Backport to release/0.11.0, one commit:

- fix(install): download the pinned kas engine in cli installers (#26384)
  (cherry picked from commit 0a49fdb); docs/system-specs/modules/cli.md
  took only this PR's sentence on the `kas install` row.
- fix(electron): never spawn a gateway once the app is quitting (#26445)
  (cherry picked from commit 6a4d4e0), clean.
- fix: hold the remaining dropped asyncio tasks until they finish (#26463)
  (cherry picked from commit 8b3c698); adapted: the five sites in modules
  0.11.0 has (crew_log, artifact_ingest, remote_session/worker,
  subagent_manager admission gate and terminal). shared_crew/admission_entry.py,
  sprites_session/terminal.py and their tests are not on 0.11.0, so neither is
  the dropped-task guard test that lists them. The crew-log publisher test
  gains the held-task assertions, with `_flush` stubbed as on main.
- fix(kas): refuse a file write a fallbackAction deny spec does not cover
  (#26597) (cherry picked from commit 84c2aaa); doc and comment
  conflicts only: kept 0.11.0's wording around the sub-agent rules (#25361
  is not on 0.11.0) and dropped main's "v2 parity" acp-client block for the
  same reason.

Not picked: #26456 needs main-only `security/argv_rules.py` and the s3
download narrowing of #24562 in `deny_gate.py`. The v0.11.1-rc.1 Release
run's test failures are fixed by #26729.

All three version files move from 0.11.1-rc.1 to 0.11.1-rc.2, and the
in-flight [0.11.1] CHANGELOG section gains the four fixes and the
filelock bump of #26536 and the cron label fix of #26729 (#24718), both
already on the branch. #26149 and #25703 are test-only and have no
reader-visible line.

Co-authored-by: w <w@k.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dropped asyncio.create_task results can be garbage-collected mid-run

2 participants