Skip to content

fix(dashboard): list the outbox off the event loop - #23967

Merged
KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
IngridMorstrad:fix/outbox-list-off-loop
Oct 25, 2026
Merged

KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
IngridMorstrad:fix/outbox-list-off-loop

Conversation

@IngridMorstrad

@IngridMorstrad IngridMorstrad commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Listing the outbox (GET /api/outbox) no longer blocks the gateway's event loop.

api_outbox_list ran is_dir(), iterdir(), and a stat() plus is_file() for every entry, all inside the coroutine. The outbox grows with every file an agent sends. On a data home whose mount has stopped answering, each of those calls blocks in the kernel, so all of the gateway's work stops while the listing waits. check_sync_io_in_async already listed the 3 calls in its baseline.

Why it matters

The gateway runs one event loop. A stalled scan freezes every session, channel, cron and MCP call at once. If the stall lasts longer than dashboard.loop_stall_exit_after_secs, the watchdog restarts the gateway and every in-flight turn is lost (#1572, #3057).

Not a goal

What changed (motivation → approach → change)

Symptom: a slow or wedged outbox mount stalls every session while the dashboard lists the outbox. Root cause: the directory scan ran on the event-loop thread.

The scan moves into a sync helper, _list_outbox_entries in file_api/outbox.py, and api_outbox_list awaits it with asyncio.to_thread. The outbox is the gateway's own directory, never a path the request picks. So the default executor is the right lane, not the bounded path-probe pool that request-chosen paths use (the same call as #22222). The helper keeps the same filters: it skips directories, names that redact() would rewrite, and entries deleted mid-scan, and it sorts newest first.

The facade dashboard/handlers/files.py re-exports the new helper, the way it re-exports every owner name. The file_api/outbox.py row is pruned from .github/sync-io-in-async-fs-baseline.txt with --update-baseline (143 to 140 calls).

Backwards compatibility

Compatible: the route, response shape, filters, ordering and the 50-entry cap are unchanged; only the thread the scan runs on moves.

Removes nothing: the baseline row is dropped because the calls it allowed no longer run on the loop. No reader, route or key goes away.

Tests

  • test/test_dashboard_files_coverage.py::TestOutboxList::test_scan_runs_off_the_event_loop_thread: makes a real request with a file, a subdirectory and a credential-shaped name in the outbox. It asserts that only the file is listed (same size, same filters) and that the outbox's iterdir and stat calls ran on a thread other than the event loop's. The four existing TestOutboxList tests (absent outbox, newest-first and skipping dirs, credential-shaped name hidden, entry removed mid-scan) still pass unchanged.
  • Determinism: 20/20 repeated runs pass, and the TestOutboxList class also passes in reversed order. The test only observes threading.get_ident() and uses no timing.

Proof: PROVEN test/test_dashboard_files_coverage.py::TestOutboxList::test_scan_runs_off_the_event_loop_thread (head 45a0a59)

Gates run locally: check_sync_io_in_async.py (140 calls / 59 files), check_black_formatting.py, check_subprocess_encoding.py, comment-history, brand-name, isort, flake8, mypy, docs-lint.sh, test_flake_pattern_ratchet.py. test_dashboard_files_composition_contract.py and test_dashboard_files_onloop_fs.py pass (239 passed together with the coverage file). local-gate.py's related set fails 70 tests in 5 unrelated files (test_ledger_sync_git.py, test_design_tweak_backend.py, test_acp_spawn_offload.py, test_apps_registry_identity_admission.py, test_cron_run_scratch.py). The same 70 node ids fail on a clean origin/main worktree on this host, so they come from the environment, not this diff. semgrep is not installed here, so the SAST lane covers it.

Manual verification

N/A: the unit test drives the real aiohttp route through TestClient and checks the thread the syscalls run on, which is the property this change exists for.

Related Issues

Closes #23942

Sibling sweep

Swept: git grep -nE "outbox_dir\(\)" -- src/ and the file_api/ rows of .github/sync-io-in-async-fs-baseline.txt
Sibling: src/kiro_crew/dashboard/file_api/channel_upload.py:100 -- not affected -- _gate_upload_file is sync and its callers already run it off the loop
Sibling: src/kiro_crew/mcp_tools/messaging.py:1060 -- not affected -- file_send is a sync MCP tool, not a coroutine on the gateway loop
Sibling: src/kiro_crew/dashboard/file_api/workspaces.py -- not affected -- a different handler family still on the baseline, out of this PR's goal

Pattern harvest

Not generalizable: the class (a blocking fs call inside async def) is already enforced repo-wide by scripts/check_sync_io_in_async.py; this PR only pays down one baselined row of it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 25, 2026
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 45a0a59672483ca87734234036f3d2aa73419dbd touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 45a0a59672483ca87734234036f3d2aa73419dbd via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (4 items) — 4 justified

Intent: keep the gateway responsive while the dashboard lists the outbox. FIX.

  1. The outbox listing's directory scan now runs off the event loop (asyncio.to_thread), and what it returns is unchanged — justified
  2. The file_api/outbox.py row (3 calls) is dropped from the sync-io baseline — justified
  3. The facade handlers/files.py re-exports _list_outbox_entries — justified
  4. A new test pins that the scan runs off the loop thread — justified

[FIRST-PRINCIPLES-REVIEWED] 45a0a59

Description read

[DESCRIPTION-READ] 1bd585f90a2991b46532fc3a6c0e85e2847b6be7eabf662377d32df2f74f780f

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 45a0a59672483ca87734234036f3d2aa73419dbd via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design concerns. This is a proportionate fix aimed at the cause: the scan of the gateway-owned outbox moves to the default executor, and the route's output and filters stay the same.

Notes

  • A wedged mount now stalls an executor worker instead of the event loop. Repeated listings against a dead mount could fill up the default thread pool. That cost can be undone and is out of this PR's goal, so it's worth a follow-up issue rather than a change here.

[DESIGN-REVIEWED] 45a0a59

Description read

[DESCRIPTION-READ] 3c42199ff73bd4f4c80d2efdc664f688e06efefce63500ab8ee7a73e5a3a3157

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 45a0a59672483ca87734234036f3d2aa73419dbd via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 45a0a59

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 45a0a59672483ca87734234036f3d2aa73419dbd via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 45a0a59

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 25, 2026
@IngridMorstrad

Copy link
Copy Markdown
Contributor Author

CI on 2acd6ceec has two red backend shards. Neither comes from this diff, which touches only the outbox listing handler, its test, a facade re-export and one baseline row:

Coverage Gate and PR Readiness fail closed only because backend-test=failure. Once #23933 lands I'll rebase, which re-runs CI. Every AI review lane reports no findings.

GET /api/outbox ran one iterdir() plus a stat() per entry inside the
coroutine. The outbox grows with every file an agent sends, and on a
data home whose mount stops answering each of those syscalls blocks in
the kernel, stalling every session on the gateway's single loop until
the watchdog restarts it. check_sync_io_in_async listed the three calls
in its baseline.

The scan now lives in _list_outbox_entries and runs on
asyncio.to_thread; the directory is the gateway's own, so the default
executor is the lane, not the bounded probe pool request-chosen paths
use. Filtering, ordering and the 50-entry cap are unchanged. The
baseline row for file_api/outbox.py is pruned with --update-baseline.
@IngridMorstrad
IngridMorstrad force-pushed the fix/outbox-list-off-loop branch from 2acd6ce to 45a0a59 Compare October 25, 2026 09:40
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 25, 2026
@IngridMorstrad

Copy link
Copy Markdown
Contributor Author

Rebased onto main after #23933 merged. The new head is 45a0a5967, and the governance test now passes. Two backend shards are still red, and neither is caused by this diff:

Coverage Gate and PR Readiness fail closed only because of those shards. A maintainer re-run of the failed jobs should clear them; this fork account can't trigger a re-run.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 25, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 25, 2026 13:04

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: GET /api/outbox moves its directory scan into asyncio.to_thread with unchanged filtering, plus a test that the scan leaves the event loop thread.

@KiroCrewCaptain
KiroCrewCaptain merged commit 2146912 into kirodotdev:main Oct 25, 2026
187 of 200 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(dashboard): GET /api/outbox scans the outbox directory on the event loop

2 participants