Skip to content

ci(sync-io): gate filesystem syscalls inside async def - #22084

Merged
chenmingwei23 merged 1 commit into
mainfrom
feat/10517-fs-family-sync-io-gate
Oct 19, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
feat/10517-fs-family-sync-io-gate

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A new async def that stats or resolves a path on the event loop fails CI, so a hung mount can no longer freeze the whole gateway through code nobody flagged.

scripts/check_sync_io_in_async.py blocks new db, subprocess, http and sleep calls inside an async def, but it skips filesystem calls on purpose. That gap is how the request-path stats in knowledge.py (#21791) and memory_embeddings.py (#22023) got in: each looked cheap, and nothing failed when they were added.

Why it matters

The gateway runs one event loop. A stat, resolve or open on an NFS or FUSE mount that stopped answering blocks every endpoint until the kernel gives up. A stall past dashboard.loop_stall_exit_after_secs makes the loop watchdog kill the process. Without a gate, each fix is one-off and the next site lands the same way.

Not a goal

  • Fixing the 139 existing sites. They are recorded once in their own baseline, which can only shrink.
  • Reads such as read_text() or iterdir(): they follow a stat that already ran, so the stat is the one gated.
  • Calls one frame down from an async def. The gate stays lexical, as it is for every other family.

What changed (motivation → approach → change)

Filesystem calls on the loop had no gate, so they kept arriving.

The existing gate gains an fs family with the same ratchet, marker and remedy text as the other four:

Matched How
os.stat / lstat / listdir / scandir / walk through an os alias
os.path.realpath / isfile / isdir / exists os.path.x, import os.path as p, from os import path, from os.path import x
builtin open(...) bare name only, so zipfile.open stays out
.resolve() / .stat() / .lstat() / .exists() / .is_file() / .is_dir() any receiver, no positional argument, so store.exists(key) stays out

A callable handed to to_thread or _run_path_probe, a nested def or lambda, and a # on-loop-io-ok: <reason> line stay clean. Pathlib reads and writes (read_text(), read_bytes(), write_text(), .open()) can hang the same way and are not gated yet; #22086 tracks adding them (20 sites in 8 files today).

The fs family keeps its own ledger. .github/sync-io-in-async-fs-baseline.txt records today's 139 fs sites in 58 files. .github/sync-io-in-async-baseline.txt stays empty, and its terminal test is unchanged. The gate judges fs sites against the new file and every other site against the old one. So no fs entry can ever cover a db, subprocess, http or sleep call. AUTOSDE.yaml adds the new file to the a-ratchet-may-only-tighten shrink-only list, and the backend-lint row in docs/ci/ci-and-reviews.md names the fs family and its ledger.

Backwards compatibility

Breaking: a new fs call inside an async def in a file a change touches now fails the gate. Writer sweep on origin/main @ 1862203: python3 scripts/check_sync_io_in_async.py --report finds 139 sites in 58 files, and every one is in the new fs baseline, so the gate passes on main as it stands. An open PR that adds such a call goes red until it offloads the call or marks it.

Removes nothing: the old baseline, its tests and TestExemplarStaysClean are unchanged; the diff only adds a family, a ledger and its tests.

Tests

test/test_sync_io_in_async_gate.py and the script's own --test corpus (44 flagged, 28 clean probes):

  • Each matched spelling above is flagged, and each fs site classifies as fs, not another family.
  • p.exists() is flagged but store.exists(key) is not; open(p) is flagged but z.open(p) is not.
  • Offloaded, nested and marked calls stay clean.
  • _split keeps the two ledgers apart, and the committed fs baseline is byte-identical to a refresh.

Determinism: the whole file passed 20/20 repeated runs and one shuffled order (seed 7). prove.py --base origin/main: PROVEN.

Manual verification

python3 scripts/check_sync_io_in_async.py --report on this head lists 139 fs calls in 58 files. The gate passes for both ledgers.

Screenshots / video

N/A: CI and backend only, no user-visible UI change.

Related Issues

Closes #10517

All three parts are then on main: part 1 is this PR, part 2 is #21791 (knowledge.py), part 3 is #14008 (terminal.py ?cwd=). #22023 cleared the one memory-handler site first.

Pattern harvest

Rule candidate: lint
Pattern: "a filesystem syscall on a caller-chosen path inside an async def". This PR is that rule.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision caf9de2d4654ec9323105c8c9c84a5a5c180daec touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of caf9de2d4654ec9323105c8c9c84a5a5c180daec — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that every zero-argument .exists()/.resolve()/.stat() in the 139-site fs baseline really is a path. Arity is the gate's only receiver filter.

What this change ships

Inventory (8 items) — 8 justified

Intent: A new on-loop stat, resolve or open on a mount that can hang should fail CI, so the next one cannot freeze the gateway. This is an ADDITION (it closes #10517).

  1. A new async def that calls os.stat, lstat, listdir, scandir or walk now fails the backend-lint gate — justified
  2. os.path.realpath, isfile, isdir and exists are flagged through every import spelling — justified
  3. A bare builtin open(...) inside an async def is flagged — justified
  4. Zero-argument path methods (.resolve(), .stat(), .exists() and the rest) are flagged on any receiver — justified
  5. A separate fs ledger, .github/sync-io-in-async-fs-baseline.txt, records the 139 existing sites in 58 files and can only shrink — justified
  6. The gate judges each ledger separately and prints which ledger each result belongs to — justified
  7. AUTOSDE.yaml adds the new ledger to the shrink-only list — justified
  8. The backend-lint row in docs/ci/ci-and-reviews.md names the fs family — justified

[FIRST-PRINCIPLES-REVIEWED] caf9de2

Description read

[DESCRIPTION-READ] 2a0fe5190ff9b8a7097612b5d70848c2e5db2049598fa898ea05949b2a15f0c6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of caf9de2d4654ec9323105c8c9c84a5a5c180daec — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] caf9de2

Description read

[DESCRIPTION-READ] c5ae5cdcc9f5c26c7641940aa286cbd85546b63c16da45fdfe45a48fda21aeaa

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of caf9de2d4654ec9323105c8c9c84a5a5c180daec and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] caf9de2

False positive or not applicable? A repository writer can comment:
/ai-review override gpt caf9de2d4654ec9323105c8c9c84a5a5c180daec: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed caf9de2d4654ec9323105c8c9c84a5a5c180daec — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] caf9de2

Verdict parsed from the review's SHA-scoped output markers for commit caf9de2d4654ec9323105c8c9c84a5a5c180daec.

False positive or not applicable? A repository writer can comment:
/ai-review override fable caf9de2d4654ec9323105c8c9c84a5a5c180daec: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@iamwhatever
iamwhatever force-pushed the feat/10517-fs-family-sync-io-gate branch from 54f662b to 1b76202 Compare October 19, 2026 05:44
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=008ea0d3b04a run_gate's unused fs_baseline_path parameter: fixed in 1b76202

The parameter is gone; run_gate takes (baseline_path, update) as on main and judges fs sites against FS_BASELINE directly.
No caller passed it, so behaviour is unchanged; test/test_sync_io_in_async_gate.py passes (115).

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=dbbe599f5e72 --report now lists every file, undeclared: fixed in the description

The change stays: TestReportMode asserts every baselined path appears in --report, and the fs ledger has 58 files, more than the old top-25 cut.
What changed now says --report lists every file with a site and why.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

The premise was wrong: read_text / read_bytes / write_text / .open() need no prior stat and can hang the same way. The script docstring now says this is a gap, not a guarantee.
Adding them is outside this PR's frozen Not-a-goal, so #22086 (deferred-finding, owner @iamwhatever, Due 2026-11-02) carries it; a dry run finds 20 sites in 8 files.
What changed names the gap and #22086.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=a2797e890d5d tie the read exclusion to the real trade-off: fixed in 1b76202

The docstring no longer rests the exclusion on a stat-ordering premise; it says reads can hang too and are follow-up work, and the description gives the count (20 sites in 8 files) and #22086.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • --baseline redirects only the old ledger; no caller expects both isolated: rebutted (verified, no change)

run_gate's one caller is main() (scripts/check_sync_io_in_async.py), and CI (ci.yml, main-ratchet-audit.yml) and the prepare-pr profile call the gate with no --baseline.
No test calls main() or run_gate with a custom path; the ledger tests drive _read_baseline / _write_baseline / _split on tmp paths. So no caller relies on --baseline isolating the fs ledger.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • 'every zero-argument receiver in this tree is a path': rebutted (verified on this head)

A scan of every zero-argument .resolve/.stat/.lstat/.exists/.is_file/.is_dir call inside an async def in src/ finds 82 distinct receivers; all are path expressions (Path(...), x / 'name', *_dir, *_path, data_home()), and the three not named as paths (p, backup, script) are pathlib objects from glob/iterdir or Path(...).
Each of those 139 sites is already in the fs ledger, so none needs a marker today. A future non-path .exists() takes # on-loop-io-ok, as the docstring's Known limits section says.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@iamwhatever
iamwhatever force-pushed the feat/10517-fs-family-sync-io-gate branch from 1b76202 to af72e97 Compare October 19, 2026 05:59
@iamwhatever

iamwhatever commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=16dc68ebc817 backend-lint row in docs/ci/ci-and-reviews.md omits the fs family: fixed in af72e97

The row now says the gate also refuses a blocking fs call (os.stat / listdir / walk, os.path.realpath / exists, bare open(), zero-argument .resolve() / .stat() / .exists() and kin) outside .github/sync-io-in-async-fs-baseline.txt, and that each ledger can only shrink.
docs-lint and the 7 test files that read ci-and-reviews.md pass (317 tests).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 19, 2026
@iamwhatever
iamwhatever force-pushed the feat/10517-fs-family-sync-io-gate branch from af72e97 to d077c59 Compare October 19, 2026 06:10
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=48f639da0112 --report change claimed to be checked by TestReportMode, which reads only the empty BASELINE: fixed in d077c59 (removed)

TestReportMode iterates _read_baseline(BASELINE), which is empty, so nothing needed --report to list more than 25 files. The ranked[:25] cut is restored to main's line and the description no longer makes the claim.
self-added: yes
mechanism: the --report widening came from an earlier round of this PR; removing it keeps the gate's goal and fixes nothing else.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • request-path fs sites like knowledge.py should move off the loop, not collect markers: rebutted (outside this PR, and the ledger already pushes that way)

The fs ledger is shrink-only: a fixed site must be pruned with --update-baseline, and a new marker needs a non-empty reason that review reads (MARKER_RE refuses a bare one).
knowledge.py's one listed fs site and the other 138 are the backlog this gate exposes; offloading them is per-module work like #21791 and #22023, not part of adding the gate.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@iamwhatever
iamwhatever force-pushed the feat/10517-fs-family-sync-io-gate branch from d077c59 to caf9de2 Compare October 19, 2026 06:15
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • docstring names security/paths.py for _resolved_forms_bounded, which lives in security/paths_resolver.py: fixed in caf9de2

main split security/paths.py into owners (#22014); def _resolved_forms_bounded is at src/kiro_crew/security/paths_resolver.py:868, and the docstring now names that file.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

A scan of every zero-argument .resolve/.stat/.lstat/.exists/.is_file/.is_dir call inside an async def in src/ finds 82 distinct receivers; all are pathlib expressions, including the three not named as paths (p = Path(spec_path); backup from _resolve_backup_name -> Path | None; script from console_script_path -> Path).
Arity is the only receiver filter by design, documented under Known limits; a future non-path match takes # on-loop-io-ok.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review-ready at caf9de2, waiting on a maintainer review.

No merge, approval or auto-merge from this Crew; the merge is the maintainer's call.

@chenmingwei23
chenmingwei23 merged commit 28ee969 into main Oct 19, 2026
103 of 104 checks passed
@chenmingwei23
chenmingwei23 deleted the feat/10517-fs-family-sync-io-gate branch October 19, 2026 14:09
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 19, 2026
bolichen97 added a commit that referenced this pull request Oct 19, 2026
…22222)

search_for_context stat'ed and read the gateway's config.json inside the
coroutine. #22084's fs gate flags that call; its baseline row named
handlers/knowledge.py, but #22073 had already moved search_for_context into
knowledge_agent_context.py, so main's ratchet lane reads one new offender
plus one stale row (red since 28ee969).

The read now runs in _read_knowledge_config on asyncio.to_thread (the path
is the gateway's own, not the request's). The stale knowledge.py row is
pruned with --update-baseline, and test_knowledge_onloop_fs.py drops the
on-loop allowance for that call and proves the read leaves the loop thread.

Refs #22209

Co-authored-by: Bolin Chen <bolichen@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extend the sync-IO-in-async gate to request-path filesystem calls, and offload knowledge.py's five sites

2 participants