Repository navigation
ci(sync-io): gate filesystem syscalls inside async def - #22084
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that every zero-argument What this change shipsInventory (8 items) — 8 justifiedIntent: A new on-loop stat, resolve or open on a mount that can hang should fail CI, so the next one cannot freeze the gateway. This is an ADDITION (it closes #10517).
[FIRST-PRINCIPLES-REVIEWED] caf9de2 Description read[DESCRIPTION-READ] 2a0fe5190ff9b8a7097612b5d70848c2e5db2049598fa898ea05949b2a15f0c6 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] caf9de2 Description read[DESCRIPTION-READ] c5ae5cdcc9f5c26c7641940aa286cbd85546b63c16da45fdfe45a48fda21aeaa That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
54f662b to
1b76202
Compare
|
|
|
|
|
1b76202 to
af72e97
Compare
|
af72e97 to
d077c59
Compare
|
|
d077c59 to
caf9de2
Compare
|
|
Review-ready at caf9de2, waiting on a maintainer review.
No merge, approval or auto-merge from this Crew; the merge is the maintainer's call. |
…22222) search_for_context stat'ed and read the gateway's config.json inside the coroutine. #22084's fs gate flags that call; its baseline row named handlers/knowledge.py, but #22073 had already moved search_for_context into knowledge_agent_context.py, so main's ratchet lane reads one new offender plus one stale row (red since 28ee969). The read now runs in _read_knowledge_config on asyncio.to_thread (the path is the gateway's own, not the request's). The stale knowledge.py row is pruned with --update-baseline, and test_knowledge_onloop_fs.py drops the on-loop allowance for that call and proves the read leaves the loop thread. Refs #22209 Co-authored-by: Bolin Chen <bolichen@amazon.com>
Problem / Motivation
Goal: A new
async defthat stats or resolves a path on the event loop fails CI, so a hung mount can no longer freeze the whole gateway through code nobody flagged.scripts/check_sync_io_in_async.pyblocks new db, subprocess, http and sleep calls inside anasync def, but it skips filesystem calls on purpose. That gap is how the request-path stats inknowledge.py(#21791) andmemory_embeddings.py(#22023) got in: each looked cheap, and nothing failed when they were added.Why it matters
The gateway runs one event loop. A
stat,resolveoropenon an NFS or FUSE mount that stopped answering blocks every endpoint until the kernel gives up. A stall pastdashboard.loop_stall_exit_after_secsmakes the loop watchdog kill the process. Without a gate, each fix is one-off and the next site lands the same way.Not a goal
read_text()oriterdir(): they follow a stat that already ran, so the stat is the one gated.async def. The gate stays lexical, as it is for every other family.What changed (motivation → approach → change)
Filesystem calls on the loop had no gate, so they kept arriving.
The existing gate gains an
fsfamily with the same ratchet, marker and remedy text as the other four:os.stat/lstat/listdir/scandir/walkosaliasos.path.realpath/isfile/isdir/existsos.path.x,import os.path as p,from os import path,from os.path import xopen(...)zipfile.openstays out.resolve()/.stat()/.lstat()/.exists()/.is_file()/.is_dir()store.exists(key)stays outA callable handed to
to_threador_run_path_probe, a nesteddeforlambda, and a# on-loop-io-ok: <reason>line stay clean. Pathlib reads and writes (read_text(),read_bytes(),write_text(),.open()) can hang the same way and are not gated yet; #22086 tracks adding them (20 sites in 8 files today).The
fsfamily keeps its own ledger..github/sync-io-in-async-fs-baseline.txtrecords today's 139fssites in 58 files..github/sync-io-in-async-baseline.txtstays empty, and its terminal test is unchanged. The gate judgesfssites against the new file and every other site against the old one. So nofsentry can ever cover a db, subprocess, http or sleep call.AUTOSDE.yamladds the new file to thea-ratchet-may-only-tightenshrink-only list, and thebackend-lintrow indocs/ci/ci-and-reviews.mdnames thefsfamily and its ledger.Backwards compatibility
Breaking: a new
fscall inside anasync defin a file a change touches now fails the gate. Writer sweep on origin/main @ 1862203:python3 scripts/check_sync_io_in_async.py --reportfinds 139 sites in 58 files, and every one is in the newfsbaseline, so the gate passes on main as it stands. An open PR that adds such a call goes red until it offloads the call or marks it.Removes nothing: the old baseline, its tests and
TestExemplarStaysCleanare unchanged; the diff only adds a family, a ledger and its tests.Tests
test/test_sync_io_in_async_gate.pyand the script's own--testcorpus (44 flagged, 28 clean probes):fssite classifies asfs, not another family.p.exists()is flagged butstore.exists(key)is not;open(p)is flagged butz.open(p)is not._splitkeeps the two ledgers apart, and the committedfsbaseline is byte-identical to a refresh.Determinism: the whole file passed 20/20 repeated runs and one shuffled order (seed 7).
prove.py --base origin/main: PROVEN.Manual verification
python3 scripts/check_sync_io_in_async.py --reporton this head lists 139fscalls in 58 files. The gate passes for both ledgers.Screenshots / video
N/A: CI and backend only, no user-visible UI change.
Related Issues
Closes #10517
All three parts are then on main: part 1 is this PR, part 2 is #21791 (
knowledge.py), part 3 is #14008 (terminal.py?cwd=). #22023 cleared the one memory-handler site first.Pattern harvest
Rule candidate: lint
Pattern: "a filesystem syscall on a caller-chosen path inside an
async def". This PR is that rule.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)