Skip to content

fix(security): stop one slow path resolution from fencing a whole drive - #10276

Merged
bolichen97 merged 1 commit into
mainfrom
fix/sensitive-path-stall-blast-radius
Sep 13, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/sensitive-path-stall-blast-radius

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Split out of #9585, where the Windows shards surfaced it. That PR does not touch
src/kiro_crew/security/, so the defect is pre-existing on main — #9585 only
raised Windows-runner load enough to reach it. It also cannot be fixed in #9585:
that PR's diff is over the Security Scope Review lane's 1 MB cap, so any
security-scope change there fails the lane closed. Hence its own PR, where the
lane can actually review it.

A sensitive-path check that misses its resolve budget charges a cooldown to the
path's prefix, and every path under that prefix is then refused — fail-closed,
without touching the filesystem — until it lapses. Three independent bugs made that
blanket far wider than intended, and a fourth left the total wait unbounded.

1. The stall prefix collapsed to C:\Users

_stall_prefix keeps "the first two components" so a cooldown is scoped to the
wedged mount. But it counted components without splitting the drive: a POSIX
absolute path carries a leading empty component, a Windows path does not, so
C:\Users\bob\AppData\... kept C:\Users — a key containing $HOME, %TEMP%,
the workspace and the checkout. One stall anywhere in the profile fenced
essentially the whole host.

Now splits the drive first (C:\Users\bob), and returns a UNC share root whole
because \\server\share is the mount point. Byte-identical on POSIX, pinned
by a test that drives both ntpath and posixpath.

2. A missed budget was treated as proof of a wedged mount

The probe that separates "wedged" from "merely slow" reads /proc/<tid>/syscall
against a per-architecture syscall table. platform.machine() is AMD64 on every
Windows host (and arm64 on Apple silicon) — neither is mapped, so the probe
returned True unconditionally and the prefix was charged for a resolution that
was only slow.

A miss now earns a bounded grace to finish; one that finishes is a success that
charges nothing. The grace is a capped fraction of the caller's own budget, so a
caller choosing a tight budget keeps a tight worst case. A resolution that misses
its full budget and its full grace still charges and still refuses — the
fail-closed stance is unchanged.

3. Budgets are sized to the work submitted

The anchor rebuild is one pool job doing ~130 realpath calls to build ~200
targets; a candidate resolution does one or two. Sharing one budget left the
rebuild running ~130× closer to its ceiling, which is what made the miss reachable.
It gets its own, larger budget.

4. One thread's resolver waits are capped together

Every bound above is per resolution. One tool call carries many path tokens, and
tokens under different slow prefixes each pay their own budget and grace, so the
per-prefix cooldown never collapses them: six serial waits inside one
on_tool_call add up past the 25 s loop-stall watchdog and the gateway exits —
the crash this whole bound exists to prevent.

Each calling thread shares a 12 s allowance across every prefix and anchor job,
counting the real time spent in each result wait: successes, timeouts and graces
alike. A wait is clamped to what is left, and once the allowance is gone the next
resolution is refused without being submitted at all. The allowance frees 25 s after
that thread's last wait, so two spans cannot combine inside one watchdog gap. A
background thread carries its own allowance and can never spend the event loop's,
which leaves 13 s of the watchdog budget for heartbeat age and the rest of the call.

A wait cut short by that clamp charges no prefix. It ended for a reason that says
nothing about the mount, so it refuses that one call — the same conclusion the
saturated-pool and starved-worker arms already reach.

A resolution that… Charges the prefix?
🟦 finishes inside its budget no — it succeeded
🟦 misses the budget, finishes inside its full grace no — it succeeded
🟨 misses its full budget and its full grace yes
🟩 had its budget or grace clamped by the thread's allowance no
🟩 started with the allowance already spent, never submitted no

🟩 added · 🟨 changed · 🟦 unchanged

A charge now needs the resolution to have been given everything it was entitled to
and still not finish, so no bound in this module can make another one look like a
wedged mount.

Why it showed up as artifact errors

The degradation structures are process-global and per-xdist-worker, so one test
whose resolution was slow failed every later test on that worker whose paths sat
under the same prefix — observed on a #9585 Windows shard as 77 ArtifactError: refusing to use sensitive path as artifact root failures across four unrelated
files, all from a single charged stall (one stall #1 warning in 23,894 outcomes;
the cascade arm logs nothing). The test floor resets that state per test, so the
cascade can neither be inherited nor exported.

_path_resolve_degraded / _path_resolve_wedged are set through the security
facade so the export-identity contract test_security_facade pins still holds;
_path_resolve_load_probes and _path_resolve_thread_waits are not re-exported and
are set on the submodule.

Tested

  • test_security_path_resolve_bounded.py, test_security_facade.py and
    test_artifacts.py green — 327 passed.
  • The full backend suite was run with and against this diff on the same host: the
    failing-test sets are identical, so this change adds no failure. (That host fails
    98 tests either way, on its own layout — /local/home owned by uid 65534, an
    aarch64 machine whose unistd_64.h carries ARM numbers, and no session dbus.)
  • mypy clean; black / comment-history / feature-map / loop-bound-locks /
    sync-io / docs-lint gates clean.
  • Each behavioural change is pinned by a test that fails when it is reverted:
    reverting _stall_prefix fails with the literal old value C:\Users; the grace
    test asserts the same slow resolution does charge the prefix with the grace
    disabled; the allowance tests assert an unsubmitted resolution and a clamped wait
    each leave _path_resolve_degraded empty, and fail against an implementation
    without the cap. So no mechanism here can be removed while the tests pass.

Security posture

The gate never answers "not sensitive" more often than before. Every arm here
raises PathResolutionStalled, so a path whose canonical form cannot be
established is still refused rather than matched on its lexical spelling.

Three of the four changes move refusals toward allowed. The thread allowance is the
one that refuses more: a thread that has already spent 12 s waiting has its
healthy paths refused too, until the allowance frees. That is deliberate. The
alternative on that path is the watchdog killing the gateway, and answering from the
lexical spelling instead would build exactly the lever this module refuses to build
— stall one token under a wedged mount and a workspace symlink into a credential
store passes.

Pattern harvest

Rule candidate: semgrep
Pattern: path-component arithmetic on split(os.sep) without os.path.splitdrive first

Three defects here are instances of a class, not one-offs.

The prefix bug is component-index arithmetic over a raw split(os.sep). Any code
that slices a path into "the first N components" is silently wrong on Windows, because
a drive-qualified path has no leading empty component and C: occupies one of the N.
A semgrep rule can flag \.split\(os\.sep\) (and Path(...).parts) followed by index
or slice arithmetic in a function that has not called splitdrive, which is cheap to
match and has few legitimate exceptions.

The stall-attribution bug is a second, more interesting class that a linter cannot
see: a capability probe whose "cannot answer" is spelled the same as one of its two
real answers.
_worker_blocked_in_filesystem returns True both for "this worker is
blocked in the filesystem" and for "this architecture is not in my table", so every
caller on an unmapped platform silently took the wrong branch — and the docstring
described that as failing toward existing behaviour, which hid it. The reviewable rule
is a review-prompt/AGENTS.md line: a predicate that cannot always answer should return
a three-valued result (or expose a separate supported flag) rather than folding
"unknown" into one of the answers, so callers must handle the unknown case explicitly.
That is the same shape as the agent_named marker discussed on #9779 and the
floorGen === undefined gate on #9646 — "treat absent as compatible" — so it is worth
stating once as a rule rather than fixing a third time.

The unbounded total is a third class, and the one worth a review-prompt line of its
own: a per-operation bound cited as if it were a per-caller bound. The comment
justifying the grace reasoned "8 s + 4 s stays well inside 25 s", which is true of one
resolution and says nothing about a caller that performs six. A bound whose stated
justification names a budget belonging to a different scope than the loop the bound
protects is reviewable by reading the comment alone: if the ceiling is per-turn or
per-request, the arithmetic beneath it has to sum over everything one turn can do, not
over one call.

@bolichen97
bolichen97 requested review from a team and krishdhasmana September 12, 2026 01:57
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 12, 2026
@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 6c4d63ec534c4eca691a42c03cda92ecab0aa941 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Sound, well-pinned fix, but the new 12 s wait allowance is sized to the watchdog's default, not its configurable floor of 10 s.

Watch

The allowance arithmetic hardcodes the watchdog at 25 s ("Allow 12s total … leaving 13s of the 25s watchdog"), but dashboard.loop_stall_exit_after_secs is operator-configurable and LOOP_STALL_EXIT_AFTER_MIN = 10 (config/sections.py:2626). For an operator at the legal low end, one rebuild miss plus its full grace (8 s + 4 s) is a 12 s continuous loop block that exceeds the whole watchdog budget — the gateway hard-exits, the exact crash class this PR exists to close. The pre-existing serial-wait exposure was no better, so this is incomplete rather than a regression, but the PR institutionalizes 12 s as a designed-safe number that is only safe against the default.
Clears when: _PATH_RESOLVE_WAIT_CAP_SECS (and the rebuild+grace ceiling) is clamped against the resolved watchdog value at runtime, or a drift-guard test pins the cap plus headroom under LOOP_STALL_EXIT_AFTER_MIN (the module already uses that literal-plus-drift-guard pattern for _KIRO_AGENTS_DIR), or the config floor is raised above the cap.

Suggestions

  • The _PATH_RESOLVE_WAIT_WINDOW_SECS = 25.0 quiet window encodes the same watchdog assumption; whatever mechanism clears the Watch item should cover both constants so they can't drift independently.

[DESIGN-REVIEWED] 6c4d63e

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 6c4d63ec534c4eca691a42c03cda92ecab0aa941 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 6c4d63e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 6c4d63ec534c4eca691a42c03cda92ecab0aa941: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 6c4d63ec534c4eca691a42c03cda92ecab0aa941 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All lenses run and counts verified. Final review:

First-Principles-Verdict: PASS

Verify on one real slow-disk host that the 12 s allowance refusing healthy paths for 25 s stays invisible during ordinary heavy tool calls.

That arm is the only one that refuses more than base, its trade is argued from the in-repo 25 s watchdog (paths.py:1024, dashboard.loop_stall_exit_after_secs) rather than observed on a production workload, and the 100 ms floor is the single constant standing between "bulk scan" and "allowance exhausted."

What this change ships

Inventory (7 items) — 7 justified

Intent: stop one slow path resolution from refusing paths host-wide (and from killing the gateway) — a FIX.

  1. A Windows stall cooldown now fences C:\Users\<user>, not C:\Users; UNC share root kept whole — justified
  2. A resolution missing its budget gets a bounded grace; finishing in it charges nothing — justified
  3. The anchor rebuild gets its own 8 s budget (new budget param; 1 non-default caller of 3 call sites counted) — justified
  4. Each thread's resolver waits capped at 12 s; exhaustion refuses without probing or charging — justified
  5. Waits under 100 ms don't count toward the allowance — justified
  6. Tests start and finish with an unstalled resolver (conftest reset, extends the existing _isolate_kirocrew_home floor) — justified
  7. security.md spec updated in the same commit — justified

No deleted pins (test hunks are pure additions); no duplicate mechanism (existing bounds are per-prefix, none per-thread); no unfixed siblings of the drive-arithmetic class (5 other .split(os.sep) sites in src/, all over drive-free relative paths).

[FIRST-PRINCIPLES-REVIEWED] 6c4d63e

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 6c4d63ec534c4eca691a42c03cda92ecab0aa941 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6c4d63e

Verdict parsed from the review's SHA-scoped output markers for commit 6c4d63ec534c4eca691a42c03cda92ecab0aa941.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 6c4d63ec534c4eca691a42c03cda92ecab0aa941: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- 🟡 CONCERNS

Which legitimate operations does 6c4d63ec534c4eca691a42c03cda92ecab0aa941 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: the reviewer's own judgement; no confirmed regression.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 7 classified, 0 newly refused, 7 still allowed.
  • ubuntu-latest (platform posix): 7 classified, 0 newly refused, 7 still allowed.
  • windows-latest (platform windows): 6 classified, 0 newly refused, 6 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: CONCERNS

TIGHTENING: A per-thread cumulative wait allowance inside the bounded sensitive-path resolver (_run_resolution_bounded in src/kiro_crew/security/paths.py) — refuses further resolutions once one calling thread has accumulated 12 seconds of resolver waits within a 25-second quiet window, refusing "the reviewer's slow-but-completing scenario: waits that each finish just under budget… 13 at ~2s apiece is ~25s of loop block" from monopolizing the event-loop thread past the loop-stall watchdog: "_PATH_RESOLVE_WAIT_CAP_SECS = 12.0 … exhaustion refuses without submitting or charging a prefix", raising PathResolutionStalled, which every gate turns into a fail-closed "path is sensitive" answer. All other hunks in the diff narrow refusals rather than widen them: the grace period stops a merely-slow resolution from charging a prefix, the rebuild gets its own larger budget, the Windows stall key narrows from C:\Users to C:\Users\<user>, and a clamped timeout no longer charges the prefix.

ADJUDICATED: 13 candidates, 0 confirmed newly-refused
No confirmed regressions were handed back for any candidate. The rows probe the boundaries the diff touches at the string level: paths under the two-component POSIX stall key (/home/runner), the pytest tmp artifact root the original cascade refused, absolute drive-letter paths under C:\Users\<user>, the %TEMP% and $HOME environment spellings, the UNC share-root branch the new drive-detection code introduces, both platforms' spellings of the touched module's own test invocation, and the three bulk workloads the diff's own wait-floor comment names (project-tree listing, directory-wide scan, absolute-path status read).

PLATFORM COVERAGE: Candidates were written in posix (7 rows) and windows (6 rows) spellings; none in any, since every row carries a path or executable name. The windows rows cover each platform assumption the diff adds: the drive-splitting logic (type C:\Users\..., Get-Content C:\Users\...), the C:\Users\<user>/%TEMP% prefixes the old key collapsed (dir C:\Users\...\Temp, dir %TEMP%), the new UNC share-root return branch (dir \\server\share\project), and the .exe interpreter spelling. The posix rows pin that the drive split is a no-op where there are no drives (/home/<user>, /tmp/pytest-of-runner, $HOME), matching the diff's own posixpath invariance test. One assumed platform I could not write a shell candidate for: Apple silicon macOS — the diff names it (alongside Windows) as a host where the syscall probe cannot discriminate, and its distinctive mount spelling (/Volumes/<share>) is a host-mount fact no classifier row on a CI runner exercises; the posix rows cover its path spelling but not its mount behavior. The per-leg verdicts are the deterministic fold's to report after this review is captured, not mine.

UNADJUDICATED: The tightening itself — the cumulative wait allowance — is a temporal, state-dependent guard inside a path resolver, and the classifier adjudicates shell command strings only. Call path an input takes to reach it: any tool-call command carrying a path → the composite's path fence → is_sensitive_path() / path_contains_sensitive() (src/kiro_crew/security/paths.py:2177, :2197) → _candidate_forms / _home_dir_targets → _run_resolution_bounded, where an exhausted allowance raises PathResolutionStalled before submitting work, and the gate returns True fail-closed (paths.py:2103). No command string can flip on this guard under classification: on a fresh thread the allowance is full, and only real filesystem waits of ≥100ms accumulate toward it, which a healthy classifier host never produces — so the zero confirmed regressions above says nothing about this guard, and I do not claim it as a pass. The candidate legitimate inputs I would have submitted, had a verdict been available: a knowledge-indexing pass or project-tree listing over a network-mounted vault where each resolution completes in 0.1–2s (six such waits spend 11.4s of the 12s cap; the seventh healthy, unrelated path on the same thread is refused — the diff's own test_a_run_of_slow_but_completing_waits_still_exhausts_the_allowance pins that refusal as intended); and an artifact-root check issued on the loop thread immediately after a cold anchor rebuild plus its grace (8s + 4s consumes the full 12s allowance, refusing the next ≥100ms resolution on that thread for up to 25s). The blast radius is deliberately bounded — one call at a time, no prefix charged, expiring 25s after the last wait — but it is a new refusal of otherwise healthy paths that only a wall-clock harness on a slow mount could confirm or clear, and none exists in this lane.

[SCOPE-REVIEWED] 6c4d63e

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 12, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 12, 2026 18:00
@bolichen97
bolichen97 force-pushed the fix/sensitive-path-stall-blast-radius branch from ede524c to 36f8978 Compare September 12, 2026 18:29
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 12, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

The cumulative path-resolution deadline fix is prepared locally. Its validation exposed an independent app dependency-lock creation failure that also reproduced on clean main. That prerequisite is now isolated in #10464, with serial local regression tests passing and the full platform checks running in CI. Keeping the two changes separate; the deadline revision will follow once the prerequisite lands.

A sensitive-path check that misses its resolve budget charges a COOLDOWN to the
path's prefix, and every path under that prefix is then refused -- fail-closed,
without touching the filesystem -- until it lapses. Two things made that blanket
far wider than intended on Windows, where one transient miss fenced essentially
the whole host for 30s (doubling to 30min).

`_stall_prefix` counted path components without splitting the drive. A POSIX
absolute path carries a leading empty component, a Windows path does not, so
"the first two components" of `C:\Users\bob\...` kept `C:\Users` -- a key that
contains $HOME, %TEMP%, the workspace and the checkout. It now splits the drive
first, keying on `C:\Users\bob`, and returns a UNC share root whole because
`\\server\share` IS the mount point. Byte-identical on POSIX.

A missed budget was also treated as proof of a wedged mount. The syscall probe
that separates "wedged" from "merely slow" reads `/proc/<tid>/syscall` against a
per-architecture table, and `platform.machine()` is `AMD64` on every Windows host
and `arm64` on Apple silicon -- neither is mapped, so the probe returned True
unconditionally and the prefix was charged for a resolution that was only slow.
A miss now earns a bounded GRACE to finish, and one that finishes is a success
that charges nothing. The grace is a capped fraction of the caller's own budget,
so a caller that chooses a tight budget keeps a tight worst case. A miss of both
the full budget and the full grace still charges and still refuses, so the
fail-closed stance is unchanged.

Each budget is now sized to the work its caller submits: the anchor REBUILD is
one pool job doing ~130 realpath calls to build ~200 targets, where a candidate
resolution does one or two, so sharing one budget left the rebuild running ~130x
closer to its ceiling. It carries its own, larger budget.

Every bound above is per resolution, and one tool call carries many path tokens.
Tokens under DIFFERENT slow prefixes each pay their own budget and grace, which
the per-prefix cooldown cannot collapse, so serial waits inside one
`on_tool_call` summed past the 25s loop-stall watchdog and exited the gateway --
the crash the whole bound exists to prevent. Each calling thread now shares a 12s
allowance across every prefix and anchor job, counting the real time spent in
each result wait that is long enough to be filesystem latency rather than pool
round-trip overhead. A wait is clamped to what is left of the allowance, and once
it is gone the next resolution is refused without being submitted. The allowance
frees 25s after that thread's last counted wait, so two spans cannot combine
inside one watchdog gap, and a background thread carries its own allowance and
never spends the event loop's.

A wait clamped short by that allowance charges no prefix: it ended for a reason
that says nothing about the mount, so it refuses that one call, the conclusion
the saturated-pool and starved-worker arms already reach. Charging requires a
resolution to have received its full entitled budget and full entitled grace and
still not finish, so no bound in this module can make another one look like a
wedged mount.

The test floor resets the four process-global structures per test. They are
per-worker state that makes `is_sensitive_path()` answer True for a whole prefix,
so one test whose resolution was slow could fail every later test on the same
xdist worker -- observed as 77 `ArtifactError: refusing to use sensitive path as
artifact root` errors across four unrelated files on one Windows shard, from a
single charged stall. `_path_resolve_degraded` and `_path_resolve_wedged` are set
through the `security` facade so the export-identity contract holds;
`_path_resolve_load_probes` and `_path_resolve_thread_waits` are not re-exported
and are set on the submodule.
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 13, 2026
@bolichen97
bolichen97 force-pushed the fix/sensitive-path-stall-blast-radius branch from 36f8978 to 6c4d63e Compare September 13, 2026 06:33
@bolichen97

bolichen97 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: per-thread cumulative resolver-wait allowance (12s, 100ms floor, 25s quiet reset) plus a full-entitlement gate on the stall charge

  • Serial grace waits can outlive the loop watchdog (span=263f056987ad) — fixed in 6c4d63ec534c4eca691a42c03cda92ecab0aa941.

Accepted. The per-resolution bound was reasoned per resolution, and the per-prefix
cooldown cannot collapse waits under distinct prefixes, so their sum was unbounded.
Each calling thread now shares a 12s allowance across every prefix and anchor job,
counting each result wait at or above a 100ms floor; a wait is clamped to what is
left and an exhausted allowance refuses without submitting. It frees 25s after that
thread's last counted wait, so two spans cannot combine in one 25s watchdog gap.
The floor is measured, not guessed: without it 3000 healthy is_sensitive_path
calls accrued 1.151s of pool round-trip (0.384ms each), exhausting 12s after ~31k
calls, so a bulk tree scan would have fenced the host; with it the same 3000 accrue
0.000s. A wait clamped by the allowance charges no prefix — a charge now requires
the full entitled budget AND the full entitled grace to have been granted and missed.

Tests that fail without the cap: an unsubmitted resolution and a clamped wait each
leave _path_resolve_degraded empty; a long run of sub-floor successes never exhausts
the allowance; a run of 1.9s slow-but-completing waits does; and the production cap is
pinned below the 25s watchdog. A background thread carries its own allowance and can
never spend the event loop's.

@bolichen97

bolichen97 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: per-thread cumulative resolver-wait allowance, replacing the per-prefix load arm as the per-call event-loop bound on the grace path

  • the grace wait neither charges the prefix nor counts against _load_arm_budget_spent (span=cf138e992ce0) — fixed in 6c4d63ec534c4eca691a42c03cda92ecab0aa941.

Accepted, and fixed with a bound stronger than the one suggested. Charging the grace
against _load_arm_budget_spent(prefix) would still be per prefix, so tokens under
distinct slow prefixes inside one call would each get a fresh allowance and the
per-call event-loop stall the load arm exists to cap would stay reachable.
A 12s allowance per CALLING THREAD, spanning every prefix and anchor job, restores
that cap on the grace path and on every other wait — including the Windows and
Apple-silicon hosts named here, where _worker_blocked_in_filesystem is always True
and the arm is never consulted. Only waits at or above a 100ms floor count, so
ordinary pool round-trip does not, and the allowance frees 25s after the last counted
wait. The load arm keeps its own per-prefix job unchanged.

The grace still charges nothing when it completes, which was the intended trade you
dropped as CANDIDATE 2; what changed is that the trade is now bounded per call rather
than only per prefix.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants