Skip to content

feat(subprocess-pool): an Executor that runs work in a separate interpreter - #12709

Merged
iamwhatever merged 1 commit into
mainfrom
fix/resolver-offload-subprocess
Sep 22, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/resolver-offload-subprocess

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Base commit: 64fbda5f9170ad4614cb683d3e44a8f735615d41

Opened as a DRAFT. The base it lands is complete and tested; what is not yet done
is wiring the path resolver to it, which needs a decision rather than more code
(see section 5). Iterating here rather than in the worktree so the review lanes
run in parallel.

1. What is the problem?

The sensitive-path gate refuses legitimate paths because its symlink-resolution
budget is spent waiting for the GIL rather than for the disk.

os.path.realpath is pure Python. posixpath._joinrealpath walks a path one
component at a time, doing an lstat/readlink pair per component, and each of
those releases and reacquires the GIL. A reacquisition waits up to one
sys.getswitchinterval(), measured 5.0 ms on this host. The cost is therefore
"components x 2 x contention", which has nothing to do with how fast the
filesystem answers.

Median ms to resolve one six-component path, with N pure-Python CPU-bound sibling
threads running alongside:

sibling threads in-thread (today)
0 0.067
8 179.538
24 760.313
48 3480.685

The default budget is 2000 ms (_PATH_RESOLVE_TIMEOUT_SECS) and the anchor
rebuild is 8000 ms, so the in-thread path crosses its budget somewhere between 24
and 48 contenders. This gateway was measured carrying 157 threads.

The filesystem is not implicated. There is one mount, local xfs, and every
underlying realpath of a refused path answered in 14-44 microseconds.

The existing knob cannot reach this. KIROCREW_PATH_RESOLVE_WORKERS (default 2,
max 64) sizes a thread pool, so each added worker is one more GIL contender.
Moving the resolution into the kernel is also not enough on its own: a single
O_PATH resolution measured 60.8 ms at 24 threads but 2387 ms at 48, still over
budget.

Refs #10255. This PR does not close it: it lands the mechanism the fix needs, and
the resolver is not wired to it yet (see section 5).

2. Why this issue matters to the user

The user sees their own tool calls refused, with a message saying the path could
not be verified within the resolver budget. Failing closed is correct once the
budget expires, but the budget is expiring for the wrong reason: interpreter load,
not an unhealthy mount.

Three consequences follow. Legitimate work is blocked, and blocked more often the
busier the gateway is, which is exactly when the user is relying on it. The stall
cooldown then widens one contention spike into a prefix-wide refusal, so a single
busy moment can lock out a whole directory tree. And the signal itself is
destroyed: a timeout was introduced (PR #8022) to mean "this mount is wedged", and
today a reader cannot tell that case from a busy interpreter.

This is not theoretical. Authoring this change, roughly twenty of my own file
reads, edits and shell calls were refused with that budget message, on ordinary
paths inside an ordinary project checkout, on a host whose disk was answering in
microseconds.

3. How our fix solves it

The chain, from what the user sees back to the cause: the path is refused, because
the budget expired, because the resolution took seconds, because each component's
syscall pays a GIL reacquisition, because the work runs on a thread sharing one
interpreter with 157 others. Only the last link is addressable, and it is
addressed by running the work in a different interpreter, where the parent's
cost is its own single wait instead of one wait per component.

This PR lands that as a reusable base, kiro_crew.subprocess_pool, because the same shape
applies to any pool doing syscall-shaped work. Two pieces:

  • subprocess_pool/_child_realpath.py, a leaf child script that imports stdlib only and
    answers one op code over stdin/stdout.
  • subprocess_pool/executor.py, SubprocessPoolExecutor, a
    concurrent.futures.Executor subclass, so a caller keeps holding a Future.

Design points that are load-bearing rather than incidental:

The transport runs on the calling thread. This is the part I got wrong first
and it is worth the space. An earlier revision of this same base handed the child
round trip to an internal ThreadPoolExecutor. It passed every correctness test
while not fixing the bug, because a pool worker puts two more thread boundaries
between the question and the answer (worker pickup, then the caller's wake-up from
a condition variable) and each waits up to a 5 ms switch interval. Same child,
same host, same contenders:

sibling threads in-thread this PR same child via a thread pool
0 0.067 0.102 0.112
8 179.538 10.292 202.201
24 760.313 25.377 1097.773
48 3480.685 81.241 2680.580

The pooled column is over budget at 48, so it would have shipped as a no-op with a
new subprocess to maintain. _CallerThreadFuture is therefore lazy: submit_op
captures the request and the work happens in result(), on the awaiting thread.
A test pins this by recording the thread that runs the transport, so the hop
cannot come back unnoticed.

The remaining 10-81 ms is understood rather than mysterious. A deadline costs two
GIL reacquisitions, one for the select and one for the read it clears, which is
the ~10 ms floor at 5 ms a handoff, plus the lease and future locks above it. That
leaves 24x headroom against the 2000 ms budget. Buying the last handoff back would
mean moving the deadline onto the reaper thread, which is itself unschedulable
under precisely the contention the deadline exists for.

spawn, not fork. Launch is Popen of python -S <script by path> with no
preexec_fn, which is the switch that would put Python back in the forked child;
without it CPython goes posix_spawn/vfork then exec, so no Python runs
between fork and exec and no other thread's lock is inherited locked.

The child never imports kiro_crew. It is run by path, never -m. Measured
cold start: python -S -c pass 9.9 ms, import os, posixpath 10.8 ms (what this
child pays), import kiro_crew 56.3 ms, import kiro_crew.security.paths
135.4 ms.

Nothing about the gate's configuration crosses the boundary either, and no anchor
list has to be passed in: the child answers exactly one question, "what does this
path resolve to", and the comparison against the denylist stays in the parent. That
is less to serialise, less to keep in sync, and it keeps the parent the only place
that knows what counts as sensitive.

Length-prefixed framing, never newline. A POSIX filename may contain a
newline; newline framing would read /a\nb as two values and shift every later
answer by one. Tested directly, including non-UTF-8 names.

One child is enough; two is the default. One sustains roughly 45k
resolutions/s at ~22 microseconds a call, so a second buys no throughput at all.
What it buys is wedge isolation, which the caller's gate already assumes: with a
single child, one stalled request refuses every path under every prefix until the
ceiling reclaims it. workers=1 is therefore permitted and documented with that
exact consequence, and 2 is the default because the resolver is the caller that
cannot accept it. Only workers=0 is rejected.

A timeout now means the mount did not answer. Each request carries an absolute
deadline enforced with select, and the budget covers waiting for a free child as
well as waiting for the answer, so lease contention cannot hand a caller a fresh
budget. This is what restores PR #8022's intent and stops #12386's cooldown being
triggered by GIL contention.

A child that dies or stalls fails closed, and is destroyed rather than reused.
Chosen explicitly between the two options: a missed deadline kills the child, and
the next request spawns a replacement (~11 ms), so a fault costs one refused
resolution rather than a degraded gateway. Nothing on any fault path returns an
empty or partial answer. SubprocessPoolUnavailable is raised instead, specifically so a
dead child can never be read as "this path resolves to nothing", which would turn
a crash into an open door. A reaper also kills any child that outlives a 20 s
ceiling, and it signals without taking the request lock, because taking that lock
would make the kill wait on the very wedge it exists to clear.

What CI surfaced, and why that is two fixes rather than one. Five backend shards
went red on the first run of this change and they did not share a cause. On shard 6
of both platforms the symptom was the spawn audit failing: the pool starts a real
interpreter with Popen, and every spawn in this repository must either route
through the sandboxed path or be declared benign, so an unrouted one is correctly a
failure. Routing it is not available here, for the reason set out under the security
questions below, so the child is listed in BENIGN_SPAWNS with its justification
recorded beside the entry. On Windows shard 3 the symptom was different and
platform-bound: every deadline-bearing request faulted, because the deadline
described above is built on select, which accepts only sockets on that platform
and therefore cannot watch the child's pipe. The deadline is POSIX-only now, with
the 20 s reaper as the bound elsewhere, and the tests whose subject is that deadline
or POSIX filename semantics are scoped accordingly; what that leaves is recorded as
a known limitation in section 4. The two shard-4 failures on the same runs were not
this change at all, but a pre-existing tripwire in the dashboard slot reader, fixed
in #12718 and cleared by rebasing onto it.

Security review of the new trust boundary

A subprocess is a new boundary, so answering the three questions directly.

Can an agent-supplied malicious path make the child return a wrong answer? The
path is carried as a length-prefixed byte string, never interpolated into a shell
command or an argv, and the child is launched with a fixed argv that contains no
caller data. There is no shell anywhere on the path. The framing is by length, so
no byte in a filename (newline, quote, NUL-adjacent, invalid UTF-8) can terminate
a field early or inject a second value; the non-UTF-8 and embedded-newline cases
are tested. Responses carry an echoed request id and the child is strictly serial,
so one path's answer cannot be attributed to another path's question; a mismatched
id is treated as fatal and the child discarded rather than reasoned about.

Does the parent trust the child beyond realpath semantics? No. The response is
a set of byte strings that the parent decodes with os.fsdecode and compares
against the denylist exactly as it compared the in-thread result. The parent takes
no action on the child's say-so, accepts no status other than ok/error, and treats
error as a refusal rather than as a permission.

Can the child answer a DIFFERENT question than the parent would? This is the one
real defect review found, and it is fixed rather than argued away. The child
originally ran with cwd pinned to the package directory, while the in-process
resolver it replaces anchors a relative path to the gateway's working directory. A
relative symlink into a credential store would therefore have resolved to a
non-existent path under the package and MISSED the denylist comparison, which is a
gate failing open. Two defences now. The caller absolutizes against its own working
directory before sending, which also pins the base to the moment of the call so a
concurrent os.chdir cannot move it underneath a request; and the child inherits
the parent's working directory rather than overriding it. abspath only joins and
normalises, so every symlink is still resolved in the child and the answer still
matches the in-process resolver for the same input.

Worth noting how that test was arrived at, because the first two attempts were
decorative. The two defences mask each other: chdir before the first request and
the child simply inherits the new directory, and with two children the second
request is served by the one that has not spawned yet, which also starts in the new
directory. The test that actually isolates the fix uses workers=1 so one child is
reused and its inherited directory goes stale. It fails without the absolutize and
passes with it, which is the only reason to believe it tests anything.

Why is the child not routed through the agent sandbox? Because sandboxing it
would break the thing it exists to do. sandboxed_spawn_argv is this repo's route
for an agent-influenced spawn, and the spawn audit rightly flags any other Popen.
This one is listed in BENIGN_SPAWNS instead, on two grounds. Nothing
agent-supplied reaches the command, the arguments or the working directory: the argv
is sys.executable -I -S <module-relative script>, the path to resolve travels over
stdin as a length-prefixed frame rather than as an argument, and no shell is
involved. And the child's whole job is to lstat and readlink the paths the
sensitive-path gate is checking, sensitive ones included, so under the agent sandbox
it would be denied exactly those reads -- and a resolver answering "cannot resolve"
where the truth is a credential symlink's target weakens the gate rather than
hardening it. The justification sits beside the allowlist entry.

Can code be injected into the child? Not now, and this was the second real defect
review found. Running outside the sandbox is what makes the child worth attacking,
and it was launched with -S but without -I, so the interpreter still honoured
PYTHONPATH. A writable directory ahead on sys.path could therefore plant a
module named for one of the four stdlib imports the child performs at startup and
get code execution in a process that can read credential files. It is launched
-I -S now, which drops PYTHON* variables, the user site directory and the
script's own directory from sys.path. That costs nothing: the child is launched by
absolute path and imports stdlib only, and -I changes neither the working
directory nor realpath semantics.

Worth stating plainly, because it is the part I got wrong rather than merely
missed: the comment justifying the omission claimed the child needed the parent's
environment "for realpath to mean the same thing", and that is simply untrue. -I
does not affect CWD-relative resolution, and the caller absolutizes before sending
regardless. A wrong reason in a comment is how a hole survives review, so the
comment now states why -I is required instead. Two tests pin it: one asserts a
planted pathlib.py does not execute in the isolated child, and a control asserts
the same plant does execute without -I, so the first is a measurement rather than
a formality.

Does the child execute anything? No. It imports os, pathlib, struct, sys
only (asserted by an AST test over its source, which also fails on any relative
import), calls resolution functions, and writes bytes back. It does not exec,
import by name from its input, eval, or touch the network. stderr goes to
DEVNULL deliberately, so a traceback cannot carry an agent-supplied path into
gateway logs; failures are reported in band as a status byte plus a class name and
no path, which is tested.

4. What tests we did

test/test_subprocess_pool.py, 42 tests, all passing. Run scoped to that
one file with an explicit -n0, per the shared-host rule:

pytest -q -n0 test/test_subprocess_pool.py gives 42 passed in 5.59s.

Coverage by group:

  • Framing: parametrised round trips including embedded newlines, trailing
    newlines, a bare newline, non-UTF-8 bytes, empty values and a 2 KB value; plus
    truncated frames rejected rather than guessed.
  • Child isolation: an AST test asserting the child imports stdlib only and uses no
    relative import; and the child answering when launched exactly as the pool
    launches it, with -S.
  • Resolution parity against the function it replaces, _resolved_spellings: a
    symlink, a name containing a newline, a non-UTF-8 name, a missing path, and a
    symlink repointed between two questions (must not be served from a pre-repoint
    reading).
  • Relative paths: parity with the in-process resolver for a bare name, a
    ./-prefixed name and a relative symlink; plus the stale-child-directory case
    with workers=1, which is the one that fails without the absolutize.
  • Fail-closed: a killed child respawns and never answers empty; a child that never
    answers is killed at the ceiling; an unknown op raises; an error response carries
    no path.
  • Caller-thread dispatch: the transport thread is recorded and asserted to be the
    calling thread, for both entry points; no worker pool exists for child work;
    an unawaited future cancels cleanly and truthfully reports nothing ran; an
    awaited one refuses to cancel; a second await reuses the answer.
  • Caller budget: a silent child raises TimeoutError at the caller's budget with
    the ceiling parked far away, the timed-out child is destroyed rather than reused,
    and the future surfaces the budget as the same exception class a pooled future
    raises.
  • Executor contract: workers=0 rejected and workers=1 permitted; submit still runs an arbitrary local
    closure on a thread; map still works.

Gates on the changed files: flake8 clean, isort clean, black clean, mypy
"Success: no issues found in 3 source files". CHANGELOG.md is untouched.

Measurements were produced by two standalone harnesses (a contention benchmark and
a cold-start probe) kept outside the repo; both are reproducible from the numbers
and method described above, and I can land them under a bench path if reviewers
want them in-tree.

Still pending, named rather than held for. The contention table is medians of 7
iterations at 0/8/24/48 contenders on one host, not a full matrix; more contender
steps, more iterations and a second host would tighten it. The conclusion does not
turn on that precision, since the shipped path sits 24x inside the budget and the
pooled control is over it.

On Windows the per-request read deadline is not enforced by the read at all,
because select accepts only sockets there and so cannot watch the child's pipe.
The ceiling reaper bounds a wedge instead, so a wedged child costs up to
ceiling_secs rather than the caller's own budget, and the five tests whose
subject is that deadline or POSIX filename semantics are scoped accordingly.
Making it exact means moving the transport onto a socket pair, which is selectable
on both platforms. The gateway this serves runs on POSIX, so this is recorded as a
known limitation rather than done here.

5. Any other suggestions on the work

Answering "is there an existing pattern or factory for this". Yes for the
interface, no for the transport.

concurrent.futures.Executor is the documented extension point and is what this
subclasses, so swapping a thread pool for a subprocess becomes a change of
factory function rather than a change at every call site. One correction worth
recording: it is not an ABC. Executor.__abstractmethods__ is None and its
MRO is (Executor, object), so a subclass that forgets submit fails at call
time with NotImplementedError rather than at construction. Nothing enforces the
contract, so honouring it is the subclass's job.

ProcessPoolExecutor is the ready-made cross-process Executor and is not
usable here, for two reasons, of which the structural one decides it. It pickles
the callable by reference, and a callable defined inside another function has no
importable name, so submitting one raises AttributeError: Can't get local object
(measured) and the resolver's pool is fed exactly such a local wrapper. More
fundamentally, its results reach the caller through a parent-side manager thread
that completes the Future, which is the same two-boundary crossing measured at
2680 ms above, so a warm pool would not rescue it either. That inference is from
the shared mechanism, not a direct measurement of that class. The import cost is
the weakest of the objections, since a pool amortises it over a warm child, and it
is 135 ms rather than the seconds one might assume.

From multiprocessing, Connection.send_bytes/recv_bytes would have supplied
length-prefixed framing for free, and get_context("spawn") the process
semantics. They were not used because reaching either drags multiprocessing and
its pickling, manager and semaphore machinery into a path whose entire value is a
child that imports almost nothing; the framing it would have saved is ~20 lines
and is directly tested. socket.socketpair over pipes buys nothing while the
protocol is serial. os.posix_spawn is already what Popen uses here.

No third-party dependency is warranted: this is ~100 lines of stdlib against a new
supply-chain surface inside a security gate.

The child is the pool's first consumer, not part of the pool. subprocess_pool/_child_realpath.py sits inside the package only because both land in this change; it says so in its own module docstring. It moves next to the sensitive-path resolver it serves in the wiring PR, so the package left behind is generic on the page as well as in the code.

Scope note, and the reason this PR stops where it does. The resolver is not
yet wired to the base, and wiring it needs a decision rather than more code.
security/paths.py submits a local wrapper whose side effects the caller reads
back: it records the worker's native thread id and runs a started/abandoned
handshake under a lock, and at the deadline the caller uses future.cancel() plus
started.is_set() to decide whether a timeout was "pool saturated, do not charge
this prefix" or "the mount is wedged, charge it", with timed-out futures counted
against a worker ceiling. Every one of those signals is thread-pool-shaped and has
no referent once the work runs on the calling thread. So the honest options are to
adapt that classifier or to keep an equivalent signal in the executor, and
paths.py currently has two other changes in flight (the stall prefix, and TTL
plus positive caching), so the sequencing is a coordination question. I have left
that file untouched and am raising it rather than guessing.

For the next pool someone wants to move off threads, four things, and only the
last two need judgement:

  1. Write a leaf child script beside _child_realpath.py that imports stdlib only
    and answers one op code. Run it by path, never -m, or kiro_crew/__init__.py
    executes and the cold-start budget is gone.
  2. Give it a request/response body in the same shape: 4-byte big-endian outer
    length, request id, op code, payload. Length prefixes, never a newline.
  3. Reach it with call_op from the thread that wants the answer. If that thread
    instead waits on another thread's result, the win is gone and it will look like
    the child is at fault. Check the work is genuinely syscall-shaped first: this
    converts "blocks in the kernel" into "blocks in the kernel one process over",
    so it does nothing for CPU-bound Python.
  4. Decide what the caller does when a child dies or outlives the ceiling, and make
    sure no fault path can return an empty answer that reads as a real one.

Refs #10255

@chenmingwei23
chenmingwei23 requested review from a team and pepmach September 22, 2026 04:36
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 77a2ba92e95ebaefe6bef27d6ef81894ccbd8f65 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Mechanism lands with zero consumers, and its Future face is shaped for a stall classifier the caller-thread design makes meaningless — wiring may orphan or reshape it.

Watch

  • submit_op/_CallerThreadFuture is speculative surface: the PR's own section 5 says the one intended caller (security/paths.py) classifies timeouts via future.cancel() + started.is_set(), signals that "have no referent once the work runs on the calling thread" — and the class docstring warns a caller reading cancel() as "pool saturated" "must be revisited". The API most likely to change in the wiring PR is being frozen before that decision.
    Clears when: the wiring PR consumes submit_op as shipped, or the Future face is deferred to that PR and call_op is the only child entry point here.
  • On Windows the per-request deadline is unenforced (_CAN_SELECT_PIPES), so once wired, the resolver's 2 s budget degrades to the 20 s ceiling there — a supported platform per platform-compat. Documented here, but the wiring PR must decide it, not inherit it silently.
    Clears when: the wiring PR states the Windows behavior explicitly (socketpair transport, or a documented ceiling-bounded budget).

Suggestions

  • Land call_op as the only op entry point and add the Future face in the wiring PR — it removes the one piece both the description and the docstring flag as possibly wrong-shaped.

[DESIGN-REVIEWED] 77a2ba9

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 77a2ba92e95ebaefe6bef27d6ef81894ccbd8f65 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The entire mechanism ships with zero production consumers — the resolver it exists for is unwired by declaration — and its generic Executor face is sized for one caller.

Not justified as shipped

  • Item 1 — zero consumers: "the resolver is not wired to it yet" (description); grepped subprocess_pool repo-wide: 0 production call sites (the executors.py hits are base's unrelated _subprocess_pool ThreadPoolExecutor variable). Until the wiring decision in the description's section 5 is made, every line is speculative surface and the reported defect (Sensitive-path gate refuses ordinary files under load: resolver budget is spent on GIL handoffs, not the disk #10255) is exactly as present with this merged as without it.
    Subtraction: defer the package to the PR that wires src/kiro_crew/security/paths.py:1747 to it, so mechanism and consumer land together.
    Clears when: a production call site consumes call_op/realpath_spellings — the resolver wiring merged.
  • Item 3 — one consumer, generalized: submit/map/_fallback/submit_op/_CallerThreadFuture exist so the class can "replace a ThreadPoolExecutor in a factory without changing any caller", but the only planned caller is the single submit site at paths.py:1747, and the class's own docstring says that caller must use submit_op explicitly (op inference "is not usable") — so the drop-in face protects nobody. Grepped: 0 non-test callers of submit/submit_op/map.
    Subtraction: delete submit, map, _fallback, submit_op and _CallerThreadFuture; keep call_op + realpath_spellings.
    Clears when: the wiring shows a caller submitting unrecognised callables through this class, or the deletion is made.

What this change ships

Inventory (8 items) — 6 justified

Intent: stop the sensitive-path gate refusing legitimate paths when the interpreter, not the disk, is busy (refs #10255) — a FIX by motivation, but what ships is an ADDITION: the mechanism alone, unwired by the author's own declaration.

  1. A new importable package kiro_crew.subprocess_pool runs symlink resolution in a child interpreter — zero consumers (resolver unwired)
  2. A stdlib-only child script answers one length-prefixed op over stdin/stdout — justified
  3. submit/map silently fall back to an internal thread pool, for drop-in ThreadPoolExecutor substitution — one consumer, generalized
  4. submit_op futures run their transport on the awaiting thread, not a worker — justified
  5. A reaper thread kills any child holding a request past a 20 s ceiling — justified
  6. A dead, wedged or faulted child raises; the pool never returns an empty answer — justified
  7. The child launches with -I, so a planted PYTHONPATH module cannot execute in it — justified
  8. The child's spawn is allowlisted in the spawn audit to run outside the agent sandbox — justified

[FIRST-PRINCIPLES-REVIEWED] 77a2ba9

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 77a2ba92e95ebaefe6bef27d6ef81894ccbd8f65 — this comment is updated in place on each push.

Review details

Both candidates concern the new subprocess_pool module. The decisive fact is that SubprocessPoolExecutor / realpath_spellings has no production caller in this diff — the module docstrings and _child_realpath.py state it is "NOT PART OF THE POOL … the pool's FIRST CONSUMER … moves there when that caller is wired up," and grep-visible usage is only tests. I verified the resolver seam too.

Candidate 1 (abspath collapses .. before symlink resolution): The divergence is real in the abstract — os.path.abspath runs normpath, which lexically folds link/.. before the child's realpath can resolve link, while _resolved_spellings(expanded) (paths.py:1354) is fed a non-normpath'd expanded (the caller at paths.py:1944 uses os.path.join, not normpath, and adds the lexical forms separately at 1955-1956). But realpath_spellings is not wired into any gate here. The only "wrong outcome" is a future integration failing open — an explicit "if a caller were to." (c) is not reachable in the changed code. Dropped.

Candidate 2 (atexit.register never unregistered): Factually accurate — shutdown() never calls atexit.unregister, so a bound method holding self persists. But for the intended single long-lived executor the registration is correct and desirable, and the accumulation requires "a process that recreates the pool" — no such production caller exists in the diff. It is a bounded retention concern, not a wrong-answer defect, and the pipeline excludes it besides. Dropped.

Step 2 turned up no independently groundable defect with a reachable caller: the whole module is unwired, so every candidate failure mode terminates at "once a caller exists."

No findings.

[OPUS-REVIEWED] 77a2ba9

Verdict parsed from the review's SHA-scoped output markers for commit 77a2ba92e95ebaefe6bef27d6ef81894ccbd8f65.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 77a2ba92e95ebaefe6bef27d6ef81894ccbd8f65: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 77a2ba92e95ebaefe6bef27d6ef81894ccbd8f65 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 77a2ba9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 77a2ba92e95ebaefe6bef27d6ef81894ccbd8f65: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the fix/resolver-offload-subprocess branch from dfb7434 to 9c8e115 Compare September 22, 2026 04:49
@chenmingwei23 chenmingwei23 changed the title Add a subprocess offload base for syscall-shaped work feat(offload): add a subprocess offload base for syscall-shaped work Sep 22, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision 77a2ba92e95ebaefe6bef27d6ef81894ccbd8f65 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging b01524589019801f3871774849036f7155683d8c. Adjudication produced no [GPT-ADJUDICATED] marker for b015245: the adjudication stage FAILED TO RUN (its raw output is in the Adjudication details block -- typically an API or setup error), so no proportionality review took place and GPT's blocking verdict stands unreviewed.

This comment is updated in place on each push.

BLOCKING -- test/test_mcp_rebuild_source_reconcile.py:79 -- New xfail masks stale command reconciliation
@pytest.mark.xfail(
Changed MCP command -> rebuild preserves prior executable -> generated spec launches stale command.
Anchor: a-ratchet-may-only-tighten
Fix: Remove the xfail and safely reconcile transport changes as a unit.
[BLOCK-MERGE] b015245
[GPT-REVIEWED] b015245
False positive or not applicable? A repository writer can comment:
/ai-review override gpt b01524589019801f3871774849036f7155683d8c: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@chenmingwei23
chenmingwei23 marked this pull request as draft September 22, 2026 06:19
@chenmingwei23
chenmingwei23 force-pushed the fix/resolver-offload-subprocess branch from b015245 to b06557d Compare September 22, 2026 06:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for b06557d26ba1d7c2b51cc82e81b9c1e6e5c9b28b; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b06557d26ba1d7c2b51cc82e81b9c1e6e5c9b28b: <one-sentence reason>

2 similar comments
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for b06557d26ba1d7c2b51cc82e81b9c1e6e5c9b28b; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b06557d26ba1d7c2b51cc82e81b9c1e6e5c9b28b: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for b06557d26ba1d7c2b51cc82e81b9c1e6e5c9b28b; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b06557d26ba1d7c2b51cc82e81b9c1e6e5c9b28b: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for a111119bf9fd25338b28783dc9a910f4b1699a23; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a111119bf9fd25338b28783dc9a910f4b1699a23: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 22, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 22, 2026 16:10
@iamwhatever
iamwhatever merged commit f7b6e6f into main Sep 22, 2026
103 checks passed
@iamwhatever
iamwhatever deleted the fix/resolver-offload-subprocess branch September 22, 2026 16:12
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 22, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor

@chenmingwei23 thanks for landing this; the diagnosis matches what #10256 and #12739 measured (rebuild 1.2 / 7.6 / 15.4 / 20.5 s median at 1/2/4/8 GIL hogs against the 8 s + 4 s budget; the same batch in a child in tens of ms).

On the scope-note question: I am writing the wiring PR against this base now, and will retire #12739 and #10256 in its favour -- their helper and client are the part your pool now provides, so carrying them would be duplicate machinery. The classifier decision I intend to take, since it is the piece those two PRs already worked through: the thread-shaped "was the worker started" signal becomes a child-shaped one. At the deadline, sample /proc/<child_pid>/syscall (POSIX) to distinguish "child blocked in the filesystem, charge the prefix" from "request still queued or interpreter starved, do not charge"; on Windows/macOS keep the grace-wait discriminator main already has. Fault paths (SubprocessPoolUnavailable, kill-at-ceiling) raise PathResolutionStalled so no empty answer reads as resolved; spawn failure falls back to in-thread realpath with a warning. The anchor rebuild goes as one batched request, and _child_realpath.py moves next to the resolver as your body says.

If you would rather do the wiring yourself or want the classifier handled differently, say so here and I will stand down or adjust; otherwise expect a draft PR shortly. Also: #10255 was closed at 07:20Z today without a linked fix; the executor swap is what actually closes it, so it may be worth reopening until the wiring lands.

bolichen97 added a commit that referenced this pull request Sep 22, 2026
…12914)

test_a_name_that_is_not_utf8_round_trips (#12709) creates a directory
whose name is not valid UTF-8 to compare the child's realpath against
the thread's. APFS/HFS+ refuse such a name unit: os.mkdir raises
EILSEQ before the resolver is reached, so the macOS lane fails on the
fixture, not on the code under test. Every other non-UTF-8 path test
in the suite already skips on darwin as well as Windows (#11566); give
this one the same marker and keep the newline-name case, which is
legal on APFS, running there.
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 24, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 24, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 24, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 24, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 24, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 25, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 25, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 25, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 25, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 26, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 26, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Pearcekieser added a commit to Pearcekieser/KiroCrew that referenced this pull request Sep 26, 2026
…, from the calling thread

Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate,
the root anchors and the target rebuild each travel as one request to the child
interpreter, made by the thread that wants the answer, so the budget is spent
waiting for the disk rather than for the GIL. The stall classifier is adapted to
child-shaped signals: a lease timeout means no child ran (not charged), and a
SubprocessPoolTimeout carries the child's syscall sampled before it was killed,
read against the existing stat/readlink table. Transport faults refuse
fail-closed with no cooldown; only a child that cannot start falls back
in-process. The child script moves beside the resolver it serves.

Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants