Repository navigation
feat(subprocess-pool): an Executor that runs work in a separate interpreter - #12709
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Mechanism lands with zero consumers, and its Future face is shaped for a stall classifier the caller-thread design makes meaningless — wiring may orphan or reshape it. Watch
Suggestions
[DESIGN-REVIEWED] 77a2ba9 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The entire mechanism ships with zero production consumers — the resolver it exists for is unwired by declaration — and its generic Executor face is sized for one caller. Not justified as shipped
What this change shipsInventory (8 items) — 6 justifiedIntent: stop the sensitive-path gate refusing legitimate paths when the interpreter, not the disk, is busy (refs #10255) — a FIX by motivation, but what ships is an ADDITION: the mechanism alone, unwired by the author's own declaration.
[FIRST-PRINCIPLES-REVIEWED] 77a2ba9 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsBoth candidates concern the new Candidate 1 ( Candidate 2 ( Step 2 turned up no independently groundable defect with a reachable caller: the whole module is unwired, so every candidate failure mode terminates at "once a caller exists." No findings. [OPUS-REVIEWED] 77a2ba9 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
dfb7434 to
9c8e115
Compare
UX Review (Fable 5) — ⏭️ skippedRevision |
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- test/test_mcp_rebuild_source_reconcile.py:79 -- New xfail masks stale command reconciliation |
b015245 to
b06557d
Compare
GPT 5.6 Review —
|
2 similar comments
GPT 5.6 Review —
|
GPT 5.6 Review —
|
GPT 5.6 Review —
|
|
@chenmingwei23 thanks for landing this; the diagnosis matches what #10256 and #12739 measured (rebuild 1.2 / 7.6 / 15.4 / 20.5 s median at 1/2/4/8 GIL hogs against the 8 s + 4 s budget; the same batch in a child in tens of ms). On the scope-note question: I am writing the wiring PR against this base now, and will retire #12739 and #10256 in its favour -- their helper and client are the part your pool now provides, so carrying them would be duplicate machinery. The classifier decision I intend to take, since it is the piece those two PRs already worked through: the thread-shaped "was the worker started" signal becomes a child-shaped one. At the deadline, sample If you would rather do the wiring yourself or want the classifier handled differently, say so here and I will stand down or adjust; otherwise expect a draft PR shortly. Also: #10255 was closed at 07:20Z today without a linked fix; the executor swap is what actually closes it, so it may be worth reopening until the wiring lands. |
…12914) test_a_name_that_is_not_utf8_round_trips (#12709) creates a directory whose name is not valid UTF-8 to compare the child's realpath against the thread's. APFS/HFS+ refuse such a name unit: os.mkdir raises EILSEQ before the resolver is reached, so the macOS lane fails on the fixture, not on the code under test. Every other non-UTF-8 path test in the suite already skips on darwin as well as Windows (#11566); give this one the same marker and keep the newline-name case, which is legal on APFS, running there.
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
…, from the calling thread Wire the sensitive-path resolver to kiro_crew.subprocess_pool: the candidate, the root anchors and the target rebuild each travel as one request to the child interpreter, made by the thread that wants the answer, so the budget is spent waiting for the disk rather than for the GIL. The stall classifier is adapted to child-shaped signals: a lease timeout means no child ran (not charged), and a SubprocessPoolTimeout carries the child's syscall sampled before it was killed, read against the existing stat/readlink table. Transport faults refuse fail-closed with no cooldown; only a child that cannot start falls back in-process. The child script moves beside the resolver it serves. Refs kirodotdev#10255, kirodotdev#12709, kirodotdev#10256, kirodotdev#12739
Base commit:
64fbda5f9170ad4614cb683d3e44a8f735615d41Opened as a DRAFT. The base it lands is complete and tested; what is not yet done
is wiring the path resolver to it, which needs a decision rather than more code
(see section 5). Iterating here rather than in the worktree so the review lanes
run in parallel.
1. What is the problem?
The sensitive-path gate refuses legitimate paths because its symlink-resolution
budget is spent waiting for the GIL rather than for the disk.
os.path.realpathis pure Python.posixpath._joinrealpathwalks a path onecomponent at a time, doing an
lstat/readlinkpair per component, and each ofthose releases and reacquires the GIL. A reacquisition waits up to one
sys.getswitchinterval(), measured 5.0 ms on this host. The cost is therefore"components x 2 x contention", which has nothing to do with how fast the
filesystem answers.
Median ms to resolve one six-component path, with N pure-Python CPU-bound sibling
threads running alongside:
The default budget is 2000 ms (
_PATH_RESOLVE_TIMEOUT_SECS) and the anchorrebuild is 8000 ms, so the in-thread path crosses its budget somewhere between 24
and 48 contenders. This gateway was measured carrying 157 threads.
The filesystem is not implicated. There is one mount, local xfs, and every
underlying
realpathof a refused path answered in 14-44 microseconds.The existing knob cannot reach this.
KIROCREW_PATH_RESOLVE_WORKERS(default 2,max 64) sizes a thread pool, so each added worker is one more GIL contender.
Moving the resolution into the kernel is also not enough on its own: a single
O_PATHresolution measured 60.8 ms at 24 threads but 2387 ms at 48, still overbudget.
Refs #10255. This PR does not close it: it lands the mechanism the fix needs, and
the resolver is not wired to it yet (see section 5).
2. Why this issue matters to the user
The user sees their own tool calls refused, with a message saying the path could
not be verified within the resolver budget. Failing closed is correct once the
budget expires, but the budget is expiring for the wrong reason: interpreter load,
not an unhealthy mount.
Three consequences follow. Legitimate work is blocked, and blocked more often the
busier the gateway is, which is exactly when the user is relying on it. The stall
cooldown then widens one contention spike into a prefix-wide refusal, so a single
busy moment can lock out a whole directory tree. And the signal itself is
destroyed: a timeout was introduced (PR #8022) to mean "this mount is wedged", and
today a reader cannot tell that case from a busy interpreter.
This is not theoretical. Authoring this change, roughly twenty of my own file
reads, edits and shell calls were refused with that budget message, on ordinary
paths inside an ordinary project checkout, on a host whose disk was answering in
microseconds.
3. How our fix solves it
The chain, from what the user sees back to the cause: the path is refused, because
the budget expired, because the resolution took seconds, because each component's
syscall pays a GIL reacquisition, because the work runs on a thread sharing one
interpreter with 157 others. Only the last link is addressable, and it is
addressed by running the work in a different interpreter, where the parent's
cost is its own single wait instead of one wait per component.
This PR lands that as a reusable base,
kiro_crew.subprocess_pool, because the same shapeapplies to any pool doing syscall-shaped work. Two pieces:
subprocess_pool/_child_realpath.py, a leaf child script that imports stdlib only andanswers one op code over stdin/stdout.
subprocess_pool/executor.py,SubprocessPoolExecutor, aconcurrent.futures.Executorsubclass, so a caller keeps holding aFuture.Design points that are load-bearing rather than incidental:
The transport runs on the calling thread. This is the part I got wrong first
and it is worth the space. An earlier revision of this same base handed the child
round trip to an internal
ThreadPoolExecutor. It passed every correctness testwhile not fixing the bug, because a pool worker puts two more thread boundaries
between the question and the answer (worker pickup, then the caller's wake-up from
a condition variable) and each waits up to a 5 ms switch interval. Same child,
same host, same contenders:
The pooled column is over budget at 48, so it would have shipped as a no-op with a
new subprocess to maintain.
_CallerThreadFutureis therefore lazy:submit_opcaptures the request and the work happens in
result(), on the awaiting thread.A test pins this by recording the thread that runs the transport, so the hop
cannot come back unnoticed.
The remaining 10-81 ms is understood rather than mysterious. A deadline costs two
GIL reacquisitions, one for the
selectand one for thereadit clears, which isthe ~10 ms floor at 5 ms a handoff, plus the lease and future locks above it. That
leaves 24x headroom against the 2000 ms budget. Buying the last handoff back would
mean moving the deadline onto the reaper thread, which is itself unschedulable
under precisely the contention the deadline exists for.
spawn, not fork. Launch is
Popenofpython -S <script by path>with nopreexec_fn, which is the switch that would put Python back in the forked child;without it CPython goes
posix_spawn/vforkthenexec, so no Python runsbetween fork and exec and no other thread's lock is inherited locked.
The child never imports
kiro_crew. It is run by path, never-m. Measuredcold start:
python -S -c pass9.9 ms,import os, posixpath10.8 ms (what thischild pays),
import kiro_crew56.3 ms,import kiro_crew.security.paths135.4 ms.
Nothing about the gate's configuration crosses the boundary either, and no anchor
list has to be passed in: the child answers exactly one question, "what does this
path resolve to", and the comparison against the denylist stays in the parent. That
is less to serialise, less to keep in sync, and it keeps the parent the only place
that knows what counts as sensitive.
Length-prefixed framing, never newline. A POSIX filename may contain a
newline; newline framing would read
/a\nbas two values and shift every lateranswer by one. Tested directly, including non-UTF-8 names.
One child is enough; two is the default. One sustains roughly 45k
resolutions/s at ~22 microseconds a call, so a second buys no throughput at all.
What it buys is wedge isolation, which the caller's gate already assumes: with a
single child, one stalled request refuses every path under every prefix until the
ceiling reclaims it.
workers=1is therefore permitted and documented with thatexact consequence, and 2 is the default because the resolver is the caller that
cannot accept it. Only
workers=0is rejected.A timeout now means the mount did not answer. Each request carries an absolute
deadline enforced with
select, and the budget covers waiting for a free child aswell as waiting for the answer, so lease contention cannot hand a caller a fresh
budget. This is what restores PR #8022's intent and stops #12386's cooldown being
triggered by GIL contention.
A child that dies or stalls fails closed, and is destroyed rather than reused.
Chosen explicitly between the two options: a missed deadline kills the child, and
the next request spawns a replacement (~11 ms), so a fault costs one refused
resolution rather than a degraded gateway. Nothing on any fault path returns an
empty or partial answer.
SubprocessPoolUnavailableis raised instead, specifically so adead child can never be read as "this path resolves to nothing", which would turn
a crash into an open door. A reaper also kills any child that outlives a 20 s
ceiling, and it signals without taking the request lock, because taking that lock
would make the kill wait on the very wedge it exists to clear.
What CI surfaced, and why that is two fixes rather than one. Five backend shards
went red on the first run of this change and they did not share a cause. On shard 6
of both platforms the symptom was the spawn audit failing: the pool starts a real
interpreter with
Popen, and every spawn in this repository must either routethrough the sandboxed path or be declared benign, so an unrouted one is correctly a
failure. Routing it is not available here, for the reason set out under the security
questions below, so the child is listed in
BENIGN_SPAWNSwith its justificationrecorded beside the entry. On Windows shard 3 the symptom was different and
platform-bound: every deadline-bearing request faulted, because the deadline
described above is built on
select, which accepts only sockets on that platformand therefore cannot watch the child's pipe. The deadline is POSIX-only now, with
the 20 s reaper as the bound elsewhere, and the tests whose subject is that deadline
or POSIX filename semantics are scoped accordingly; what that leaves is recorded as
a known limitation in section 4. The two shard-4 failures on the same runs were not
this change at all, but a pre-existing tripwire in the dashboard slot reader, fixed
in #12718 and cleared by rebasing onto it.
Security review of the new trust boundary
A subprocess is a new boundary, so answering the three questions directly.
Can an agent-supplied malicious path make the child return a wrong answer? The
path is carried as a length-prefixed byte string, never interpolated into a shell
command or an argv, and the child is launched with a fixed argv that contains no
caller data. There is no shell anywhere on the path. The framing is by length, so
no byte in a filename (newline, quote, NUL-adjacent, invalid UTF-8) can terminate
a field early or inject a second value; the non-UTF-8 and embedded-newline cases
are tested. Responses carry an echoed request id and the child is strictly serial,
so one path's answer cannot be attributed to another path's question; a mismatched
id is treated as fatal and the child discarded rather than reasoned about.
Does the parent trust the child beyond
realpathsemantics? No. The response isa set of byte strings that the parent decodes with
os.fsdecodeand comparesagainst the denylist exactly as it compared the in-thread result. The parent takes
no action on the child's say-so, accepts no status other than ok/error, and treats
error as a refusal rather than as a permission.
Can the child answer a DIFFERENT question than the parent would? This is the one
real defect review found, and it is fixed rather than argued away. The child
originally ran with
cwdpinned to the package directory, while the in-processresolver it replaces anchors a relative path to the gateway's working directory. A
relative symlink into a credential store would therefore have resolved to a
non-existent path under the package and MISSED the denylist comparison, which is a
gate failing open. Two defences now. The caller absolutizes against its own working
directory before sending, which also pins the base to the moment of the call so a
concurrent
os.chdircannot move it underneath a request; and the child inheritsthe parent's working directory rather than overriding it.
abspathonly joins andnormalises, so every symlink is still resolved in the child and the answer still
matches the in-process resolver for the same input.
Worth noting how that test was arrived at, because the first two attempts were
decorative. The two defences mask each other: chdir before the first request and
the child simply inherits the new directory, and with two children the second
request is served by the one that has not spawned yet, which also starts in the new
directory. The test that actually isolates the fix uses
workers=1so one child isreused and its inherited directory goes stale. It fails without the absolutize and
passes with it, which is the only reason to believe it tests anything.
Why is the child not routed through the agent sandbox? Because sandboxing it
would break the thing it exists to do.
sandboxed_spawn_argvis this repo's routefor an agent-influenced spawn, and the spawn audit rightly flags any other
Popen.This one is listed in
BENIGN_SPAWNSinstead, on two grounds. Nothingagent-supplied reaches the command, the arguments or the working directory: the argv
is
sys.executable -I -S <module-relative script>, the path to resolve travels overstdin as a length-prefixed frame rather than as an argument, and no shell is
involved. And the child's whole job is to
lstatandreadlinkthe paths thesensitive-path gate is checking, sensitive ones included, so under the agent sandbox
it would be denied exactly those reads -- and a resolver answering "cannot resolve"
where the truth is a credential symlink's target weakens the gate rather than
hardening it. The justification sits beside the allowlist entry.
Can code be injected into the child? Not now, and this was the second real defect
review found. Running outside the sandbox is what makes the child worth attacking,
and it was launched with
-Sbut without-I, so the interpreter still honouredPYTHONPATH. A writable directory ahead onsys.pathcould therefore plant amodule named for one of the four stdlib imports the child performs at startup and
get code execution in a process that can read credential files. It is launched
-I -Snow, which dropsPYTHON*variables, the user site directory and thescript's own directory from
sys.path. That costs nothing: the child is launched byabsolute path and imports stdlib only, and
-Ichanges neither the workingdirectory nor
realpathsemantics.Worth stating plainly, because it is the part I got wrong rather than merely
missed: the comment justifying the omission claimed the child needed the parent's
environment "for realpath to mean the same thing", and that is simply untrue.
-Idoes not affect CWD-relative resolution, and the caller absolutizes before sending
regardless. A wrong reason in a comment is how a hole survives review, so the
comment now states why
-Iis required instead. Two tests pin it: one asserts aplanted
pathlib.pydoes not execute in the isolated child, and a control assertsthe same plant does execute without
-I, so the first is a measurement rather thana formality.
Does the child execute anything? No. It imports
os,pathlib,struct,sysonly (asserted by an AST test over its source, which also fails on any relative
import), calls resolution functions, and writes bytes back. It does not exec,
import by name from its input, eval, or touch the network.
stderrgoes toDEVNULLdeliberately, so a traceback cannot carry an agent-supplied path intogateway logs; failures are reported in band as a status byte plus a class name and
no path, which is tested.
4. What tests we did
test/test_subprocess_pool.py, 42 tests, all passing. Run scoped to thatone file with an explicit
-n0, per the shared-host rule:pytest -q -n0 test/test_subprocess_pool.pygives 42 passed in 5.59s.Coverage by group:
newlines, a bare newline, non-UTF-8 bytes, empty values and a 2 KB value; plus
truncated frames rejected rather than guessed.
relative import; and the child answering when launched exactly as the pool
launches it, with
-S._resolved_spellings: asymlink, a name containing a newline, a non-UTF-8 name, a missing path, and a
symlink repointed between two questions (must not be served from a pre-repoint
reading).
./-prefixed name and a relative symlink; plus the stale-child-directory casewith
workers=1, which is the one that fails without the absolutize.answers is killed at the ceiling; an unknown op raises; an error response carries
no path.
calling thread, for both entry points; no worker pool exists for child work;
an unawaited future cancels cleanly and truthfully reports nothing ran; an
awaited one refuses to cancel; a second await reuses the answer.
TimeoutErrorat the caller's budget withthe ceiling parked far away, the timed-out child is destroyed rather than reused,
and the future surfaces the budget as the same exception class a pooled future
raises.
workers=0rejected andworkers=1permitted;submitstill runs an arbitrary localclosure on a thread;
mapstill works.Gates on the changed files: flake8 clean, isort clean, black clean, mypy
"Success: no issues found in 3 source files".
CHANGELOG.mdis untouched.Measurements were produced by two standalone harnesses (a contention benchmark and
a cold-start probe) kept outside the repo; both are reproducible from the numbers
and method described above, and I can land them under a bench path if reviewers
want them in-tree.
Still pending, named rather than held for. The contention table is medians of 7
iterations at 0/8/24/48 contenders on one host, not a full matrix; more contender
steps, more iterations and a second host would tighten it. The conclusion does not
turn on that precision, since the shipped path sits 24x inside the budget and the
pooled control is over it.
On Windows the per-request read deadline is not enforced by the read at all,
because
selectaccepts only sockets there and so cannot watch the child's pipe.The ceiling reaper bounds a wedge instead, so a wedged child costs up to
ceiling_secsrather than the caller's own budget, and the five tests whosesubject is that deadline or POSIX filename semantics are scoped accordingly.
Making it exact means moving the transport onto a socket pair, which is selectable
on both platforms. The gateway this serves runs on POSIX, so this is recorded as a
known limitation rather than done here.
5. Any other suggestions on the work
Answering "is there an existing pattern or factory for this". Yes for the
interface, no for the transport.
concurrent.futures.Executoris the documented extension point and is what thissubclasses, so swapping a thread pool for a subprocess becomes a change of
factory function rather than a change at every call site. One correction worth
recording: it is not an ABC.
Executor.__abstractmethods__isNoneand itsMRO is
(Executor, object), so a subclass that forgetssubmitfails at calltime with
NotImplementedErrorrather than at construction. Nothing enforces thecontract, so honouring it is the subclass's job.
ProcessPoolExecutoris the ready-made cross-process Executor and is notusable here, for two reasons, of which the structural one decides it. It pickles
the callable by reference, and a callable defined inside another function has no
importable name, so submitting one raises
AttributeError: Can't get local object(measured) and the resolver's pool is fed exactly such a local wrapper. More
fundamentally, its results reach the caller through a parent-side manager thread
that completes the
Future, which is the same two-boundary crossing measured at2680 ms above, so a warm pool would not rescue it either. That inference is from
the shared mechanism, not a direct measurement of that class. The import cost is
the weakest of the objections, since a pool amortises it over a warm child, and it
is 135 ms rather than the seconds one might assume.
From
multiprocessing,Connection.send_bytes/recv_byteswould have suppliedlength-prefixed framing for free, and
get_context("spawn")the processsemantics. They were not used because reaching either drags
multiprocessingandits pickling, manager and semaphore machinery into a path whose entire value is a
child that imports almost nothing; the framing it would have saved is ~20 lines
and is directly tested.
socket.socketpairover pipes buys nothing while theprotocol is serial.
os.posix_spawnis already whatPopenuses here.No third-party dependency is warranted: this is ~100 lines of stdlib against a new
supply-chain surface inside a security gate.
The child is the pool's first consumer, not part of the pool.
subprocess_pool/_child_realpath.pysits inside the package only because both land in this change; it says so in its own module docstring. It moves next to the sensitive-path resolver it serves in the wiring PR, so the package left behind is generic on the page as well as in the code.Scope note, and the reason this PR stops where it does. The resolver is not
yet wired to the base, and wiring it needs a decision rather than more code.
security/paths.pysubmits a local wrapper whose side effects the caller readsback: it records the worker's native thread id and runs a
started/abandonedhandshake under a lock, and at the deadline the caller uses
future.cancel()plusstarted.is_set()to decide whether a timeout was "pool saturated, do not chargethis prefix" or "the mount is wedged, charge it", with timed-out futures counted
against a worker ceiling. Every one of those signals is thread-pool-shaped and has
no referent once the work runs on the calling thread. So the honest options are to
adapt that classifier or to keep an equivalent signal in the executor, and
paths.pycurrently has two other changes in flight (the stall prefix, and TTLplus positive caching), so the sequencing is a coordination question. I have left
that file untouched and am raising it rather than guessing.
For the next pool someone wants to move off threads, four things, and only the
last two need judgement:
_child_realpath.pythat imports stdlib onlyand answers one op code. Run it by path, never
-m, orkiro_crew/__init__.pyexecutes and the cold-start budget is gone.
length, request id, op code, payload. Length prefixes, never a newline.
call_opfrom the thread that wants the answer. If that threadinstead waits on another thread's result, the win is gone and it will look like
the child is at fault. Check the work is genuinely syscall-shaped first: this
converts "blocks in the kernel" into "blocks in the kernel one process over",
so it does nothing for CPU-bound Python.
sure no fault path can return an empty answer that reads as a real one.
Refs #10255