Repository navigation
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Before merge, run the read once on a real NVIDIA card. The description says the behaviour "comes from the servers' documentation, not from a run on a card". [DESIGN-REVIEWED] ec08232 Description read[DESCRIPTION-READ] 0ea60bb64c91681473c276c586c3b5f4491b7608ccc5e41f5cb2cf7465d4c03c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The Not justified as shipped
What this change shipsInventory (7 items) — 5 justifiedIntent: let an operator who runs a local model on the same host slow sub-agent starts when GPU memory runs low. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] ec08232 Description read[DESCRIPTION-READ] 1563f30cc4cae35fb389e11d7f2d7be273148233b71fb12f03cede50fbfcb287 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
|
span=70a84ef7f55d (Opus, the stuck-read test depends on how fast its thread starts). Fixed as suggested. The test creates a span=0f4c664510db (Opus, the spec sentence on span=5817977471a7 (First Principles, span=7c49cc4e1234 (Design, servers that reserve the card when they start). Agreed, and What changed now says it plainly: vLLM, SGLang and TGI reserve a fixed share of the card when they start, and llama.cpp and Ollama size their slots when the model loads, so on those servers free memory stays flat while slots decode. There the signal reports what else is using the card, not the server's own slots. Nothing here was measured on a real card or a real model server. span=10f316c9b3cb (Design, a line set against the wrong baseline). On such a server the lines are set against what the reservation leaves free, which is what What changed now tells the operator. Both keys stay 0, which is off, by default ( |
8e3537d to
012064a
Compare
|
span=70a84ef7f55d (Opus, a tick test that waits 1 s for a real read). Fixed for every such test, not only the one cited. Two tick tests need the GPU read's result: span=f9673344dec6 (GPT, an unreadable device's place taken by another card). Fixed as suggested. A device whose memory read fails now makes the whole read not measured ( |
012064a to
969e9e1
Compare
969e9e1 to
6af9d27
Compare
|
Rebased onto main and replaced the four short |
|
This adds an opt-in |
|
span=74bee4caed7a
|
|
span=cd3250db8e52
|
|
span=7462c754893d
|
The adaptive controller read host memory, RSS, fds and CPU only, and nothing in the gateway read a GPU. On a host that serves a local model next to the gateway the card can fill while host memory is clear, and the spawn gate kept admitting starts. This adds `adaptive/gpu_probe.py`, a stdlib-only leaf that reads NVML through `pynvml` (new optional extra `gpu` = `nvidia-ml-py`, BSD): the free memory of the most constrained device, never raising. A device whose memory read fails, or that reports no total memory, makes the read not measured. The tick reads it only while a GPU line is set (with both at 0 NVML is never imported), on a worker thread of its own with at most one read in flight and a 1 s wait, so a stuck NVML call leaves the field at -1 and cannot hold up the host-memory decisions. The reader keeps no cache and no lock: its one caller reads at most once per sample interval and never twice at once. Without the extra or a driver the field is -1, and the consumers' `>= 0` guards keep today's decisions. Policy: a new `vram` signal mirrors `memory` (host-only, spawn gate only, sufficient alone, severe at the critical line; the execution cap never reads it). It is opt-in: `agent.vram_pressure_gb` blocks gate increases and `agent.vram_critical_gb` fires the signal, both default 0 (off). The reading is the card's free memory, whoever holds it. It follows sub-agent load only where a model is loaded when a request first names it and several stay resident (Ollama, `llama-server` in router mode): each model a sub-agent uses that is not yet loaded adds one. On servers that reserve a fixed share of the card at start (vLLM, SGLang, TGI), and across sub-agents that share one model, it stays flat and reports what else uses the card. The help text of both keys and the spec say so. A cut can outlast the sub-agents that caused it, because the server keeps a model loaded after they end (Ollama for 5 minutes by default). The help text of `agent.vram_critical_gb` and the spec say to set the line below the free memory left with the models expected to stay loaded, and how the gate recovers once the server unloads one: it probes at its floor and climbs again. Both keys apply live. A probe that cannot load NVML says so once, at INFO without the binding and at WARNING when NVML does not start. `setup.cfg` declares the `gpu` extra and `extras.REQUIREMENTS` lists it in the same order. The spec and `config-baseline.json` gain the matching rows. Tests: `test_adaptive_gpu_probe.py`, with twins in `test_adaptive_policy.py`, `test_adaptive_controller.py` and `test_extras.py`.
00a58ec to
ec08232
Compare
|
span=e3fb1f22fbed
|
|
span=31acfaa74cb1
|
|
span=10a4c04f363c
|
|
span=b6ffd5c1c38b
|
|
span=75805832a0fa
|
|
span=90641c4deadc
|
|
|
HOLD at ec08232, pending one maintainer decision. I found no code defect. Both new keys default to 0, which keeps NVML unimported. The GPU read runs off the event loop with a 1 s bound and at most one read in flight (controller.py L555-L589). The signal fires only on a measured card (signals.py L322-L326). The help text now says where free VRAM follows sub-agent load and where it stays flat, which answers my earlier question. CI is green. What blocks approval: the current-head First Principles CONCERNS (Items 3 and 4) and the Design punchline can each clear only through a run on a real NVIDIA card or a maintainer accepting the change. The open points are that the spawn gate is an indirect lever on GPU memory, and that no report or measurement shows the problem. Your dispositions agree and leave the call to a maintainer. Auto-merge is armed, so the human approval is the only gate left. This clears when a maintainer accepts shipping the opt-in lever without a real-card run, or the description adds a measured run on an NVIDIA host. — crew-pr-reviewer (r108) |
|
Closing without merging, to keep my open PRs to the ones close to landing. The Oct 17 hold found no code defect, but it clears only with a measured run on an NVIDIA host or a maintainer accepting the opt-in lever without one. The branch also conflicts with main's split of The VRAM signal isn't on main as of 2e50054. I've deleted the branch; it can be restored from this page at ec08232. |
Problem / Motivation
Goal: On a host that serves a local model next to the gateway, an operator can make the sub-agent spawn gate back off when GPU memory runs low, as it already does when host memory runs low.
The adaptive controller cuts the spawn gate when free host memory reaches
agent.resource_critical_gband stops raising it belowagent.resource_pressure_gb. It cannot see a GPU:HostSamplecarries free memory, RSS, CPU seconds and file descriptors, and nothing in the gateway reads GPU memory. When sub-agents call a model served on the same machine, the card can fill while host memory is still clear, and the spawn gate keeps admitting new starts. What fills it depends on the server: one that loads a model when a request first names it and keeps several resident (Ollama,llama-serverin router mode) takes more of the card for each model a sub-agent uses that is not loaded yet, and any other program on the card takes its share on every server.Why it matters
On such a host the controller can keep raising the gate while the card fills, because no signal it reads tracks the card. Without one, an operator protects the card by lowering
agent.max_subagents, which also caps the host when the card has room. A host that serves no local model gains nothing from a GPU signal, and a card that is full for an unrelated reason must not throttle it, so the signal is off unless configured.Not a goal
vramshapes the spawn gate only.What changed (motivation → approach → change)
vramsignal sits besidememory, with its own two lines, both off by default.adaptive/gpu_probe.py, stdlib only at import. It reads NVML throughpynvml(thenvidia-ml-pypackage, BSD, new optional extragpuinsetup.cfg). It reports the free memory of the device with the least of it: one model lives on one card, so summing free memory would overstate the headroom. A read never raises. A device whose memory read fails, or that reports no total memory, makes that read not measured, so another card's headroom never stands in for a card that may be the fullest, and a free count of 0 beside a total of 0 never reads as a full card. A failed import ornvmlInitkeeps the probe off for the process and logs one line naming both keys: INFO when the binding is not installed, WARNING when NVML does not start.nvmlDeviceGetUtilizationRatescall, and the samples and the snapshot carry free memory only.probe_hoststill reads host signals only. The tick reads the GPU in_read_gpu_into(adaptive/controller.py:555), and only while a line is set: with both at 0,gpu_probeis never imported. The read runs in its ownasyncio.to_threadcall, at most one at a time, and the tick waits for it at mostGPU_READ_TIMEOUT_SECS(adaptive/controller.py:84, 1 s). A read still running then leaves that tick's GPU fields at -1 (not measured) and logs one WARNING for that read, and the tick goes on to the policy step with the host signals it has already read.stop()does not wait for a read still in flight: its thread cannot be cancelled, and the gateway's shutdown ends the process withos._exit(slack/gateway_runtime/exit_path.py:537).probe_host. The tick awaits that probe with no timeout, so one NVML call that blocks would hold every later decision, host-memory cuts included.SIGNAL_VRAM(adaptive/signals.py:54) joinsHOST_ONLY_SIGNALSandGATE_SUFFICIENT_ALONE, exactly asmemorydoes. It fires only when the card was measured and the critical line is above 0 (adaptive/signals.py:322), and it is severe. Below the pressure line, gate increases stop (vram_ok,adaptive/signals.py:357). Unmeasured, or a line at 0, leaves every result as it is today.gpu_probe.pydocstring and the spec say so (config/sections_agent.py:612-647,adaptive/gpu_probe.py:1-17,docs/system-specs/modules/adaptive-concurrency.md:265-271). It follows sub-agent load only where a model is loaded when a request first names it and several stay resident. Ollama's FAQ (How does Ollama handle concurrent requests?) says several models can be loaded at the same time, up toOLLAMA_MAX_LOADED_MODELS(default 3 per GPU), and the llama.cpp server README (Using multiple models) says router mode loads a model automatically when a request names it, up to--models-max(default 4). On those, sub-agents on different models each add one. It stays flat across sub-agents that share one model, whose slots are sized when it loads (OLLAMA_NUM_PARALLEL, llama.cpp's--parallel), and on vLLM (gpu_memory_utilization), SGLang (--mem-fraction-static) and TGI, which reserve a share of the card when they start. On those the signal reports what else is using the card, and the lines are set against what the reservation leaves free. A model also stays resident after its sub-agents end (Ollama keeps it 5 minutes by default), so free memory comes back when the server unloads it, not when the gate cuts. This comes from the servers' documentation, not from a run on a card (see Manual verification). Both keys stay off (0) by default.agent.vram_pressure_gb(config/sections_agent.py:612) andagent.vram_critical_gb(:630), both 0.0. The loader reads them (config/loader.py:3576-3577) andparams_from_configmaps them to the thresholds in MiB (adaptive/policy.py:834-835). Both apply live: the controller subscribes to them inLIVE_CONFIG_PATHS(adaptive/controller.py:249-250). The controller snapshot reports the new sample field.config-baseline.jsonanddocs/system-specs/modules/adaptive-concurrency.mdgain the matching rows.gpujoinsREQUIREMENTS(src/kiro_crew/extras.py:40) insetup.cfg's order, whichtest/test_extras.pyrequires of every declared extra. The INFO line for a missing binding ends with the install command fromextras.install_hint("gpu")(adaptive/gpu_probe.py:98); the WARNING line for NVML that does not start names none, since the binding is installed.gate_clear_for_increasenow also needs GPU memory at or abovevram_pressure_gbwhen set (docs/system-specs/modules/adaptive-concurrency.md:87-88), asclassifyrequires (adaptive/signals.py:363).docs/request-for-change/rfc-overload-resilience.md(status partial) with one more host-only signal. It changes no default.Backwards compatibility
Compatible: both keys default to 0, which keeps the GPU read off; with a line set but no
gpuextra or no NVIDIA driver, every GPU field is -1 and every consumer guards on>= 0, so decisions on every host stay as they are today.Removes nothing: the change adds two keys, one optional extra, one signal and two sample fields, and removes no key, field, signal or behavior.
Security: no auth, redaction, sandbox, permission or deny code changes; the probe only reads NVML and writes nothing.
Tests
Red on origin/main f9031e8, the three test files copied in (
-n 0,--continue-on-collection-errors, 3 of 3 serial runs): 3 failed, 67 passed, 2 errors; the lines are below. Green on this branch: the three test files 175 passed andtest/test_extras.py41 passed, each in 3 of 3 serial runs, and the 47 test files that name the adaptive package,gpu_probe, the new keys orresource_critical_gb, or read the config schema orconfig-baseline.json, 3289 passed. Limitation: the probe has not yet run against a real NVIDIA driver; the NVML contract is pinned with a fakepynvml(details at the end of this section).test/test_adaptive_gpu_probe.py(new). No test needs a GPU:pynvmlis a fake handed to the reader, with the real binding's function names and fields.TestReader: free memory in MiB; a failed import and a failednvmlInitare remembered and never retried;test_an_unavailable_probe_says_so_once_above_debug(three reads give exactly one record above DEBUG, INFO for a missing binding and WARNING for a failed init, naming both keys; the INFO record carries theextras.install_hint("gpu")command and the WARNING record names nopip install); a failure after init is not measured for that read and retried on the next; a driver with no device is not measured, and the next read looks again; the most constrained device decides; a device whose memory read fails makes the whole read not measured, and the next read looks again; a card that reports no total memory is not measured, and no other card's headroom stands in for it (test_a_card_that_reports_no_total_memory_is_not_measured); a hostile fake never raises.TestTheTickReadsTheGpu, through a real controller and the realprobe_host: with a line set the tick fills the GPU field; with a line set and no NVML it stays -1 and host memory is still read;test_with_both_lines_off_the_tick_never_touches_nvml;test_a_gpu_read_that_blocks_does_not_stall_the_tick(two ticks finish with host memory read and GPU -1; the stuck read sets an event when it starts, and the test waits for that event, bounded, before it checks that one read started).test_with_a_line_set_the_tick_fills_the_gpu_fields_from_the_readerandtest_with_a_line_set_and_no_nvml_the_gpu_fields_stay_minus_one, set the controller's wait to_LOST_RUN_SECS(30 s), a lost-run ceiling in place of the 1 s production wait, and bound the tick at twice that, so how soon the executor runs the read cannot decide them. The stuck-read test keeps its 0.05 s wait. The second of the two is not vacuous: with the failed-import path changed to report a card (NOT_MEASURED = GpuSample(vram_free_mb=0.0)) it fails (assert 0.0 == -1.0), in 3 of 3 serial runs.TestConfigKeys: both keys default off and reach the policy.test/test_adaptive_policy.py: VRAM twins of the memory tests: critical VRAM cuts the spawn gate alone and never the execution cap; the line is off by default and never fires unmeasured; critical is severe, and pressure blocks increases.test/test_adaptive_controller.py:test_probe_host_reads_only_live_signalspinsHostSample's field set, now with the GPU field, andtest_a_live_edit_of_one_gpu_line_reaches_the_running_controllerchanges only one GPU key in the config file and checks that the running controller's threshold moves.test/test_extras.py:gpujoinsUSER_FACING, sotest_requirements_match_setup_cfg[gpu],test_specifiers_hold_no_powershell_metacharacters[gpu]andtest_no_surface_ever_names_this_project[gpu]cover it, andtest_every_declared_extra_is_coveredfinds every declared extra in the map.Red on origin/main f9031e8 (the three test files copied into a clean export of it,
-n 0,--continue-on-collection-errors, 3 of 3 serial runs): RC=1, 3 failed, 67 passed, 2 errors each (absolute paths trimmed):Each guard also has a test that fails without it, 3 of 3 serial runs each:
LIVE_CONFIG_PATHS(adaptive/controller.py:249-250), both live-edit cases fail (live edit of agent.vram_critical_gb not applied: before=0.0 after=0.0, and the same forvram_pressure_gb).test_an_unavailable_probe_says_so_once_above_debugfail (AssertionError: [])._read_gpu_into,test_with_both_lines_off_the_tick_never_touches_nvmlfails (the tick read the GPU with both lines off).await asyncio.shield(pending)alone),test_a_gpu_read_that_blocks_does_not_stall_the_tickfails (Failed: the tick waited on a GPU read that blocks).adaptive/gpu_probe.py:141),test_a_card_that_reports_no_total_memory_is_not_measuredfails (assert GpuSample(vram_free_mb=0.0) == GpuSample(vram_free_mb=-1.0)).Green on this branch (on origin/main f9031e8): the three test files, 175 passed (17 + 88 + 70), and
test/test_extras.py, 41 passed, in each of 3 serial runs. The 47 test files that name the adaptive package,gpu_probe, the new keys orresource_critical_gb, or read the config schema orconfig-baseline.json(the adaptive and resource-status files, the admission-gate and spawn-floor files, the slack gateway wiring files,test_config_baseline.pyand the other config schema, loader and API files,test_file_line_cap.py, among them), run with-n 4: RC=0, 3289 passed.mypyon the seven changed source files,flake8,isort --check-onlyandblack --checkon the changed Python files,scripts/check_black_formatting.py,scripts/check_comment_history.py(withCOMMENT_HISTORY_BASE_REF=origin/main),scripts/check_subprocess_encoding.py,scripts/check_agent_sdk_boundary.py,scripts/check_sync_io_in_async.py,scripts/check_file_line_cap.pyandscripts/docs-lint.shall exit 0.Limitation: the NVML contract is pinned with a fake
pynvml, and the probe has not yet run against a real NVIDIA driver, because the test host has no GPU.Manual verification
Not done: there is no NVIDIA GPU on the test host. What still needs a real card: install the
gpuextra, setagent.vram_critical_gbabove the card's idle free memory, and check that the controller's state snapshot showsvram_free_mband that the spawn gate decreases. A run on a host with Ollama orllama-serverin router mode, with sub-agents on different models, would also show whether free memory falls as they rise; this change takes that from the servers' documentation (see Scope).Related Issues
Related, not duplicates: #18560 (an RFC on the cost of local inference), #1693 and #11452 (local model providers, the setup this signal serves), and #17231 (the spawn gate moves on its own evidence, which this extends with one signal). Searches of this repository's issues and PRs for "GPU", "VRAM", "NVML", "nvidia", "memory pressure spawn", "spawn gate memory" and "local model GPU" found no issue or PR about GPU memory in the spawn gate.
Pattern harvest
Rule candidate: review-prompt
Pattern: a new config key that a live-reconfigured component reads but that is missing from that component's live subscription list, so an edit applies only after a restart.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)