Skip to content

feat(adaptive): back off the spawn gate when GPU memory runs low - #19100

Closed
javenciu wants to merge 1 commit into
kirodotdev:mainfrom
javenciu:feat/adaptive-vram-signal
Closed

javenciu wants to merge 1 commit into
kirodotdev:mainfrom
javenciu:feat/adaptive-vram-signal

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: On a host that serves a local model next to the gateway, an operator can make the sub-agent spawn gate back off when GPU memory runs low, as it already does when host memory runs low.

The adaptive controller cuts the spawn gate when free host memory reaches agent.resource_critical_gb and stops raising it below agent.resource_pressure_gb. It cannot see a GPU: HostSample carries free memory, RSS, CPU seconds and file descriptors, and nothing in the gateway reads GPU memory. When sub-agents call a model served on the same machine, the card can fill while host memory is still clear, and the spawn gate keeps admitting new starts. What fills it depends on the server: one that loads a model when a request first names it and keeps several resident (Ollama, llama-server in router mode) takes more of the card for each model a sub-agent uses that is not loaded yet, and any other program on the card takes its share on every server.

Why it matters

On such a host the controller can keep raising the gate while the card fills, because no signal it reads tracks the card. Without one, an operator protects the card by lowering agent.max_subagents, which also caps the host when the card has room. A host that serves no local model gains nothing from a GPU signal, and a card that is full for an unrelated reason must not throttle it, so the signal is off unless configured.

Not a goal

  • A GPU signal for the execution cap. Like host memory, vram shapes the spawn gate only.
  • GPUs other than NVIDIA. Only NVML is read; on any other host every GPU field stays not measured.
  • Turning the lines on by default, or deriving them from the card's size. Both keys default to 0, which is off.
  • Per-process GPU accounting. The sample is the card's free memory, whoever holds it.
  • Installing the NVML binding by default. It is an optional extra.
  • Starting NVML again after it failed to start in this process. A restart does that, and the probe says so.

What changed (motivation → approach → change)

  • Approach: treat the card's free memory like free host memory. A new vram signal sits beside memory, with its own two lines, both off by default.
  • Probe: new adaptive/gpu_probe.py, stdlib only at import. It reads NVML through pynvml (the nvidia-ml-py package, BSD, new optional extra gpu in setup.cfg). It reports the free memory of the device with the least of it: one model lives on one card, so summing free memory would overstate the headroom. A read never raises. A device whose memory read fails, or that reports no total memory, makes that read not measured, so another card's headroom never stands in for a card that may be the fullest, and a free count of 0 beside a total of 0 never reads as a full card. A failed import or nvmlInit keeps the probe off for the process and logs one line naming both keys: INFO when the binding is not installed, WARNING when NVML does not start.
  • Rejected: reading GPU utilization. No policy decision would use it, so the probe makes no nvmlDeviceGetUtilizationRates call, and the samples and the snapshot carry free memory only.
  • Controller: probe_host still reads host signals only. The tick reads the GPU in _read_gpu_into (adaptive/controller.py:555), and only while a line is set: with both at 0, gpu_probe is never imported. The read runs in its own asyncio.to_thread call, at most one at a time, and the tick waits for it at most GPU_READ_TIMEOUT_SECS (adaptive/controller.py:84, 1 s). A read still running then leaves that tick's GPU fields at -1 (not measured) and logs one WARNING for that read, and the tick goes on to the policy step with the host signals it has already read. stop() does not wait for a read still in flight: its thread cannot be cancelled, and the gateway's shutdown ends the process with os._exit (slack/gateway_runtime/exit_path.py:537).
  • Rejected: reading the GPU inside probe_host. The tick awaits that probe with no timeout, so one NVML call that blocks would hold every later decision, host-memory cuts included.
  • Rejected: starting a new read on each tick while one is stuck. A thread cannot be cancelled, so each tick would park one more worker of the shared executor in the same blocked NVML call.
  • Policy: SIGNAL_VRAM (adaptive/signals.py:54) joins HOST_ONLY_SIGNALS and GATE_SUFFICIENT_ALONE, exactly as memory does. It fires only when the card was measured and the critical line is above 0 (adaptive/signals.py:322), and it is severe. Below the pressure line, gate increases stop (vram_ok, adaptive/signals.py:357). Unmeasured, or a line at 0, leaves every result as it is today.
  • Scope: the reading is the card's free memory, whoever holds it, and the help text of both keys, the gpu_probe.py docstring and the spec say so (config/sections_agent.py:612-647, adaptive/gpu_probe.py:1-17, docs/system-specs/modules/adaptive-concurrency.md:265-271). It follows sub-agent load only where a model is loaded when a request first names it and several stay resident. Ollama's FAQ (How does Ollama handle concurrent requests?) says several models can be loaded at the same time, up to OLLAMA_MAX_LOADED_MODELS (default 3 per GPU), and the llama.cpp server README (Using multiple models) says router mode loads a model automatically when a request names it, up to --models-max (default 4). On those, sub-agents on different models each add one. It stays flat across sub-agents that share one model, whose slots are sized when it loads (OLLAMA_NUM_PARALLEL, llama.cpp's --parallel), and on vLLM (gpu_memory_utilization), SGLang (--mem-fraction-static) and TGI, which reserve a share of the card when they start. On those the signal reports what else is using the card, and the lines are set against what the reservation leaves free. A model also stays resident after its sub-agents end (Ollama keeps it 5 minutes by default), so free memory comes back when the server unloads it, not when the gate cuts. This comes from the servers' documentation, not from a run on a card (see Manual verification). Both keys stay off (0) by default.
  • Config: agent.vram_pressure_gb (config/sections_agent.py:612) and agent.vram_critical_gb (:630), both 0.0. The loader reads them (config/loader.py:3576-3577) and params_from_config maps them to the thresholds in MiB (adaptive/policy.py:834-835). Both apply live: the controller subscribes to them in LIVE_CONFIG_PATHS (adaptive/controller.py:249-250). The controller snapshot reports the new sample field. config-baseline.json and docs/system-specs/modules/adaptive-concurrency.md gain the matching rows.
  • Extras: gpu joins REQUIREMENTS (src/kiro_crew/extras.py:40) in setup.cfg's order, which test/test_extras.py requires of every declared extra. The INFO line for a missing binding ends with the install command from extras.install_hint("gpu") (adaptive/gpu_probe.py:98); the WARNING line for NVML that does not start names none, since the binding is installed.
  • Spec: gate_clear_for_increase now also needs GPU memory at or above vram_pressure_gb when set (docs/system-specs/modules/adaptive-concurrency.md:87-88), as classify requires (adaptive/signals.py:363).
  • Context: this extends the controller of docs/request-for-change/rfc-overload-resilience.md (status partial) with one more host-only signal. It changes no default.

Backwards compatibility

Compatible: both keys default to 0, which keeps the GPU read off; with a line set but no gpu extra or no NVIDIA driver, every GPU field is -1 and every consumer guards on >= 0, so decisions on every host stay as they are today.
Removes nothing: the change adds two keys, one optional extra, one signal and two sample fields, and removes no key, field, signal or behavior.
Security: no auth, redaction, sandbox, permission or deny code changes; the probe only reads NVML and writes nothing.

Tests

Red on origin/main f9031e8, the three test files copied in (-n 0, --continue-on-collection-errors, 3 of 3 serial runs): 3 failed, 67 passed, 2 errors; the lines are below. Green on this branch: the three test files 175 passed and test/test_extras.py 41 passed, each in 3 of 3 serial runs, and the 47 test files that name the adaptive package, gpu_probe, the new keys or resource_critical_gb, or read the config schema or config-baseline.json, 3289 passed. Limitation: the probe has not yet run against a real NVIDIA driver; the NVML contract is pinned with a fake pynvml (details at the end of this section).

test/test_adaptive_gpu_probe.py (new). No test needs a GPU: pynvml is a fake handed to the reader, with the real binding's function names and fields.

  • TestReader: free memory in MiB; a failed import and a failed nvmlInit are remembered and never retried; test_an_unavailable_probe_says_so_once_above_debug (three reads give exactly one record above DEBUG, INFO for a missing binding and WARNING for a failed init, naming both keys; the INFO record carries the extras.install_hint("gpu") command and the WARNING record names no pip install); a failure after init is not measured for that read and retried on the next; a driver with no device is not measured, and the next read looks again; the most constrained device decides; a device whose memory read fails makes the whole read not measured, and the next read looks again; a card that reports no total memory is not measured, and no other card's headroom stands in for it (test_a_card_that_reports_no_total_memory_is_not_measured); a hostile fake never raises.
  • TestTheTickReadsTheGpu, through a real controller and the real probe_host: with a line set the tick fills the GPU field; with a line set and no NVML it stays -1 and host memory is still read; test_with_both_lines_off_the_tick_never_touches_nvml; test_a_gpu_read_that_blocks_does_not_stall_the_tick (two ticks finish with host memory read and GPU -1; the stuck read sets an event when it starts, and the test waits for that event, bounded, before it checks that one read started).
  • The tick tests that need the read's result, test_with_a_line_set_the_tick_fills_the_gpu_fields_from_the_reader and test_with_a_line_set_and_no_nvml_the_gpu_fields_stay_minus_one, set the controller's wait to _LOST_RUN_SECS (30 s), a lost-run ceiling in place of the 1 s production wait, and bound the tick at twice that, so how soon the executor runs the read cannot decide them. The stuck-read test keeps its 0.05 s wait. The second of the two is not vacuous: with the failed-import path changed to report a card (NOT_MEASURED = GpuSample(vram_free_mb=0.0)) it fails (assert 0.0 == -1.0), in 3 of 3 serial runs.
  • TestConfigKeys: both keys default off and reach the policy.

test/test_adaptive_policy.py: VRAM twins of the memory tests: critical VRAM cuts the spawn gate alone and never the execution cap; the line is off by default and never fires unmeasured; critical is severe, and pressure blocks increases.
test/test_adaptive_controller.py: test_probe_host_reads_only_live_signals pins HostSample's field set, now with the GPU field, and test_a_live_edit_of_one_gpu_line_reaches_the_running_controller changes only one GPU key in the config file and checks that the running controller's threshold moves.
test/test_extras.py: gpu joins USER_FACING, so test_requirements_match_setup_cfg[gpu], test_specifiers_hold_no_powershell_metacharacters[gpu] and test_no_surface_ever_names_this_project[gpu] cover it, and test_every_declared_extra_is_covered finds every declared extra in the map.

Red on origin/main f9031e8 (the three test files copied into a clean export of it, -n 0, --continue-on-collection-errors, 3 of 3 serial runs): RC=1, 3 failed, 67 passed, 2 errors each (absolute paths trimmed):

E   ImportError: cannot import name 'gpu_probe' from 'kiro_crew.adaptive'
E   ImportError: cannot import name 'SIGNAL_VRAM' from 'kiro_crew.adaptive.signals'
E       AssertionError: assert {'cpu_clock',...mb', 'rss_mb'} == {'cpu_clock',...'rss_mb', ...}
E         Extra items in the right set:
E         'vram_free_mb'
E           AttributeError: 'Thresholds' object has no attribute 'vram_critical_mb'. Did you mean: 'mem_critical_mb'?
E           AttributeError: 'Thresholds' object has no attribute 'vram_pressure_mb'. Did you mean: 'mem_pressure_mb'?
ERROR test/test_adaptive_gpu_probe.py
ERROR test/test_adaptive_policy.py
FAILED test/test_adaptive_controller.py::TestTick::test_probe_host_reads_only_live_signals
FAILED test/test_adaptive_controller.py::TestVisibilityAndConfig::test_a_live_edit_of_one_gpu_line_reaches_the_running_controller[vram_critical_gb-vram_critical_mb]
FAILED test/test_adaptive_controller.py::TestVisibilityAndConfig::test_a_live_edit_of_one_gpu_line_reaches_the_running_controller[vram_pressure_gb-vram_pressure_mb]

Each guard also has a test that fails without it, 3 of 3 serial runs each:

  • With the two keys missing from LIVE_CONFIG_PATHS (adaptive/controller.py:249-250), both live-edit cases fail (live edit of agent.vram_critical_gb not applied: before=0.0 after=0.0, and the same for vram_pressure_gb).
  • With the unavailable-probe record logged at DEBUG only, both cases of test_an_unavailable_probe_says_so_once_above_debug fail (AssertionError: []).
  • With the both-lines-off return removed from _read_gpu_into, test_with_both_lines_off_the_tick_never_touches_nvml fails (the tick read the GPU with both lines off).
  • With the 1 s bound removed (await asyncio.shield(pending) alone), test_a_gpu_read_that_blocks_does_not_stall_the_tick fails (Failed: the tick waited on a GPU read that blocks).
  • With the reader's zero-total check removed (adaptive/gpu_probe.py:141), test_a_card_that_reports_no_total_memory_is_not_measured fails (assert GpuSample(vram_free_mb=0.0) == GpuSample(vram_free_mb=-1.0)).

Green on this branch (on origin/main f9031e8): the three test files, 175 passed (17 + 88 + 70), and test/test_extras.py, 41 passed, in each of 3 serial runs. The 47 test files that name the adaptive package, gpu_probe, the new keys or resource_critical_gb, or read the config schema or config-baseline.json (the adaptive and resource-status files, the admission-gate and spawn-floor files, the slack gateway wiring files, test_config_baseline.py and the other config schema, loader and API files, test_file_line_cap.py, among them), run with -n 4: RC=0, 3289 passed. mypy on the seven changed source files, flake8, isort --check-only and black --check on the changed Python files, scripts/check_black_formatting.py, scripts/check_comment_history.py (with COMMENT_HISTORY_BASE_REF=origin/main), scripts/check_subprocess_encoding.py, scripts/check_agent_sdk_boundary.py, scripts/check_sync_io_in_async.py, scripts/check_file_line_cap.py and scripts/docs-lint.sh all exit 0.

Limitation: the NVML contract is pinned with a fake pynvml, and the probe has not yet run against a real NVIDIA driver, because the test host has no GPU.

Manual verification

Not done: there is no NVIDIA GPU on the test host. What still needs a real card: install the gpu extra, set agent.vram_critical_gb above the card's idle free memory, and check that the controller's state snapshot shows vram_free_mb and that the spawn gate decreases. A run on a host with Ollama or llama-server in router mode, with sub-agents on different models, would also show whether free memory falls as they rise; this change takes that from the servers' documentation (see Scope).

Related Issues

Related, not duplicates: #18560 (an RFC on the cost of local inference), #1693 and #11452 (local model providers, the setup this signal serves), and #17231 (the spawn gate moves on its own evidence, which this extends with one signal). Searches of this repository's issues and PRs for "GPU", "VRAM", "NVML", "nvidia", "memory pressure spawn", "spawn gate memory" and "local model GPU" found no issue or PR about GPU memory in the spawn gate.

Pattern harvest

Rule candidate: review-prompt
Pattern: a new config key that a live-reconfigured component reads but that is missing from that component's live subscription list, so an edit applies only after a restart.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 13:00
@javenciu
javenciu requested a review from dodgeblaster October 11, 2026 13:00
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision ec08232f15980d6c98d538ef889b69305c840aac touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of ec08232f15980d6c98d538ef889b69305c840aac via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Before merge, run the read once on a real NVIDIA card. The description says the behaviour "comes from the servers' documentation, not from a run on a card".

[DESIGN-REVIEWED] ec08232

Description read

[DESCRIPTION-READ] 0ea60bb64c91681473c276c586c3b5f4491b7608ccc5e41f5cb2cf7465d4c03c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of ec08232f15980d6c98d538ef889b69305c840aac via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The vram signal cuts the MCP spawn gate, but MCP server forks are not what fills the card. The requirement is also only symmetry with host memory, with no reported operator behind it.

Not justified as shipped

  • Item 3 — symptom-level: the spec says "The gate admits forks inside the MCP gateway daemon" (adaptive-concurrency.md:31). The card fills from model loads that sub-agents trigger. That makes the gate an indirect lever on the card, and the Goal itself names "the sub-agent spawn gate". Host memory for sub-agent starts is owned by the admission floor (subagent_manager/admission/gate.py, spawn_min_memory_gb), not by this gate. On a card full for some other reason, the gate pauses at its floor, and the spec records that new chats then miss the 90 s MCP readiness deadline.
    Clears when: the description shows that cutting the MCP spawn gate lowers GPU use on a real host, or a human accepts the indirect lever.
  • Item 4 — inherited: the Goal is "as it already does when host memory runs low". No issue is linked and no affected operator is named, and the description says "This comes from the servers' documentation, not from a run on a card."
    Clears when: a linked report or a measured run on an NVIDIA host shows the card filling while the gate keeps admitting starts.

What this change ships

Inventory (7 items) — 5 justified

Intent: let an operator who runs a local model on the same host slow sub-agent starts when GPU memory runs low. ADDITION.

  1. Two opt-in keys, agent.vram_pressure_gb and agent.vram_critical_gb, both default 0 (off) — justified
  2. New optional gpu extra (nvidia-ml-py) — justified
  3. Free GPU memory at or below the critical line cuts or pauses the MCP spawn gate — symptom-level, the gate does not govern model loads
  4. Free GPU memory below the pressure line stops the gate rising — inherited, symmetry with host memory, no report
  5. Controller snapshot shows vram_free_mb — justified
  6. A single log line when NVML is missing or fails to start — justified
  7. The tick reads the GPU on its own thread, waits at most 1 s, and runs one read at a time — justified

[FIRST-PRINCIPLES-REVIEWED] ec08232

Description read

[DESCRIPTION-READ] 1563f30cc4cae35fb389e11d7f2d7be273148233b71fb12f03cede50fbfcb287

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed ec08232f15980d6c98d538ef889b69305c840aac via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ec08232

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed ec08232f15980d6c98d538ef889b69305c840aac via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ec08232

@javenciu

Copy link
Copy Markdown
Contributor Author

span=70a84ef7f55d (Opus, the stuck-read test depends on how fast its thread starts). Fixed as suggested. The test creates a threading.Event (test/test_adaptive_gpu_probe.py:297), stuck_read sets it on entry (test/test_adaptive_gpu_probe.py:302), and after the two ticks the test waits for that event, bounded at 10 s, before it counts the reads (test/test_adaptive_gpu_probe.py:321-322). The stuck read is still released in finally (test/test_adaptive_gpu_probe.py:324). To check that the timing dependence is gone, a scratch pytest plugin held the read's worker thread until an event-loop callback scheduled by the second tick had run, so the read started only after both 0.05 s waits had closed. Under it the previous test failed 5 of 5 (0 reads started while one was stuck) and this one passed 5 of 5. The three adaptive test files passed in 50 of 50 runs under -n 4.

span=0f4c664510db (Opus, the spec sentence on gate_clear_for_increase). Fixed: it now reads "no gate signal with memory at or above the pressure line and GPU memory at or above vram_pressure_gb when set" (docs/system-specs/modules/adaptive-concurrency.md:87-88). That is what classify does: gate_clear needs both mem_ok and vram_ok (src/kiro_crew/adaptive/signals.py:364), and vram_ok holds when the card is not measured, the line is 0, or free GPU memory is at or above the line (src/kiro_crew/adaptive/signals.py:358-362).

span=5817977471a7 (First Principles, gpu_util_pct has no consumer). Removed, as the subtraction says. gpu_util_pct is gone from GpuSample (src/kiro_crew/adaptive/gpu_probe.py:55-63), HostSample (src/kiro_crew/adaptive/controller.py:167-168), Sample (src/kiro_crew/adaptive/signals.py:185-186) and the snapshot state() builds (src/kiro_crew/adaptive/controller.py:930-931), and the per-device loop in _read_uncached (src/kiro_crew/adaptive/gpu_probe.py:141-156) no longer calls NVML's utilization read. git grep finds no gpu_util_pct, util_max or nvmlDeviceGetUtilizationRates in src, test or docs. The free memory decides in classify (src/kiro_crew/adaptive/signals.py:323), and the total decides GpuSample.measured (src/kiro_crew/adaptive/gpu_probe.py:63).

span=7c49cc4e1234 (Design, servers that reserve the card when they start). Agreed, and What changed now says it plainly: vLLM, SGLang and TGI reserve a fixed share of the card when they start, and llama.cpp and Ollama size their slots when the model loads, so on those servers free memory stays flat while slots decode. There the signal reports what else is using the card, not the server's own slots. Nothing here was measured on a real card or a real model server.

span=10f316c9b3cb (Design, a line set against the wrong baseline). On such a server the lines are set against what the reservation leaves free, which is what What changed now tells the operator. Both keys stay 0, which is off, by default (src/kiro_crew/config/sections.py:1485, src/kiro_crew/config/sections.py:1497), so no host changes behaviour unless an operator sets a line.

@javenciu
javenciu force-pushed the feat/adaptive-vram-signal branch from 8e3537d to 012064a Compare October 11, 2026 17:23
@javenciu

Copy link
Copy Markdown
Contributor Author

span=70a84ef7f55d (Opus, a tick test that waits 1 s for a real read). Fixed for every such test, not only the one cited. Two tick tests need the GPU read's result: test_with_a_line_set_the_tick_fills_the_gpu_fields_from_the_reader (test/test_adaptive_gpu_probe.py:271) and test_with_a_line_set_and_no_nvml_the_gpu_fields_stay_minus_one (test/test_adaptive_gpu_probe.py:283). Both now set GPU_READ_TIMEOUT_SECS to _LOST_RUN_SECS (test/test_adaptive_gpu_probe.py:33, 30 s, a quarter of the suite's 120 s timeout) and bound their tick at twice that (test/test_adaptive_gpu_probe.py:279, test/test_adaptive_gpu_probe.py:291). The second could not fail on a slow worker, but there it passed because the controller gave up before its read ran, so it checked nothing. No other test in the three adaptive files waits on a read that is meant to finish: the stuck-read test keeps its 0.05 s wait (test/test_adaptive_gpu_probe.py:335), and every other tick injects its host probe or has both GPU lines off. A scratch pytest plugin held the read's worker thread until a loop timer 1 s past the production wait had fired. Under it the previous version of the first test failed (assert (-1.0, -1.0) == (6144.0, 32768.0)), and the previous second test still passed with the reader's failed-import path changed to report a card. These versions pass under the plugin, and the second one fails on that change (assert (0.0, 1.0) == (-1.0, -1.0)). The three files passed in 20 of 20 runs under -n 4.

span=f9673344dec6 (GPT, an unreadable device's place taken by another card). Fixed as suggested. A device whose memory read fails now makes the whole read not measured (src/kiro_crew/adaptive/gpu_probe.py:155), as the module's rule for a failure after init says (src/kiro_crew/adaptive/gpu_probe.py:20-22), and the most-constrained rule says so too (src/kiro_crew/adaptive/gpu_probe.py:32-33). test_a_device_that_fails_its_read_makes_the_read_not_measured (test/test_adaptive_gpu_probe.py:180) fails on the previous head, where the read reported the roomy card's 20480 MiB, and checks that the next read looks again.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
@javenciu
javenciu force-pushed the feat/adaptive-vram-signal branch from 012064a to 969e9e1 Compare October 12, 2026 13:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@javenciu
javenciu force-pushed the feat/adaptive-vram-signal branch from 969e9e1 to 6af9d27 Compare October 12, 2026 20:30
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased onto main and replaced the four short wait_for bounds test/test_wait_discipline_ratchet.py flagged: in test/test_adaptive_gpu_probe.py the tick waits at lines 313 (10 s) and 343 (5 s) now take 2 * _LOST_RUN_SECS (60 s), the drain of the released GPU read at line 358 (10 s) awaits the task with kiro_crew.testing.wait.await_until, and in test/test_adaptive_controller.py the live-edit config refresh at lines 1204-1206 (10 s) does the same.
The stuck read now blocks for 3 * _LOST_RUN_SECS (line 333) so a tick that waits on it still fails at its own bound with the same message, and the wait for that read to start (line 351, 10 s) takes _LOST_RUN_SECS; the ratchet file passes and the three adaptive test files are green in 3 of 3 runs (175 passed).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@javenciu
javenciu enabled auto-merge (squash) October 12, 2026 23:19
@iamwhatever

Copy link
Copy Markdown
Contributor

This adds an opt-in vram signal: a new NVML probe (adaptive/gpu_probe.py) reports the most constrained card's free memory, and agent.vram_critical_gb / agent.vram_pressure_gb (both 0 = off) cut or freeze the spawn gate the way host memory does. I read every production file: the read runs off the loop with a 1 s bound and at most one read in flight, a failing device makes the read not measured, both lines at 0 never import NVML, and every consumer guards on >= 0, so with defaults nothing changes on any host and I found no defect in the code. The open question is the premise, and the Design and First Principles CONCERNS for this head (6af9d27) are still unanswered: the PR's own text says vLLM, SGLang, TGI, llama.cpp and Ollama hold free VRAM flat while slots decode, so on every server it names the signal does not follow how many sub-agents the gate admits, and cutting the gate never frees the card. Yet the help text in config/sections.py:1485-1495 still says the lines stop "concurrently decoding sub-agents" from filling the card, and the gpu_probe.py module docstring says each sub-agent "holds a KV-cache slot on the card". Question for the author/maintainer: is there a named local-serving setup where free GPU memory actually falls as concurrent sub-agents rise? If yes, name it in the help text; if not, should this ship at all, or with help text that says it only reacts to other users of the card? Smaller point: First Principles notes vram_total_mb has no policy reader (only the snapshot and measured), which is fine to keep but needs an answer. — crew-pr-reviewer (deep re-check)

@javenciu

Copy link
Copy Markdown
Contributor Author

span=74bee4caed7a

  • The premise is unverified on a real card. kept: the test host has no NVIDIA card, so no run shows free memory falling as sub-agents rise, and whether to ship without one is the maintainer's call. The item's other way to clear is taken: the help text of both keys now names the setups where the signal tracks load, Ollama with several models resident and llama-server in router mode (each model a sub-agent uses that is not yet loaded adds one), and the ones where it stays flat, vLLM, SGLang and TGI, which reserve a share of the card at start, and sub-agents that share one model, and it says the line reacts to a nearly full card whatever fills it (src/kiro_crew/config/sections_agent.py:612-641); the gpu_probe.py docstring (src/kiro_crew/adaptive/gpu_probe.py:9-17) and the spec (docs/system-specs/modules/adaptive-concurrency.md:265-271) say the same. The two setups come from the servers' documentation, not from a run: Ollama's FAQ lists OLLAMA_MAX_LOADED_MODELS (default 3 per GPU) and the llama.cpp server README lists --models-max (default 4) with automatic loading on request.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=cd3250db8e52

  • Item 1, inherited: no measurement shows a card filling as sub-agents scale. kept: this repository has no issue or measurement of it (the body's searches of issues and PRs found none) and the test host has no card, so none can be added here; whether a report from a real card is needed before this ships is the maintainer's call. What changed is the item's complaint that the description names no server where the signal tracks sub-agent load: it now names two from the servers' documentation. Ollama loads several models at once, up to OLLAMA_MAX_LOADED_MODELS, and llama.cpp's router mode loads a model when a request names it, up to --models-max, so sub-agents on different models each add one (links in the body's Scope bullet). The same text says where the signal stays flat, on vLLM, SGLang, TGI and across sub-agents that share one model, and the help text of both keys says so too (src/kiro_crew/config/sections_agent.py:612-641).

@javenciu

Copy link
Copy Markdown
Contributor Author

span=7462c754893d

  • Item 5, zero consumers: the policy never reads vram_total_mb. fixed: the field is deleted from GpuSample (src/kiro_crew/adaptive/gpu_probe.py:65-72), HostSample (src/kiro_crew/adaptive/controller.py:163-167), Sample (src/kiro_crew/adaptive/signals.py:182-185) and the snapshot (src/kiro_crew/adaptive/controller.py:927), and git grep -n vram_total now matches nothing. measured checks free memory alone (src/kiro_crew/adaptive/gpu_probe.py:71-72). The old total > 0 half of that guard stopped one case, a card that reports free 0 beside total 0 and would read as full, so the equivalent check now sits in the reader where the total is still known (src/kiro_crew/adaptive/gpu_probe.py:166-169); test_a_card_that_reports_no_total_memory_is_not_measured (test/test_adaptive_gpu_probe.py:180) pins it and fails without it (assert GpuSample(vram_free_mb=0.0) == GpuSample(vram_free_mb=-1.0), 3 of 3 serial runs).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 16, 2026
The adaptive controller read host memory, RSS, fds and CPU only, and
nothing in the gateway read a GPU. On a host that serves a local model
next to the gateway the card can fill while host memory is clear, and the
spawn gate kept admitting starts.

This adds `adaptive/gpu_probe.py`, a stdlib-only leaf that reads NVML
through `pynvml` (new optional extra `gpu` = `nvidia-ml-py`, BSD): the
free memory of the most constrained device, never raising. A device whose
memory read fails, or that reports no total memory, makes the read not
measured. The tick reads it only while a GPU line is set (with both at 0
NVML is never imported), on a worker thread of its own with at most one
read in flight and a 1 s wait, so a stuck NVML call leaves the field at
-1 and cannot hold up the host-memory decisions. The reader keeps no
cache and no lock: its one caller reads at most once per sample interval
and never twice at once. Without the extra or a driver the field is -1,
and the consumers' `>= 0` guards keep today's decisions.

Policy: a new `vram` signal mirrors `memory` (host-only, spawn gate only,
sufficient alone, severe at the critical line; the execution cap never
reads it). It is opt-in: `agent.vram_pressure_gb` blocks gate increases
and `agent.vram_critical_gb` fires the signal, both default 0 (off).

The reading is the card's free memory, whoever holds it. It follows
sub-agent load only where a model is loaded when a request first names it
and several stay resident (Ollama, `llama-server` in router mode): each
model a sub-agent uses that is not yet loaded adds one. On servers that
reserve a fixed share of the card at start (vLLM, SGLang, TGI), and across
sub-agents that share one model, it stays flat and reports what else uses
the card. The help text of both keys and the spec say so.

A cut can outlast the sub-agents that caused it, because the server keeps
a model loaded after they end (Ollama for 5 minutes by default). The help
text of `agent.vram_critical_gb` and the spec say to set the line below
the free memory left with the models expected to stay loaded, and how the
gate recovers once the server unloads one: it probes at its floor and
climbs again.

Both keys apply live. A probe that cannot load NVML says so once, at INFO
without the binding and at WARNING when NVML does not start.
`setup.cfg` declares the `gpu` extra and `extras.REQUIREMENTS` lists it in
the same order.

The spec and `config-baseline.json` gain the matching rows. Tests:
`test_adaptive_gpu_probe.py`, with twins in `test_adaptive_policy.py`,
`test_adaptive_controller.py` and `test_extras.py`.
@javenciu
javenciu force-pushed the feat/adaptive-vram-signal branch from 00a58ec to ec08232 Compare October 16, 2026 15:26
@javenciu

Copy link
Copy Markdown
Contributor Author

span=e3fb1f22fbed

  • The NVML branch only runs on hardware CI doesn't have. kept: this change has no NVIDIA card or Ollama host to run on, so no tick on a real card can be shown here, and whether to require one before this ships is the maintainer's call. The description says so in three places: the Scope bullet, which sits before the 8000-byte mark the review read to ("This comes from the servers' documentation, not from a run on a card"), the Tests section ("Limitation: the probe has not yet run against a real NVIDIA driver"), and Manual verification ("Not done: there is no NVIDIA GPU on the test host"). Manual verification lists what a real card still has to show: install the gpu extra, set agent.vram_critical_gb above the card's idle free memory, and check that the controller snapshot shows vram_free_mb and that the spawn gate decreases.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=31acfaa74cb1

  • A cut caused by the GPU can outlast the sub-agents that caused it. fixed: the help text of agent.vram_critical_gb now says that a server keeps a model loaded after the sub-agents that used it end (Ollama for 5 minutes by default, per its FAQ), that the gate stays at its floor while free memory is at or below the line, to set the line below the free memory left with the models you expect to stay loaded, and that once the server unloads one and free memory is above the line the gate probes and climbs again (src/kiro_crew/config/sections_agent.py:640-646). The spec says the same and ties it to the decision rules: after severe_samples (2) consecutive samples at the line the gate is paused at its floor (the Pause row), recovery goes through the Probe, Resume and Idle recovery rows, and increases wait for free memory at or above vram_pressure_gb when it is set (docs/system-specs/modules/adaptive-concurrency.md:273-281). The Decrease (gate) and Pause rows named only host memory; they now name vram too (docs/system-specs/modules/adaptive-concurrency.md:130, :137), as GATE_SUFFICIENT_ALONE and gate_severe do (src/kiro_crew/adaptive/signals.py:82-84, :302-304). config-baseline.json is regenerated with scripts/generate_config_baseline.py; the new help string is its only change, and test/test_config_baseline.py passes.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=10a4c04f363c

  • Item 1, inherited: the defect has no issue or reproduction. kept: this repository has no issue or measurement of it, and no NVIDIA card or Ollama host is available to this change, so neither a linked issue nor a run showing the gate rising while the card fills can be added here. A new search of this repository's issues and PRs for "VRAM", "NVML", "GPU memory", "nvidia", "vram_critical_gb" and "GPU spawn gate" finds none about GPU memory in the spawn gate (the hits are about NVIDIA speech models, a frame compositing layer and Electron integrations), and the Related Issues section lists the nearest neighbours. The description says the behaviour comes from the servers' documentation, not from a run on a card, and both keys default to 0, which is off. Whether to require a linked issue or such a run before this ships is the maintainer's call.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=b6ffd5c1c38b

  • Item 7, oversized: DEFAULT_TTL_SECS plus NvmlReader._lock sit under a single caller. fixed: both claims hold in the code, so the cache and the locks are deleted. The only caller of read_gpu is the tick (src/kiro_crew/adaptive/controller.py:574), which starts a read only when none is in flight (src/kiro_crew/adaptive/controller.py:570-575, no await between the check and the assignment), and controller_sample_secs is clamped to 1..300 (src/kiro_crew/config/loader.py:3647, help text at src/kiro_crew/config/sections_agent.py:960-961), so a read never runs twice at once and never starts more than once a second. Removed: DEFAULT_TTL_SECS, ttl_secs, clock, _cached, _cached_at, NvmlReader._lock and the module-level _reader_lock (the inventory counts two locks). src/kiro_crew/adaptive/gpu_probe.py keeps only the init-failure latch (:75-146) and builds its shared reader without a lock (:154-170). The module docstring (src/kiro_crew/adaptive/gpu_probe.py:32-38) and the _read_gpu_into docstring (src/kiro_crew/adaptive/controller.py:555-566) now say the read is uncached and the reader holds no lock, and the spec drops "cached for 2 s" (docs/system-specs/modules/adaptive-concurrency.md:256-258). Tests: test_reads_are_cached_for_the_ttl is deleted with the fake clock, the ttl arguments and a counter nothing else read (test/test_adaptive_gpu_probe.py); the 17 probe tests that remain pass in 3 of 3 serial runs.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 16, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

span=75805832a0fa

  • Item 3, symptom-level: the spawn gate is an indirect lever on the card. kept: that is accurate, and the spec and the help text say so. The reading is the card's free memory, whoever holds it; it follows sub-agent load only where a model is loaded when a request first names it and several stay resident, and free memory comes back when the server unloads a model, not when the gate cuts (docs/system-specs/modules/adaptive-concurrency.md:265-281; src/kiro_crew/config/sections_agent.py:612-648). The gate is the one actuator a free-memory signal may move here: the execution cap was taken off memory because a count cut moved by memory stalled other chats' sub-agents while memory was still free (adaptive-concurrency.md:17-29), and HOST_ONLY_SIGNALS keeps vram, like memory, off that cap (src/kiro_crew/adaptive/signals.py:64-67, :281-283). On main (f9031e8) the gate already cuts and pauses on host free memory, through memory in GATE_SUFFICIENT_ALONE (signals.py:74 there) and in gate_severe (:283-285 there); this change puts vram beside it (signals.py:82-84, :301-304). The exposure the item names is real on a host that sets a line: a paused gate still admits its floor, one start at a time at the default floor of 1 (adaptive-concurrency.md:201-203, src/kiro_crew/adaptive/policy.py:125), and the 90 s readiness record is about loop lag on an idle host (docs/system-specs/modules/adaptive-concurrency.md:31-38), not about a card. It reaches only hosts that opt in: both keys default to 0 (sections_agent.py:613, :631), and the help of agent.vram_critical_gb tells the operator to set the line below the free memory left with the models expected to stay loaded (sections_agent.py:643-645). Neither way to clear the item can be met here: no run on an NVIDIA host exists (the test host has no card, as the description's Manual verification says), and accepting the indirect lever is the maintainer's call, the question their review comment on this PR asks.

@javenciu

Copy link
Copy Markdown
Contributor Author

span=90641c4deadc

@javenciu

Copy link
Copy Markdown
Contributor Author
  • Run the read once on a real NVIDIA card before merge. kept: no NVIDIA card is available to this change, so that run cannot be added here, and whether to require it before merge is the maintainer's call. The description says so in three places: the Scope bullet ("This comes from the servers' documentation, not from a run on a card"), the Tests section ("Limitation: the probe has not yet run against a real NVIDIA driver") and Manual verification ("Not done: there is no NVIDIA GPU on the test host"), which lists what a real card still has to show: install the gpu extra, set agent.vram_critical_gb above the card's idle free memory, and check that the controller snapshot shows vram_free_mb and that the spawn gate decreases. What stands in for the run is a fake pynvml that has the real binding's function names and fields (test/test_adaptive_gpu_probe.py), and both keys default to 0, which keeps the read off (src/kiro_crew/config/sections_agent.py:613, :631; src/kiro_crew/adaptive/controller.py:567-569).

@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at ec08232, pending one maintainer decision.

I found no code defect. Both new keys default to 0, which keeps NVML unimported. The GPU read runs off the event loop with a 1 s bound and at most one read in flight (controller.py L555-L589). The signal fires only on a measured card (signals.py L322-L326). The help text now says where free VRAM follows sub-agent load and where it stays flat, which answers my earlier question. CI is green.

What blocks approval: the current-head First Principles CONCERNS (Items 3 and 4) and the Design punchline can each clear only through a run on a real NVIDIA card or a maintainer accepting the change. The open points are that the spawn gate is an indirect lever on GPU memory, and that no report or measurement shows the problem. Your dispositions agree and leave the call to a maintainer. Auto-merge is armed, so the human approval is the only gate left.

This clears when a maintainer accepts shipping the opt-in lever without a real-card run, or the description adds a measured run on an NVIDIA host.

— crew-pr-reviewer (r108)

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 17, 2026
@javenciu
javenciu disabled auto-merge October 18, 2026 05:31
@javenciu

Copy link
Copy Markdown
Contributor Author

Closing without merging, to keep my open PRs to the ones close to landing.

The Oct 17 hold found no code defect, but it clears only with a measured run on an NVIDIA host or a maintainer accepting the opt-in lever without one. The branch also conflicts with main's split of config/loader.py (#20550).

The VRAM signal isn't on main as of 2e50054. I've deleted the branch; it can be restored from this page at ec08232.

@javenciu javenciu closed this Oct 18, 2026
@javenciu
javenciu deleted the feat/adaptive-vram-signal branch October 18, 2026 05:31
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants