Skip to content

test: cover the spawn admission seams behind bugs 14, 15 and 20 - #14160

Merged
iamwhatever merged 1 commit into
mainfrom
test/integration-governance
Sep 28, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
test/integration-governance

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Three reported spawn-admission and security-baseline behaviours (bugs 14, 15, 20) are pinned as tests through the running gateway, so a regression in any of them fails CI by route and by message.

The reports behind bugs 14, 15 and 20 had no test that drove the real route: a spawn outside the parent spec's allow-list, a memory-held spawn described as a capacity queue, and a report that the agent read a credential file were each investigated by hand and left no artefact that would catch the same regression again.

Why it matters

Spawn admission is an authorization boundary and the security snapshots are what a report is triaged against; without a route-level pin a fix can silently regress, and the next report costs the same investigation again.

Not a goal

  • Fixing any product code: this slice only pins behaviour that main already has.
  • Reproducing bug 15's credential read: the in-process boot runs neither the OS sandbox nor the shell gate's hook process, so this pins the baseline a report is triaged against, not the read itself.
  • The network.egress floor or any other governance change.

What changed (motivation -> approach -> change)

Third slice of the in-process integration layer (harness: #13621; memory seams: #13736). Six tests in test/integration/test_governance.py drive POST /api/spawn and the /api/security/* snapshots through a booted gateway with the credentials a real MCP caller presents (gw.mcp_headers), and pin what the product answers today for three reported bugs.

Based on main (#13736 supplied the cookie auth and mcp_headers this slice uses; #14269 supplied the sandbox consent in integration_home and the 95 ratchet). Independent of #14258.

Fixes #14180 (bug 14: the allow-list gate landed on main in #13683; this test pins it through the real route, so the tracking issue closes with this PR). Bugs 15 and 20 come from Slack reports with no GitHub issue; no linked issue for them: 20 is already fixed on main and 15 is a baseline pin, not a defect.

What each test pins

Test Seam Today
test_spawn_is_guarded /api/spawn without credentials refused
test_a_memory_deferral_is_named_as_one persistent parent, memory bar raised via config.local.json queued / low_memory / detail names GB, not the cap (bug 20)
test_a_memory_refusal_names_the_memory incognito parent, same bar 400 spawn refused: only … GB memory available, counted: true (bug 20)
test_a_spawn_outside_the_allow_list_is_refused parent spec toolsSettings.subagent.availableAgents: ["ally"], spawn stranger 400 agent_not_available; message names stranger, gatekeeper and ally (bug 14)
test_security_snapshots_are_guarded three /api/security/* reads without credentials refused
test_a_fresh_home_ships_the_credential_ceiling_intact fresh home: denied-command opt-outs, sensitive paths, counts no opt-out, every builtin on, ~/.aws listed, counts agree (bug 15 baseline)

Why the shapes are what they are

  • Bug 20 is asserted against the wording main produces after fix(subagent): name why a queued spawn waits instead of blaming the cap #13725 and fix(subagent): cap concurrent startups, reset start clock at gate exit #13890; the memory bar is raised through the operator's own override file, never by patching the guard.
  • Bug 14 is pinned as the positive contract under kiro-cli's documented field: toolsSettings.subagent.availableAgents is the spawn allow-list (docs/reference/kiro-cli/chat/subagents.md); trustedAgents beside it only waives permission prompts, which is why the reporter's spelling is not the one asserted. The gate (_vet_parent_available_agents, landed in fix: honour the parent agent spec's availableAgents at the spawn gate #13683) refuses before any child starts with agent_not_available; the test asserts the code and the three names in the message (not the prose), so a rewording that keeps them still passes. The _settled guard stays so a regression that admits the spawn cannot leave a child running past the boot.
  • Bug 15: the in-process boot runs neither the OS sandbox nor the shell gate's hook process, so the layer cannot try the read itself. It pins the baseline a report is triaged against instead: if these hold on the reporter's home, the read did not go through the gate these snapshots describe. No credential material is written or read.

Seams documented

Agent specs a slot or a spawn must resolve are written to <home>/kiro/agents before the boot (the loader answers names from a snapshot scanned once; a later write is not dispatchable until the next registration). Boot-time config goes in <home>/config.local.json. Both added to testing-conventions.md.

Harness change

None. Every spawn this slice makes is refused at a gate that runs before any child starts (the memory guard for bug 20, the allow-list check for bug 14), so the layer's conftest is untouched here; _pin_memory_bar merges into config.local.json rather than replacing it, so whatever the harness writes there for another slice survives. No test in this slice depends on a sandbox backend being present or absent.

Coverage

No workflow change: the route sweep (#14269) has merged and holds the Integration (in-process gateway) ratchet at 95, which this slice keeps (pytest test/integration alone on this branch: 1240/1252 routes, 99%). /api/spawn and the three /api/security/* snapshots gain behaviour pins here; the sweep already counted them.

Tests

  • Local, on main: pytest test/integration -n 2 --dist loadgroup -> 27 passed, 34s. CI Integration (in-process gateway) green on the previous head with the same 27.
  • black / isort / flake8 / mypy on the new file; comment-history gate; docs lint.
  • The bug-14 test starts no child: the gate refuses before the spawn, and the _settled guard only runs if a regression admits it.

Backwards compatibility

Compatible: tests and a CI ratchet only; no product code changes.

@CrysisDeu
CrysisDeu requested review from a team and krishdhasmana and removed request for a team September 26, 2026 07:52
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of fe454ec89158f3cf502b89e7269a7ed420a2c8cd — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm the Integration (in-process gateway) lane is green on this head — the description cites the previous head's run.

[DESIGN-REVIEWED] fe454ec

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ⏭️ skipped

Revision fe454ec89158f3cf502b89e7269a7ed420a2c8cd ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of fe454ec89158f3cf502b89e7269a7ed420a2c8cd and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fe454ec

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fe454ec89158f3cf502b89e7269a7ed420a2c8cd: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-governance branch from 5faee32 to f03d12f Compare September 26, 2026 08:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Newly added xfail suppresses a failing admission contract (span=03d49c1dfc2b) -- rebutted

The proposed fix (enforce trustedAgents, drop the xfail) is a product change to spawn admission, out of scope for a test-only slice; this PR reproduces the gap through the real route and records it. A strict xfail on a test this PR creates whole loosens no ratchet (the adjudication confirms it): the assertion is the fixed contract, strict=True makes an unmarked fix fail CI, and the same convention was accepted on #13736. Two things did change in response on f03d12f: the gap is now tracked by #14180, cited from the xfail reason and the assertion message, and the assertion is status-only so a fix whose wording differs still flips the marker (the Design lane's point).

@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Let the bug-14 xfail assert only the refusal status (span=c160e0f26e4c) -- fixed

Agreed: a fix whose error text omits "trusted" would have stayed xfailed and the tracking artefact would never fire. On f03d12f the test asserts status in (400, 403) only; the wording is left to the fixing PR, which is what the docstring now says. The gap is tracked by #14180.

@CrysisDeu
CrysisDeu force-pushed the test/integration-governance branch from f03d12f to c0b473a Compare September 26, 2026 09:04
@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Item 3 -- premise risk: trustedAgents is not the admission contract (span=f58f3405fc9e) -- fixed

Correct, and it changes the reproduction, not the finding. kiro-cli's contract puts the spawn allow-list at toolsSettings.subagent.availableAgents; trustedAgents beside it only waives permission prompts. On c0b473a the parent spec carries toolsSettings.subagent.availableAgents: ["ally"] and the spawn of stranger is still admitted (status: spawned, run to completion locally with --runxfail), so the defect stands under the documented field. The test names the path through a SPAWN_ALLOW_LIST constant and the module docstring says why trustedAgents is not the one asserted. #14180 is updated to record availableAgents as the admission contract and the reporter's trustedAgents spelling as the other half of the report.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Flip condition should be "not admitted", not a status set (span=e5c179a94202) -- fixed

Agreed: a refusal shaped like the deferral path (200 with a non-spawned body) would have kept the xfail and stopped tracking #14180. On c0b473a the assertion is not (status == 200 and body.status in ("spawned", "queued")), so any refusal of any status or shape flips it.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Narrow the flip condition to a deliberate refusal if CI shows a transient non-admission (span=8f39e4becd22) -- accepted-and-deferred

Agreed on the trigger you name: a transient non-admission (a 503 while subagents warm, a 409 on the parent's memory mode) would flip a strict xfail into a loud XPASS failure rather than a silent one, so it cannot hide the defect, but it would misattribute the run. Nothing observed yet -- locally the test settles as xfail on the contract assertion every run, and the Integration (in-process gateway) job only runs once this PR is retargeted to main (stacked PRs get the reduced check set). Deferred to that first main-targeted CI round: if it shows a transient shape, the condition narrows to a 4xx that names the spawn, per your wording; otherwise "not admitted" stays, since it is what flips on a fix of any shape (the previous round's point).

@CrysisDeu
CrysisDeu force-pushed the test/integration-memory branch from 87323cb to 59c25bd Compare September 26, 2026 15:17
@CrysisDeu
CrysisDeu force-pushed the test/integration-governance branch from c0b473a to aaba646 Compare September 26, 2026 15:17
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed fe454ec89158f3cf502b89e7269a7ed420a2c8cd — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fe454ec

Verdict parsed from the review's SHA-scoped output markers for commit fe454ec89158f3cf502b89e7269a7ed420a2c8cd.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fe454ec89158f3cf502b89e7269a7ed420a2c8cd: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-governance branch from d4cf41a to 187b77c Compare September 27, 2026 07:57
@CrysisDeu
CrysisDeu requested a review from a team as a code owner September 27, 2026 07:57
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-governance branch from 187b77c to bd8b888 Compare September 27, 2026 08:13
@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Item 8: the sandbox consent has zero consumers in this slice (span=649a1a77e6c1) -- fixed

Correct: every spawn this slice makes is refused before a child starts (memory guard for bug 20, allow-list check for bug 14), so nothing here needs the consent. The conftest hunk is dropped from this PR on bd8b888; the layer's conftest is untouched by this slice (3 files now). The consent stays where it has consumers: the streaming slice's chat turn and the route sweep's fresh-home answers. Body's Harness section rewritten to say so. 27 passed locally.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Fresh-home baseline pinned on a home the harness configured (span=d7d518d27c9f) -- fixed

The consent write is no longer part of this PR (dropped on bd8b888), so on this branch the home that test boots carries no agent override at all. When another slice lands the consent, it sets agent.sandbox_allow_unsandboxed_exec, which is agent config, not the credential ceiling the test reads (security_policy.json-backed denied-commands, posture, stats); the baseline the test pins is unaffected either way.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 27, 2026
iamwhatever
iamwhatever previously approved these changes Sep 27, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 27, 2026 17:32
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 27, 2026
Third slice of the in-process integration layer: six tests in
test/integration/test_governance.py drive POST /api/spawn and the
/api/security snapshots through a booted gateway with the credentials a
real MCP caller presents, and pin what the product answers today.

Bug 20 (a memory-held spawn reported as "queued behind concurrency
limit"): the memory bar is raised through the operator's own
config.local.json (agent.spawn_min_memory_gb), never by patching the
guard. A persistent parent gets a durable row, so the guard defers it
and the answer must be `queued` with reason `low_memory` and a detail
that names the gigabytes, not the cap. An incognito parent has no row
to park, so the guard refuses, and the refusal names the memory. Both
assert the wording the fix on main now produces.

Bug 14 (the parent agent spec's spawn allow-list): kiro-cli reads it
from toolsSettings.subagent.availableAgents, with trustedAgents beside
it only waiving permission prompts. The spawn gate honours the list: a
parent slot runs a spec whose list names one agent, and a spawn of
another installed agent is refused with agent_not_available and a
message that names the agent, the parent and the list -- pinned as the
positive contract through the real route, no child started.

Bug 15 (a report that the agent read ~/.aws/credentials): the
in-process boot cannot run the OS sandbox or the shell gate's hook
process, so the layer pins the baseline a report is triaged against
instead -- on a fresh home no denied-command opt-out is recorded, every
built-in rule is on, the AWS credential directory is among the
sensitive paths, and the three snapshots agree on their counts.

Agent specs a slot or a spawn must resolve are written to
<home>/kiro/agents before the boot: the loader answers names from a
snapshot it scans once, and a spec written afterwards is not
dispatchable until the next registration. The conventions doc gains
that seam and the config.local.json one. Route coverage moves to
13/1245 and the ratchet tightens from 0.5% to 0.7%.

The harness also grants the disposable home unsandboxed-exec consent,
as the E2E suite does: the CI container has no sandbox backend, and
without it every child the gateway spawns is refused before the fake
model runs.
@CrysisDeu
CrysisDeu force-pushed the test/integration-governance branch from bd8b888 to fe454ec Compare September 27, 2026 22:07
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Sep 27, 2026
@iamwhatever
iamwhatever merged commit 4cf7b3e into main Sep 28, 2026
99 of 105 checks passed
@iamwhatever
iamwhatever deleted the test/integration-governance branch September 28, 2026 05:30
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 28, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests-only slice; read all six cases. Operator seams (agents dir before boot, config.local.json) are documented in testing-conventions.md. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

toolsSettings.subagent.availableAgents in an agent spec is not enforced on spawn

3 participants