Skip to content

test: pin that two chats stream at once through the gateway - #14258

Merged
iamwhatever merged 1 commit into
mainfrom
test/integration-streaming
Sep 28, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
test/integration-streaming

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Two chats on two slots stream at the same time through the gateway, and a regression to one-at-a-time streaming fails CI.

Bug 16 reported that one session streams while every other one sits in "thinking" until it finishes, then a blob arrives. The fix landed as #13203 with no test that drives two real chat turns concurrently and watches both SSE streams.

Why it matters

Concurrent streaming is the difference between a usable multi-session dashboard and one that serialises every user; a regression is invisible to unit tests and only shows up as user lag.

Not a goal

  • Bug 17 (session/new timing out with every MCP server reported): agent.session_start_timeout_secs is clamped to at least 90s, so a hang test costs 90s per attempt and belongs at the unit level.
  • Pinning absolute latency: the assertion is on the ORDER of what two SSE clients saw.

What changed (motivation -> approach -> change)

Fourth slice of the in-process integration layer (harness #13621; memory seams #13736; spawn admission #14160). One test, test/integration/test_streaming.py, drives two real POST /api/chat SSE streams on two slots with the fake model's [[SLOW]] turn and pins the fairness contract behind bug 16.

Based on main (#13736 supplied the integration_home agent-spec pin release this slice needs; #14269 supplied the sandbox consent and the 95 ratchet). Independent of #14160.

no linked issue: bug 16 comes from a Slack report; its fix landed as #13203 and this is the regression pin for it.

What the test pins

Step Assertion
both slots warmed with one plain turn session start is outside the measured window
slot A sends [[SLOW]]; wait for its first chunk A streams
slot B sends [[SLOW]] while A is streaming B's first chunk arrives before A's last ([DONE])
meanwhile A keeps producing chunks after B starts streaming
end both streams delivered all thirty chunks

Local timings for the record (not asserted), warm slots: A first chunk +0.02s, B sent +0.05s, B first chunk +0.08s, A done +15.1s, B done +15.2s. The report's symptom would put B's first chunk after A's [DONE].

Why the shape is what it is

  • Assertions are on ORDER of what two SSE clients saw, never on absolute latency. Both slots are warmed first so a cold session start (several seconds, host-dependent) is outside the measured window and the margin is the whole slow turn.
  • Nothing inside the process is patched or timed. The [[SLOW]] turn is an existing fake-model behaviour (SLOW_CHUNKS, SLOW_CHUNK_DELAY_SECS are imported, not restated).
  • Bug 17 (session/new timed out after 90s (4/4 MCP server(s) reported) + Resume loop) is deliberately not here: agent.session_start_timeout_secs is clamped to >= 90s in the loader, so a hang test costs 90s+ per attempt, and the wording fix (session-injected MCP server(s) reported) is already on main. Its pin belongs at the unit level.

Harness changes

None on this branch: the unsandboxed-exec consent this slice's chat turn needs (the CI container has no sandbox backend, so without it the spawn is refused before the fake stub runs) landed in integration_home with #14269 and is inherited here.

Coverage

No workflow change: the route sweep (#14269) has merged and holds the Integration (in-process gateway) ratchet at 95, which this slice keeps (pytest test/integration alone on this branch: 1240/1252 routes, 99%). POST /api/chat gains a behaviour pin here; the sweep already counted it. docs/system-specs/common/testing-conventions.md gains the SSE reading pattern.

Tests

Backwards compatibility

Compatible: tests and a CI ratchet only; no product code changes.

@CrysisDeu
CrysisDeu requested review from a team and chenmingwei23 and removed request for a team September 26, 2026 16:32
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 4e942b175267a6bffc25d0d3a2e55b562c36066f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the test stays green under a loaded shared CI runner — the order assertion's margin is one full 15s slow turn on warm slots.

[DESIGN-REVIEWED] 4e942b1

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 4e942b175267a6bffc25d0d3a2e55b562c36066f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4e942b1

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4e942b175267a6bffc25d0d3a2e55b562c36066f: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ⏭️ skipped

Revision 4e942b175267a6bffc25d0d3a2e55b562c36066f ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@CrysisDeu
CrysisDeu force-pushed the test/integration-streaming branch from 940a065 to df4f2e4 Compare September 26, 2026 16:56
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • The order assertion's window includes B's unbounded cold session start (span=21d2f099d3b4) -- fixed

Agreed. On df4f2e4 both slots are warmed with one plain turn before the race, so the measured window holds concurrent streaming only. Local timings with warm slots: A first chunk +0.02s, B sent +0.05s, B first chunk +0.08s, A done +15.1s, B done +15.2s -- the margin is now the whole slow turn rather than what a cold session/new left of it. The interleaving assertion was re-pointed at the same time: with warm slots B's first chunk lands 30ms after it is sent, before A's next half-second tick, so "A chunked between B being sent and B's first chunk" was the wrong window; it now asserts A produced chunks after B started streaming.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Warm both slots with a trivial turn before the race (span=a6e7c02fd7d8) -- fixed

Done on df4f2e4: one plain turn on each slot, awaited, before the [[SLOW]] race; the docstring says why the warm-up is there.

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 4e942b175267a6bffc25d0d3a2e55b562c36066f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4e942b1

Verdict parsed from the review's SHA-scoped output markers for commit 4e942b175267a6bffc25d0d3a2e55b562c36066f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4e942b175267a6bffc25d0d3a2e55b562c36066f: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
Base automatically changed from test/integration-memory to main September 26, 2026 22:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 26, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-streaming branch from df4f2e4 to d1dad73 Compare September 27, 2026 04:32
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Sep 27, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-streaming branch from d1dad73 to 660029c Compare September 27, 2026 05:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-streaming branch from 660029c to 0caa52d Compare September 27, 2026 05:13
@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Item 3 -- the unsandboxed-exec consent is undeclared (span=216a7e58576b) -- fixed

Correct. It is now the first entry under "Harness changes" in the description, with the same rationale the E2E suite records for its gateway (test/test_playwright_e2e.py): the binary is the fake stub, the CI container has no sandbox backend, and the sandboxed-spawn proof lives in the e2e-private-namespace / e2e-boot-matrix lanes. The first CI round on main is what surfaced the need -- the chat turn was refused before the stub ran.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 27, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Sep 27, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-streaming branch from b7310da to 33c2491 Compare September 28, 2026 01:55
@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Documented chat call passes a set, not a dict (span=90dfb3da3a34) -- fixed

Correct: {"message", "slot"} is a set literal and aiohttp cannot serialize it. On 33c2491 the sentence shows the call as the test actually makes it, gw.post("/api/chat", {"message": text, "slot": slot_key}, timeout=TURN_SECS), so a reader copying it gets the SSE stream.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 28, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-streaming branch from 33c2491 to 1f5db1f Compare September 28, 2026 02:31
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 28, 2026
@CrysisDeu
CrysisDeu force-pushed the test/integration-streaming branch from 1f5db1f to 133c354 Compare September 28, 2026 03:12
@CrysisDeu

Copy link
Copy Markdown
Contributor Author
  • Unreachable timeout ceilings in test_streaming.py (span=8827740fed78) -- fixed

Correct: with TURN_SECS = 120 and no module mark, the 120 s first-chunk wait and the 240 s gather sat at or above the item budget, so on the wedge this file exists to catch pytest-timeout would have killed the worker before pytest.fail ran. On 133c354 the module carries pytest.mark.timeout(ITEM_SECS) with ITEM_SECS = 180, and TURN_SECS = ITEM_SECS / 4 (45 s) is derived from it at the definition: the largest single wait is 2 * TURN_SECS = 90 s, leaving the other half of the budget for boot, the warm-up turns and teardown. The docs sentence now says a turn is bounded from the module's own mark rather than "generously". Layer re-run: 26 passed, 99% at min 95.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 28, 2026
Fourth slice of the in-process integration layer. One test in
test/integration/test_streaming.py drives two real POST /api/chat SSE
streams on two slots with the fake model's [[SLOW]] turn (thirty chunks
half a second apart) and asserts the fairness contract behind bug 16:
slot B, sent once slot A is streaming, must produce its first chunk
before A produces its last, and A must keep producing chunks while B
starts. The report was the opposite -- one session streams, every
other one sits in "thinking" until it finishes -- and the fix landed
as #13203; this is the regression pin for it. Nothing inside the
process is patched or timed, only what the two SSE clients see, and
the assertions are on ORDER, never on absolute latency (a cold session
start costs several seconds before the first chunk).

Bug 17 (session/new timing out with every MCP server reported) is not
in this slice: agent.session_start_timeout_secs is clamped to at least
90 seconds, so a hang test costs a minute and a half per attempt, and
the wording fix that separates the MCP count from the verdict is
already on main. Its pin belongs at the unit level.

Route coverage moves to 9/1247 and the ratchet tightens from 0.5% to
0.7%. The conventions doc gains the SSE reading pattern.

The harness also grants the disposable home unsandboxed-exec consent,
as the E2E suite does: the CI container has no sandbox backend, and
without it every child the gateway spawns is refused before the fake
model runs.
@CrysisDeu
CrysisDeu force-pushed the test/integration-streaming branch from 133c354 to 4e942b1 Compare September 28, 2026 05:34
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants