Skip to content

test: build a large crew-log unit in groups, skip an unreported MCP wait - #36747

Merged
iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/slow-tests-2
Nov 13, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/slow-tests-2

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Seven more slow tests take the same code path with less waiting or less fixture cost, saving about 105 s of test time per run with no check weakened.

The large crew-log savepoint test spent most of its time writing its fixture one fsync at a time. The real-backend consolidation tests waited out a 6 s MCP drain on every start, for a server the fake backend never reports.

Why it matters

Every PR runs these tests on every shard. Time spent building a fixture or waiting on a constant the test never asserts on buys no coverage.

Not a goal

  • No src/ change.
  • No test removed, skipped, renamed or marked slow.
  • test_crew_log_emit_continuity.py is not changed; it is reported below as a finding for its owner.

What changed (motivation → approach → change)

File Tests Why it was slow Fix Before → after (s)
test/test_crew_log_slot_savepoint_large_state.py 4 300 to 19,200 filler entries appended one fsync each; a quadratic hex builder the same entries through append_many in groups of 300; the hex built in one pass 79.1 → 4.0
test/test_consolidation_fresh_conversation.py 3 each real start waited out the 6 s no-report MCP drain _MCP_DRAIN_NO_REPORT_CEILING 0.05 s, set through the session_handle facade 37.5 → 7.4

Measured alone at -n0. Every filler size (300, 1,200, 4,800, 19,200) is kept, and so is the real child process.

Finding for the owner: test_crew_log_emit_continuity.py

test_a_claim_does_not_take_back_a_pin_a_queued_closer_still_owes (20 s) claims its terminal event is queued, held behind a blocked writer, while a re-claim lands. In practice the writer runs that append inline on the test thread. A stack dump at 5 s shows on_turn_completed → writer.submit → the test's _held, all on the main thread. So _held's release.wait(20.0) runs out before the re-claim, its assert raises inside the job, and the writer retains the failed entry.

The re-claim therefore meets a closer that already failed and was retained, not one waiting in the queue. The test exercises the retained-after-failure path, not the queued path its docstring describes. Its 20 s is that wait running out. I did not check whether its mutation guard still turns red.

Backwards compatibility

Compatible: test-only change.

Removes nothing: no test, fixture or helper is removed; the collected test ids are identical before and after.

Tests

Proof: not provable -- test-only change

Planted bugs in src/, each reverted after the run:

Planted bug Caught by
a large slot state is never compressed (crew_log/slot_state_codec.py) all 4 large-state cases
a stored savepoint is never read back (crew_log/checkpoint.py) all 4 large-state cases
no fresh conversation between jobs (llm_helpers_parts/background_turns.py) test_back_to_back_jobs_each_send_only_their_own_prompt_on_one_runtime, test_a_failed_new_conversation_falls_back_to_a_fresh_runtime and four seam tests
a failed new_conversation has no reset fallback test_a_failed_new_conversation_falls_back_to_a_fresh_runtime

Both files passed 10 times at -n0, 5 times at -n 2 beside test_crew_log_slot_savepoint_growth.py, and once in reversed order. Flake-pattern, wait-discipline and facade-mirror guards pass; the short budget carries a # flake-ok: reason. flake8, isort, black, comment-history and brand checks pass.

Looked at and left as they are:

Test Why
test_taskq_admission_integration.py::test_2000_submissions_... the 2000-task burst is the acceptance test named in rfc-overload-resilience.md
test_crew_log_slot_savepoint_growth.py, ..._seal_on_rollover.py already grouped; the full host-shaped unit is the point
test_session_storage_empty_trash_removal.py deep batch depth 600 is set against the interpreter stack limit, on the trash-deletion path
test_kas_bundle_api.py install test red inside an agent sandbox (the installer refuses there), so not a speed issue
test_remote_git_channel_bounds.py (10 s teardown) the time is the shared conftest joining a leftover executor job

Manual verification

N/A — test-only change; the timings and planted-bug runs above are the verification.

Related Issues

Refs #25195

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

The large-state savepoint test appended its filler one entry and one
fsync at a time, and built its hex fixture in quadratic time; it now
writes the same entries with append_many in groups and builds the hex
in one pass. The real-backend consolidation tests waited out the 6 s
no-report MCP drain on every start; the fake reports no server, so the
ceiling is cut to 0.05 s through the session_handle facade.
@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 13, 2026 16:42
@iamwhatever
iamwhatever enabled auto-merge (squash) November 13, 2026 16:42
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 236581415248a3605e31f53e4ef9cfdcc9c982d9 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 236581415248a3605e31f53e4ef9cfdcc9c982d9 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 236581415248a3605e31f53e4ef9cfdcc9c982d9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. This is a test-only speed-up: the filler entries and the code path under test stay the same, and the MCP drain ceiling is shortened through the facade that already mirrors it into session_handle_mcp.

Notes

  • With append_many, filler entries reach the crew log in groups of 300, one write and one fsync per group, instead of one per entry. If a later savepoint or rollover test depends on what happens between individual appends, it needs its own fixture rather than this grouped one.

[DESIGN-REVIEWED] 2365814

Description read

[DESCRIPTION-READ] f6ff97f5ad8d48271c2a62393bbce3defeacdd3a56c53509184b159923f828bb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 236581415248a3605e31f53e4ef9cfdcc9c982d9 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2365814

Verdict parsed from the review's SHA-scoped output markers for commit 236581415248a3605e31f53e4ef9cfdcc9c982d9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 236581415248a3605e31f53e4ef9cfdcc9c982d9 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 236581415248a3605e31f53e4ef9cfdcc9c982d9 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2365814

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 236581415248a3605e31f53e4ef9cfdcc9c982d9 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 13, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. On head 2365814 the review checked that _MCP_DRAIN_NO_REPORT_CEILING is patched through the session_handle facade, which test_session_handle_bindings.py pins as reaching the owner and being undone. It also checked that the fake backend never sends an MCP report, so the old 6 s ceiling was pure waiting, and that the new _hex loop runs ceil(chars/64) times, so its output is byte-for-byte the same. Finally, it checked that append_many runs the same validation and writes the same types, data and seqs as one append per entry. The diff merges cleanly with #36730.
Done well: every filler size from 300 to 19,200 and every assertion stay as they were. The savings come only from grouping writes through the store's own API.

  • Optional: test/test_crew_log_slot_savepoint_large_state.py:124 could call the existing _append_grouped helper in test/crew_log_projection_helpers.py instead of keeping its own _FILLER_GROUP loop. That helper already has an equivalence test.
  • Optional: the comment near line 132 says these are "the same entries" an append each would write. In fact each group of 300 shares one time stamp. No assertion reads entry time, so this only affects the wording.

@iamwhatever
iamwhatever merged commit e20f2a0 into main Nov 13, 2026
97 of 99 checks passed
@iamwhatever
iamwhatever deleted the crew/testoptimizer/slow-tests-2 branch November 13, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants