[earakely-scale]

Why

integration-local-slow runs its tests serially, because they build the same image tags and use the same artifact ids, so parallel workers on one Docker daemon race. On main it took 31–36 minutes last week, and it now reaches its 40-minute timeout: three of the last four runs on main were cancelled. integration-local-gateway (gateway_test.py alone) takes about 19 minutes.

Almost all of that is per-test deploys of envs and agents, 30–130 seconds each. The setup steps (install, base images, registry) take under a minute.

What

Split both across runners. Each runner has its own Docker daemon, so the races that keep the tier serial can't happen between runners. The tests stay serial on each one.

• AGENTENV_TEST_SHARD=i/N (tst/util/shards.py, a trylast hook in tst/conftest.py) keeps the i-th of N groups of the selected tests.
  • It runs after -m and the other filters, so only the selected tests are split.
  • A file's tests stay together, so its module fixtures run once per runner. gateway_test.py, a single file, is split by test.
  • Groups are balanced by the seconds in tst/integration/slow_durations.json, and a unit the table doesn't know weighs the median.
  • The same table and selection always give the same groups. Without the variable, nothing changes.
• CI:
  • The slow tier runs on four runners, integration-local-slow (1/4) to (4/4), and the gateway tests on two.
  • integration-local-slow and integration-local-gateway are now small jobs that pass when every shard of theirs did. So the required check keeps its name, and the gateway tests stay a separate check that doesn't block the merge.
  • Each shard uploads its JUnit report (junit-integration-local-*, kept 14 days).
• .github/scripts/slow_test_durations.py rewrites the table from those reports, summing each file's tests, and each test's own time for a file in split_by_test.
  • The first table comes from the per-test times in a recent green main run's logs.
• Docs: AGENTS.md (Tests, CI) and the Makefile's slow-tier note say how to run one shard locally: AGENTENV_TEST_SHARD=2/4 make int-test-slow.

Expected: about 10 minutes of tests per slow shard and about 9.5 per gateway shard. On top of that, each runner adds about a minute of setup plus its own first image builds.

Tests

Command	Result
.venv/bin/python -m pytest tst/unit/test_shards.py tst/unit/test_slow_test_durations_script.py	20 passed
.venv/bin/python -m pytest tst/unit -n auto	10280 passed, 66 skipped
.venv/bin/ruff check	All checks passed
pytest --collect-only -m int_test_slow tst/integration --ignore=tst/integration/env/gateway/gateway_test.py, unsharded and with AGENTENV_TEST_SHARD=1/4 … 4/4	The four shards are disjoint and together are the 157 collected tests. The table predicts 9.9, 9.2, 9.9 and 9.9 minutes.
The same for gateway_test.py with 1/2 and 2/2	13 and 605 tests (most skip without a model or remote sandbox); 9.5 and 9.3 minutes predicted

This PR's own CI run is the end-to-end check of the shards' times.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; the refresh example has a non-blocking filename mismatch.

Findings

1.  Refresh command finds no reports ▶

Fix with agent prompt

### Issue 1
.github/scripts/slow_test_durations.py:3
Every shard uploads `junit-integration-slow.xml`, but this example looks for `junit-integration-slow-*.xml`. After downloading the artifacts into separate folders, the pattern matches nothing and the script fails before updating the table. Developers must work out the correct paths themselves.

Use a command that reads the reports from those folders: `python .github/scripts/slow_test_durations.py junit-integration-local-*/junit-integration-slow.xml`.

```suggestion
    python .github/scripts/slow_test_durations.py junit-integration-local-*/junit-integration-slow.xml
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Splits slow integration tests across four runners and gateway tests across two, while keeping tests serial within each runner.

• Selected tests run in repeatable, duration-balanced groups.
• Slow and gateway integration checks run across separate runners.
• JUnit reports can refresh the shard duration table.
• Module cleanup closes local grant servers between test files.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Tests[Selected slow tests] --> Selection{Test selection}
  Selection --> Slow[Non-gateway tests]
  Selection --> Gateway[Gateway tests]
  Table[Saved durations] --> Groups[Assign weighted groups]
  Slow --> Groups
  Gateway --> Groups
  Groups --> Four[Four slow runners]
  Groups --> Two[Two gateway runners]
  Four --> SlowCheck[Existing slow check]
  Two --> GatewayCheck[Existing gateway check]
  Four --> Reports[JUnit artifacts]
  Two --> Reports
  Reports --> Refresh[Manual table refresh]
  Refresh --> Table

Loading

_{Reviews (4) · Last reviewed commit: "Merge branch 'main' into ci/shard-slow-i..." · Reviewed by Greptile}