Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/scripts/slow_test_durations.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
"""Rewrite tst/integration/slow_durations.json from the slow tier's JUnit reports, so the CI shards stay balanced.

python .github/scripts/slow_test_durations.py junit-integration-slow-*.xml
gh run download <run id> --pattern 'junit-integration-local-*'
python .github/scripts/slow_test_durations.py junit-integration-local-*/junit-integration-slow.xml

Download the reports from a green ``main`` run (the ``junit-integration-local-*`` artifacts of every
``integration-local-slow`` and ``integration-local-gateway`` shard). Each test's time counts toward its file,
Use a green ``main`` run: it has a ``junit-integration-local-*`` artifact for every ``integration-local-slow`` and
``integration-local-gateway`` shard, which ``gh run download`` puts in a folder each. Each test's time counts toward its file,
or toward the test itself for a file the table names in ``split_by_test``; that list is kept as it is.
"""

Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/local-backends.yml
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,11 @@ jobs:
# (tst/util/shards.py). The gateway tests are a check of their own, split the same way. Each check passes
# when all of its shards do (the integration-local-slow and integration-local-gateway jobs below).
integration-local-slow-shard:
name: integration-local-slow (${{ matrix.shard }}/4)
name: integration-local-slow (${{ matrix.shard }}/8)
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4]
shard: [1, 2, 3, 4, 5, 6, 7, 8]
runs-on: ubuntu-latest
timeout-minutes: 40
steps:
Expand All @@ -140,7 +140,7 @@ jobs:
timeout 30 sh -c 'until curl -sf http://localhost:5000/v2/ >/dev/null; do sleep 1; done'
- name: Slow integration tier on the local defaults
env:
AGENTENV_TEST_SHARD: ${{ matrix.shard }}/4
AGENTENV_TEST_SHARD: ${{ matrix.shard }}/8
run: |
.venv/bin/python -m pytest -m int_test_slow -rA --timeout=1200 --junitxml=junit-integration-slow.xml \
tst/integration --ignore=tst/integration/env/gateway/gateway_test.py
Expand Down
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,8 @@ orchestrators call it the same way. What callers rely on:
backends. Minutes-long tests (real image builds, sandbox VMs, gateways, agents) carry
`@pytest.mark.int_test_slow` (or module-level `pytestmark`) and run serially. CI splits them across
runners instead: `AGENTENV_TEST_SHARD=i/N` (`tst/util/shards.py`) runs the i-th of N groups, keeping a
file's tests together and balancing the groups by `tst/integration/slow_durations.json`. When the
file's tests together unless the table's `split_by_test` lists the file, and balancing the groups by
`tst/integration/slow_durations.json`. When the
shards' times drift apart, rewrite the table from a `main` run's `junit-integration-local-*`
artifacts with `.github/scripts/slow_test_durations.py`.
- `tst/manual/` and `tst/benchmarks/` run only by hand, as each module's docstring says; no CI job
Expand All @@ -281,7 +282,7 @@ resolved, and the checks the `main` ruleset requires: `unit`, `integration-local
pass, but a failing one does not block the merge.
`.github/workflows/local-backends.yml` runs `unit`, `ui`, `integration-local`, `integration-local-slow`,
`integration-local-gateway` (the slow tier's `gateway_test.py`, run beside the rest of it) and
`installer`. The slow tier runs on four runners (`integration-local-slow (1/4)` to `(4/4)`) and the
`installer`. The slow tier runs on eight runners (`integration-local-slow (1/8)` to `(8/8)`) and the
gateway tests on two, and `integration-local-slow` and `integration-local-gateway` each pass when all
of their shards do. The Python jobs install from `uv.lock` (public PyPI only), with the uv that
`.github/ci-requirements.txt` pins, and use no secrets and no external services (a `registry:2`
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ unit-test: ## Run the unit test suite in parallel (no external services)
# clock but is the only way to keep slow runs reliable without rewriting
# every module's fixtures with worker-unique tags. CI splits the slow tier
# across runners instead (tst/util/shards.py). To run one runner's share:
# AGENTENV_TEST_SHARD=2/4 make int-test-slow PYTEST_ARGS=--ignore=tst/integration/env/gateway/gateway_test.py
# AGENTENV_TEST_SHARD=2/8 make int-test-slow PYTEST_ARGS=--ignore=tst/integration/env/gateway/gateway_test.py
# AGENTENV_TEST_SHARD=1/2 .venv/bin/python -m pytest -m int_test_slow tst/integration/env/gateway/gateway_test.py
INT_PYTEST_ARGS_FAST ?= --dist=loadscope --tb=short --durations=20 -p no:cacheprovider
INT_PYTEST_ARGS_SLOW ?= --tb=short --durations=20 -p no:cacheprovider
Expand Down
Loading
Loading