Repository navigation

ci(slow tier): keep the computer tests' resolved packages between modules and runs #3285

Workflow file for this run

.github/workflows/local-backends.yml at 40c3338

	name: Local backends

	# The public tier: the unit suite and the backend-agnostic integration suite under the built-in
	# local defaults (SQLite, filesystem objects, a local registry, Docker sandboxes), with no secrets
	# and no external services. Tests whose assertions are about one particular backend live with
	# that backend's own suite, not here.

	on:
	pull_request:
	push:
	branches: [main]

	permissions:
	contents: read

	# One group per pull request, so a newer push cancels the superseded run; one group per commit on
	# main, so no push to main is ever cancelled or queued behind another (a group holds at most one
	# pending run, and the version bump lands seconds after every merge).
	concurrency:
	group: local-backends-${{ github.event_name == 'pull_request' && github.ref \|\| github.sha }}
	cancel-in-progress: ${{ github.event_name == 'pull_request' }}

	env:
	AGENT_ENV_ENVIRONMENT: dev
	PIP_DISABLE_PIP_VERSION_CHECK: "1"

	jobs:
	# The unit tier on two runners, AGENTENV_TEST_SHARD picking each one's share by the times in
	# tst/integration/slow_durations.json (tst/util/shards.py); unit passes when both shards do. The
	# lockfile and lint checks run on the first shard only.
	unit-shard:
	name: unit (${{ matrix.shard }}/2)
	strategy:
	fail-fast: false
	matrix:
	shard: [1, 2]
	runs-on: ubuntu-latest
	timeout-minutes: 20
	steps:
	- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
	- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
	with:
	python-version: "3.12"
	- name: Install from uv.lock
	run: \|
	python -m pip install --quiet -r .github/ci-requirements.txt
	uv sync --locked --extra dev --python "$(command -v python)"
	echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
	- name: Lockfile resolves from PyPI only
	if: matrix.shard == 1
	run: \|
	if grep 'source = { registry = ' uv.lock \| grep -v 'registry = "https://pypi.org/simple"' \| grep -q .; then echo "uv.lock has a registry source other than PyPI"; exit 1; fi
	grep -A2 '^name = "agentenv-framework-protocol"$' uv.lock \| grep -q 'source = { editable = "packages/agentenv-protocol" }'
	- name: Lockfile is current, before and after a release bump
	if: matrix.shard == 1
	run: \|
	uv lock --check
	uv run --no-project --with tomlkit python scripts/bump_version.py
	uv lock --check
	git checkout -- pyproject.toml packages/agentenv-protocol/pyproject.toml uv.lock
	- name: Lint
	if: matrix.shard == 1
	run: .venv/bin/ruff check
	# worksteal: a worker that runs out of tests takes queued ones from the others, so no worker idles at the end.
	- name: Unit tier
	env:
	AGENTENV_TEST_SHARD: ${{ matrix.shard }}/2
	run: \|
	.venv/bin/python -m pytest tst/unit packages/agentenv-protocol/tests -n auto --dist worksteal -q --timeout=300 \
	--durations=30 --junitxml=junit-unit.xml
	# The tests' times, for .github/scripts/slow_test_durations.py to rebalance the shards with.
	- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	if: always()
	with:
	name: junit-unit-${{ matrix.shard }}
	path: junit-unit.xml
	retention-days: 14

	# One check under the name the main ruleset requires: it passes when both unit shards did.
	unit:
	if: always()
	needs: unit-shard
	runs-on: ubuntu-latest
	timeout-minutes: 5
	steps:
	- name: Every shard passed
	run: test "${{ needs.unit-shard.result }}" = success

	ui:
	runs-on: ubuntu-latest
	timeout-minutes: 15
	defaults:
	run:
	working-directory: src/agent_env/explorer/ui
	steps:
	- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
	- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
	with:
	node-version: "22"
	cache: npm
	cache-dependency-path: src/agent_env/explorer/ui/package-lock.json
	- run: npm ci
	- name: Typecheck
	run: npm run typecheck
	- name: Lint
	run: npm run lint
	- name: Smoke suite
	run: npm run test:smoke
	- name: Static export
	run: npm run build:static

	integration-local:
	runs-on: ubuntu-latest
	timeout-minutes: 45
	services:
	registry:
	image: mirror.gcr.io/library/registry:2
	ports:
	- 5000:5000
	steps:
	- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
	- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
	with:
	python-version: "3.12"
	- name: Install from uv.lock
	run: \|
	python -m pip install --quiet -r .github/ci-requirements.txt
	uv sync --locked --extra dev --python "$(command -v python)"
	echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
	# loadscope keeps a module's or class's tests on one worker, so its fixtures run once, as make int-test-fast does.
	- name: Integration tier on the local defaults
	run: \|
	.venv/bin/python -m pytest -m 'not int_test_slow' -n auto --dist loadscope -rA --timeout=900 \
	--junitxml=junit-integration.xml tst/integration
	- name: Skips must be declared capability gaps
	if: always()
	run: \|
	.venv/bin/python .github/scripts/check_skip_policy.py junit-integration.xml \
	--allow model_endpoint_configured remote_sandbox \
	--no-skips-under tst/integration/store/

	# The slow tier: builds the env images and the echo A2A agent (tst/data/a2a_agent) and deploys
	# them on the runner's Docker. The tests that need a real model, a remote sandbox, a registered
	# default A2A agent or MCP server sources from outside this repository skip with the capability reason.
	# Its tests run serially, since they share image tags and artifact ids, so it is split across runners
	# instead: AGENTENV_TEST_SHARD picks each runner's share, balanced by tst/integration/slow_durations.json
	# (tst/util/shards.py). The gateway tests are a check of their own, split the same way. Each check passes
	# when all of its shards do (the integration-local-slow and integration-local-gateway jobs below).
	integration-local-slow-shard:
	name: integration-local-slow (${{ matrix.shard }}/12)
	strategy:
	fail-fast: false
	matrix:
	shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
	runs-on: ubuntu-latest
	timeout-minutes: 40
	steps:
	- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
	- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
	with:
	python-version: "3.12"
	- name: Install from uv.lock
	run: \|
	python -m pip install --quiet -r .github/ci-requirements.txt
	uv sync --locked --extra dev --python "$(command -v python)"
	echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
	# ECR Public's anonymous pulls fail here for hours at a time. This can restart dockerd, so the registry
	# starts afterwards.
	- name: Pull the base images from mirror.gcr.io
	run: python3 .github/scripts/mirror_base_images.py src tst
	# A computer image or disk is tagged with a key of everything it is built from, and the composer reuses one the
	# image store already holds for its key. That key includes the lock of the packages the image installs, which the
	# composer resolves from Ubuntu's archive, PyPI and download URLs unless the store holds it. The registry keeps
	# its storage on the runner, and the computer tests share their locks and files in the packages folder
	# (tst/util/package_cache.py); both are restored from what main saved latest this week, so a run neither resolves
	# nor builds again what hasn't changed. A shard that runs no computer tests misses.
	- name: Week of the computer images' cache
	id: week
	run: echo "week=$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT"
	- name: Restore the computer images and packages
	id: computers
	uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: \|
	${{ runner.temp }}/registry
	${{ runner.temp }}/packages
	key: computers-${{ matrix.shard }}-${{ steps.week.outputs.week }}
	restore-keys: computers-${{ matrix.shard }}-${{ steps.week.outputs.week }}-
	- name: Start the local registry
	run: \|
	mkdir -p "$RUNNER_TEMP/registry"
	docker run --detach --name registry --publish 5000:5000 --volume "$RUNNER_TEMP/registry:/var/lib/registry" \
	mirror.gcr.io/library/registry:2
	timeout 30 sh -c 'until curl -sf http://localhost:5000/v2/ >/dev/null; do sleep 1; done'
	# x64 hosted runners have /dev/kvm; the disk form's tests boot under it, and skip without it, which fails below.
	- name: Let the local provider boot disks with KVM
	run: sudo chmod 666 /dev/kvm
	- name: Slow integration tier on the local defaults
	id: tests
	env:
	AGENTENV_TEST_SHARD: ${{ matrix.shard }}/12
	AGENTENV_TEST_PACKAGES: ${{ runner.temp }}/packages
	run: \|
	.venv/bin/python -m pytest -m int_test_slow -rA --timeout=1200 --junitxml=junit-integration-slow.xml \
	tst/integration --ignore=tst/integration/env/gateway/gateway_test.py
	# Only the computer images this run used are kept, with the packages, and only main saves them, once its tests
	# pass, so pull requests reuse main's without filling the repository's cache. An image is used when the registry's
	# log has a request for it by its tag (prune_registry.py), so what a week's earlier runs built and this one no
	# longer asks for is dropped. The garbage collection frees what only the dropped tags and repositories referenced;
	# with --delete-untagged it would also delete the untagged platform manifests of an image index, and their layers.
	# The entry is named by the image keys the registry holds and the locks the packages hold, so a run that built a
	# new image or resolved a new lock saves a new one.
	- name: Keep only the computer images this run used
	id: kept
	if: always()
	run: \|
	repositories="$RUNNER_TEMP/registry/docker/registry/v2/repositories"
	if [ ! -d "$repositories/agentenv-computers" ]; then echo "computers=false" >> "$GITHUB_OUTPUT"; exit 0; fi
	docker logs registry > "$RUNNER_TEMP/registry.log" 2>&1
	docker stop registry
	sudo find "$repositories" -mindepth 1 -maxdepth 1 ! -name agentenv-computers -exec rm -rf {} +
	sudo python3 .github/scripts/prune_registry.py prune "$RUNNER_TEMP/registry" agentenv-computers \
	"$RUNNER_TEMP/registry.log"
	docker run --rm --volume "$RUNNER_TEMP/registry:/var/lib/registry" mirror.gcr.io/library/registry:2 \
	garbage-collect /etc/docker/registry/config.yml > /dev/null
	sudo python3 .github/scripts/prune_registry.py links "$RUNNER_TEMP/registry" agentenv-computers
	mkdir -p "$RUNNER_TEMP/packages"
	du -sh "$RUNNER_TEMP/registry" "$RUNNER_TEMP/packages"
	if [ -z "$(sudo ls "$repositories/agentenv-computers/_manifests/tags")" ]; then
	echo "computers=false" >> "$GITHUB_OUTPUT"; exit 0
	fi
	key=$({ sudo ls "$repositories/agentenv-computers/_manifests/tags"
	ls "$RUNNER_TEMP/packages/agentenv-packages/locks" 2>/dev/null; } \| sort \| sha256sum \| cut -c1-16)
	echo "key=computers-${{ matrix.shard }}-${{ steps.week.outputs.week }}-$key" >> "$GITHUB_OUTPUT"
	echo "computers=true" >> "$GITHUB_OUTPUT"
	- name: Save the computer images and packages
	if: >-
	always() && github.event_name == 'push' && steps.tests.outcome == 'success'
	&& steps.kept.outputs.computers == 'true' && steps.kept.outputs.key != steps.computers.outputs.cache-matched-key
	uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
	with:
	path: \|
	${{ runner.temp }}/registry
	${{ runner.temp }}/packages
	key: ${{ steps.kept.outputs.key }}
	- name: Skips must be declared capability gaps
	if: always()
	run: \|
	.venv/bin/python .github/scripts/check_skip_policy.py junit-integration-slow.xml \
	--allow model_endpoint_configured remote_sandbox default_a2a_agent mcp_server_sources windows_media \
	omarchy_image
	# The tests' times, for .github/scripts/slow_test_durations.py to rebalance the shards with.
	- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	if: always()
	with:
	name: junit-integration-local-slow-${{ matrix.shard }}
	path: junit-integration-slow.xml
	retention-days: 14

	integration-local-gateway-shard:
	name: integration-local-gateway (${{ matrix.shard }}/5)
	strategy:
	fail-fast: false
	matrix:
	shard: [1, 2, 3, 4, 5]
	runs-on: ubuntu-latest
	timeout-minutes: 40
	steps:
	- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
	- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
	with:
	python-version: "3.12"
	- name: Install from uv.lock
	run: \|
	python -m pip install --quiet -r .github/ci-requirements.txt
	uv sync --locked --extra dev --python "$(command -v python)"
	echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
	- name: Pull the base images from mirror.gcr.io
	run: python3 .github/scripts/mirror_base_images.py src tst
	- name: Start the local registry
	run: \|
	docker run --detach --name registry --publish 5000:5000 mirror.gcr.io/library/registry:2
	timeout 30 sh -c 'until curl -sf http://localhost:5000/v2/ >/dev/null; do sleep 1; done'
	- name: Slow integration tier on the local defaults
	env:
	AGENTENV_TEST_SHARD: ${{ matrix.shard }}/5
	run: \|
	.venv/bin/python -m pytest -m int_test_slow -rA --timeout=1200 --junitxml=junit-integration-slow.xml \
	tst/integration/env/gateway/gateway_test.py
	- name: Skips must be declared capability gaps
	if: always()
	run: \|
	.venv/bin/python .github/scripts/check_skip_policy.py junit-integration-slow.xml \
	--allow model_endpoint_configured remote_sandbox default_a2a_agent mcp_server_sources
	- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	if: always()
	with:
	name: junit-integration-local-gateway-${{ matrix.shard }}
	path: junit-integration-slow.xml
	retention-days: 14

	# One check per tier under its own name (the main ruleset requires integration-local-slow): each passes
	# when every shard of its tier passed.
	integration-local-slow:
	if: always()
	needs: integration-local-slow-shard
	runs-on: ubuntu-latest
	timeout-minutes: 5
	steps:
	- name: Every shard passed
	run: test "${{ needs.integration-local-slow-shard.result }}" = success

	integration-local-gateway:
	if: always()
	needs: integration-local-gateway-shard
	runs-on: ubuntu-latest
	timeout-minutes: 5
	steps:
	- name: Every shard passed
	run: test "${{ needs.integration-local-gateway-shard.result }}" = success

	# Whether this change touches the code the installer tier covers; the tier costs minutes, so
	# it runs only then. A skipped job reports as passed to a required check.
	changes:
	runs-on: ubuntu-latest
	timeout-minutes: 5
	outputs:
	plugins: ${{ steps.diff.outputs.plugins }}
	steps:
	- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
	with:
	fetch-depth: 0
	- id: diff
	env:
	# A pull request's merge ref sits on the current base, which base.sha lags once the base
	# moves; diffing against base.sha would count the base's own later commits as changes.
	BASE: ${{ github.event_name == 'pull_request' && 'HEAD^1' \|\| github.event.before }}
	run: \|
	paths='^(src/agent_env/cli/(__init__\|__main__\|_installers\|_plugin_changes\|plugin)\.py\|src/agent_env/(plugins\|config)/\|src/agent_env/py\.typed\|packages/agentenv-protocol/\|tst/installer/\|tst/util/capabilities\.py\|pyproject\.toml\|uv\.lock\|Makefile\|\.github/(ci-requirements\.txt\|workflows/local-backends\.yml\|scripts/check_skip_policy\.py))'
	if ! changed=$(git diff --name-only "$BASE" HEAD 2>/dev/null); then
	echo "plugins=true" >> "$GITHUB_OUTPUT"
	elif grep -qE "$paths" <<< "$changed"; then
	echo "plugins=true" >> "$GITHUB_OUTPUT"
	else
	echo "plugins=false" >> "$GITHUB_OUTPUT"
	fi

	# `plugin add` and `remove` through the real pip, uv and pipx, offline against wheels built from
	# this checkout; the dependencies are downloaded from PyPI once, before the tests go offline. It runs on
	# two runners, split the same way as the unit tier.
	installer-shard:
	name: installer (${{ matrix.shard }}/2)
	needs: changes
	if: needs.changes.outputs.plugins == 'true'
	strategy:
	fail-fast: false
	matrix:
	shard: [1, 2]
	runs-on: ubuntu-latest
	timeout-minutes: 30
	steps:
	- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
	- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
	with:
	python-version: "3.12"
	- name: Install from uv.lock
	run: \|
	python -m pip install --quiet -r .github/ci-requirements.txt pipx
	uv sync --locked --extra dev --python "$(command -v python)"
	echo "$PWD/.venv/bin" >> "$GITHUB_PATH"
	- name: Installer tier
	env:
	AGENTENV_TEST_SHARD: ${{ matrix.shard }}/2
	run: .venv/bin/python -m pytest tst/installer -n auto --dist worksteal -rA --timeout=900 --junitxml=junit-installer.xml
	- name: Skips must be declared capability gaps
	if: always()
	run: .venv/bin/python .github/scripts/check_skip_policy.py junit-installer.xml
	- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	if: always()
	with:
	name: junit-installer-${{ matrix.shard }}
	path: junit-installer.xml
	retention-days: 14

	# Passes when both installer shards did, or when the change touches nothing the tier covers.
	installer:
	if: always()
	needs: [changes, installer-shard]
	runs-on: ubuntu-latest
	timeout-minutes: 5
	steps:
	- name: Every shard passed, or the tier had nothing to check
	run: \|
	test "${{ needs.changes.result }}" = success
	test "${{ needs.installer-shard.result }}" = success -o "${{ needs.installer-shard.result }}" = skipped

Navigation Menu

Repository navigation

ci(slow tier): keep the computer tests' resolved packages between modules and runs #3285

Workflow file

ci(slow tier): keep the computer tests' resolved packages between modules and runs #3285

Uh oh!

Workflow file for this run