feat(results): split mix test partitions by recorded duration (T24)

[calvin-archastro]

Merge order: independent of the P4 quarantine PR. After merge: release aster, deploy the new aster-srv image, bump firstlanding's ASTER_VERSION. Shards keep Mix's split until the server answers.

What (T24)

Splits partitioned mix test suites by recorded time rather than by file count.

1. Server: GET /v1/test-results/durations?target=…&day=YYYY-MM-DD returns each test file's median duration.
   • Per run, a file's time is the sum of its tests' first-attempt durations.
   • The median is taken over the latest 20 complete, uncached runs of each named target.
   • Only runs received in the 7 days before day began count, less 15 minutes for uploads still committing.
2. Client: a MIX_TEST_PARTITION=K mix test --partitions N target, including the firstlanding bash -c 'cd ../platform && …' shards, runs an explicit file list instead.
   • It takes the same file set Mix would use and splits it longest-first. A file with no recorded duration counts as the median.
   • --partitions is dropped. MIX_TEST_PARTITION stays set, because projects name their per-partition databases after it.
   • It prints one line saying which split it used:

     [aster] //services/platform-test-shard-2:test-ci: partition 2/4 balanced by duration: 97 of 412 files, ~212s of a slowest ~214s (398 files timed, medians before 2026-10-15)
     

3. Lanes: unchanged. They are configured selectors, not a split. The spec's T24 entry now says this.

Determinism: a file must run on exactly one shard

Every shard computes its split independently, so every shard must work from the same inputs.

• File set: comes from the checkout.
• Durations: depend only on (targets, the commit's UTC day).
  • The server refuses a day that has not begun (409), so it never answers from a window that is still filling.
  • Each answer is cached, per replica and in .aster/durations/.
• Ties: broken by path, then by partition index.
• Unknowable file sets: if Mix's file set can't be known for sure, the command runs unchanged. That covers custom test_paths/test_pattern/load filters, umbrella apps, file:line, --failed/--stale, and symlinks under test/.
• Fallbacks: without ASTER_SRV_URL, with no history, after an error, or with ASTER_BALANCE_PARTITIONS=off, the command runs byte-for-byte as written.

Known risk: a partial outage. If one shard can't reach the server after 3 tries (10s timeout each) while the others can, that shard falls back to Mix's split. Some files would then run twice and some not at all. Its log line says so ("split by Mix (durations unavailable: …)"). A failed fetch can't fail the build safely, and checking across shards would need coordination between jobs that this PR does not add.

Test identity change (deliberate, one-time)

Before this PR, a test's id hashed in the shard project's address, e.g. //services/platform-test-shard-1. Two things break that:

• Mix deals the sorted file list out round-robin (see Mix's own docs), so adding or removing one test file already moves files between shards and changes their test ids.
• A split by duration would move files more often.

Shard tests are now identified by the project that owns the suite: the directory the one-liner cds to, e.g. //services/platform. The project is derived from the target's configured command, so overrides never change it.

• target on each result is still the shard. Only test_id changes.
• Effect on existing history: platform and core shard tests get new ids at rollout. Their 30-day history, flake scores and any quarantine entry stay under the old ids and age out. The new ids start fresh, and from then on are the same on every shard.

Measured vs simulated

• Simulated only. I can't read production durations without new credentials, so I couldn't replay per-file times.
• Measured from firstlanding CI logs: the four partitions' target times from Aster's PASS //…-shard-N:test-ci (…) lines, taken from runs where all four ran.

Suite	Runs	Avg slowest partition	Avg mean partition	Headroom if perfectly balanced
Platform (4)	18	264s	214s	50s (19%)
Core (4)	7	379s	303s	76s (20%)

That headroom is an upper bound: run-to-run noise and each partition's fixed boot time stay. The real drop gets measured after rollout from the same log lines.

Query cost on a seeded copy with 14M result rows (4 shard targets of 3,000 tests, 20 runs each): 1–2.5 s per answer. Each answer is then cached.

Verification

• Unit tests:
  • Parsing and rewriting partitioned commands.
  • Listing the file set, and refusing when it can't be known.
  • The split: longest first, median for unknown files, stable ties, every partition non-empty.
  • Retry rerunning only the failing files of a balanced command.
  • ExUnit formatter capture on a balanced command.
  • Owner identity in the results stream.
• End to end: two shards of one workspace against a stub server.
  • Disjoint shares that together cover the suite.
  • One request, with the second shard served from the cache.
  • The commit's UTC day sent in the request.
• Postgres (crates/aster-srv/tests/durations.rs): medians across shards; cached, partial and other-target runs left out; window edges; the 20-latest-runs cap; 409/403/401.
• Local checks: cargo fmt, clippy -D warnings (all features), cargo doc with -D warnings, and the workspace tests with Postgres all pass. The one failure is tests/dev_services.rs, a known local flake.

🤖 Generated with Claude Code