perf(executor): start each target as soon as its dependencies finish

[calvin-archastro]

Review on ArchCode

Problem and author intent

aster ran targets one DAG level at a time (compute_target_levels gives level = 1 + max dep level; the loop in execute_target_set waited for every target in a level before starting the next). A target whose dependencies were done still waited for unrelated slower targets at lower levels, so lane wall time was the sum of per-level maxima, not the critical path.

Seen in firstlanding CI on 2026-10-15 (Aster 0.19.0):

• Go lane (job 116578614010): services/rust/archdev:build (level 1, 3m13s) held back :test-ci, cli-foundations:build, developer-platform-sdk:build, which started the second it ended. cross-project-reads-1 (6m24s) only started after devboxctl:test-ci (3m54s) at a lower level finished, although it depended on neither.
• CLI e2e lanes (jobs 116595128019, 116595048380): TS :build targets, ensure-node-pty and web:build ran only after build-rust-archdev (5m47s) although nothing depended on it.

Goal: start every target as soon as its dependencies have finished, with no change to exclusive resources, failure semantics, output or caching.

What changed (backend-only, src/executor/runner.rs)

Before: levels = compute_target_levels(..); per level: block targets with failed deps, spawn the rest, drain the channel, join.
After: ReadyQueue tracks unmet in-run dependencies per target. The loop launches everything ready, then blocks on the result channel (no polling); each completion releases dependents.

Property	Before	After
Start condition	whole previous level done	all in-run deps done
Launch order among ready targets	arbitrary (hash set)	(level, address), deterministic
exclusive_resources	one mutex per resource, built per level	one mutex per resource for the whole run, so targets at different levels that share a resource still serialize (new case: they can now overlap in time)
Failed dep	dependents get Blocked: prerequisite target(s) failed: .. once their level starts	same text, emitted once all their deps finished; cascades transitively; independent branches keep running (no fail-fast, unchanged)
Result order	by level, blocked targets first, completion order inside	same (stable sort by level then blocked-first)
Cycle	remaining targets dumped into one level	when nothing runs and nothing is ready, remaining targets are launched together (same outcome)
Shutdown	stop starting levels	stop launching, drain running targets (their threads report Cancelled)
Thread panic	result silently lost	reported as a failed result (needed because completions are now counted)

Unchanged: caching, --json schema, logs, progress output, test capture/retry/quarantine, partition balancing. compute_target_levels stays and is still used for dry-run plans (aster affected --dry-run), result grouping and tie-breaking. There is no existing jobs/parallelism limit in the executor (every ready target already got its own thread); I did not add one. Peak concurrency can now be higher than before because ready targets from different levels overlap.

Testing

Canonical end-to-end proof: dependent_starts_while_unrelated_lower_level_target_still_runs in src/executor/runner.rs. It drives Executor::execute_targets with real sh child processes: slow target A waits (cap 5s) for a marker file that only C writes, where C depends on fast B. A passes only if C started while A was still running; under the level barrier A times out and fails. No wall-clock threshold.

Other new tests (same module; run by CI's cargo test --locked --all-targets --all-features):

• failure_blocks_transitive_dependents_but_not_independent_targets
• results_are_grouped_by_level_with_blocked_targets_first
• dependency_cycle_still_runs_every_target
• exclusive_resource_sections_never_interleave_across_levels: holders at levels 0/1/2 append start/end events to a log; first holds its section until second (ready via a fast gate) is runnable, so contention is guaranteed. Asserts every target ran and the log reads strict start/end pairs.
• without_the_shared_resource_the_same_sections_do_overlap: positive control, same graph without the resource; first waits for second to start, so the log must interleave.
• targets_holding_several_resources_do_not_deadlock_or_overlap: targets with [res_b,res_a], [res_a,res_b], subsets and a dependent; the run ends and sections of targets sharing a resource never overlap.
• panicking_target_is_reported_blocks_dependents_and_frees_its_resource: injected panic (#[cfg(test)] hook) gives a failed "panicked" result, blocks its dependent, releases its resource, run terminates.
• cache_hit_and_skipped_target_each_release_their_dependents.
• tests/integration.rs::terminating_aster_mid_run_launches_nothing_more_and_exits: SIGTERM mid-run; the dependent never launches, running target drains, exit 143, no hang.
• different_exclusive_resources_run_at_the_same_time (rendezvous via marker files)
• ready_queue_releases_targets_as_dependencies_complete

Replaced the two timing-based execute_target_level resource tests with the marker-based ones above. Local: cargo fmt --check, clippy -D warnings, cargo doc -D warnings, full cargo test --all-targets --all-features pass except tests/dev_services.rs::dynamic_port_bundles_are_distinct_propagated_and_released, which fails identically on unmodified origin/main on this machine (port/environment, unrelated to the executor).

Simulated savings (simulation, not a measured run)

Method: per-target START/PASS timestamps from firstlanding CI logs (run 38942230866 jobs, plus the 2026-10-15 Go and CLI e2e jobs above), dependency edges from aster --json graph on current firstlanding main, unlimited parallelism (as Aster has today). "Level model" replays the old barrier and matches the logged wall time within ~1 min on every lane (validation); "ready model" is the critical path. Ignores exclusive_resources contention and slowdown from extra concurrent load, so real savings will be somewhat lower. Lanes with fewer than 4 targets in the main invocation are omitted.

lane job	targets	actual (s)	level model (s)	ready model (s)	saving (s)	saving %
Test (Source Elixir Core 1)	9	177	177	177	0	0%
Test (JavaScript)	5	126	126	126	0	0%
Test (Services Platform)	66	623	622	479	143	23%
Test (Source Elixir Core 3)	12	238	238	236	1	0%
Test (Other Elixir)	15	652	652	648	4	1%
Test (Services Platform 3)	62	562	562	550	12	2%
Test (Services Platform 2)	61	570	569	549	20	4%
Test (Services Platform 4)	61	524	524	508	16	3%
Test (JavaScript Browser)	33	113	80	76	3	4%
Test (ArchDev Platform Reads 2)	44	464	424	416	8	2%
Test (Go) [run 2026-10-15]	58	958	940	486	454	48%
Test (archdev CLI e2e Rust) A [2026-10-15]	39	578	577	533	44	8%
Test (archdev CLI e2e Rust) B [2026-10-15]	39	598	597	548	49	8%

Risk: low-medium

• Touches the core scheduler, but each target's launch code is the same function body as before (moved, not rewritten).
• Higher peak concurrency on small CI runners (CPU/memory); no limit exists today to cap it. If a lane gets slower or OOMs, a jobs limit is the follow-up.
• Targets that depend on each other only implicitly (no depends_on) were previously separated by accident of level ordering and may now overlap. Such hidden ordering would show up as new flakes.

User impact

Faster runs on graphs with uneven level durations; no CLI/config change. Progress lines (START/PASS) for later-level targets can now appear before earlier-level targets finish.

Rollout caveat: cache decision moves earlier

The cache-hit check now runs when a target becomes ready, which can be earlier than before (previously after every target of the lower levels had finished). A generator that writes into a target's inputs without being declared as one of its dependencies could therefore be missed: the target can get a stale cache hit where the level barrier used to give the generator time to finish first. Declare such generators in depends_on before bumping Aster.

Follow-ups

• firstlanding: bump ASTER_VERSION in .github/workflows/ci.yml after a release (not part of this PR).
• Optional: a --jobs limit and longest-duration-first ordering using recorded durations.

🤖 Generated with Claude Code