Skip to content

feat(ci): measured local overhead harness, budgets and regression workflow (P7-CI-46) - #282

Open
acamarata wants to merge 19 commits into
mainfrom
p7/P7-CI-46
Open

acamarata wants to merge 19 commits into
mainfrom
p7/P7-CI-46

Conversation

@acamarata

Copy link
Copy Markdown
Contributor

What

free/ci/bench: Go benchmarks plus the cibench process harness that times the V15 engine against the frozen v1.4 gate (startup, detect, plan, store open, executor overhead, end-to-end on a tiny Go and a medium Go plus Node fixture, service start behind -services). It emits perfbench/v1 result JSON that the cli's own perfbench check reads. Budgets: perf-budget.json (linux/amd64) and perf-budget-darwin-arm64.json, each the worst p95 of three 20-sample runs plus 25 percent. ci-bench.yml runs on PRs that touch free/ci/: harness tests, measure, perfbench check from a pinned cli checkout, the relative budget, and controls (a result ten times over budget must fail both checks).

Result you need to read first

The relative budget (V15 tiny median <= v1.4 tiny median x 1.10 + 250 ms) is missed on both platforms: linux/amd64 about 1.3 s against a 0.59 s limit, macos-arm64 about 1.5 s. Cause (measured, not guessed): the exec funnel forks ps -A three times per tracker snapshot for every funnel call, and a bare run makes about 22 funnel git calls; strace counts 135 ps execs in one tiny run. A miss fails the Ticket unless the owner accepts a written exception, so ci-bench.yml fails its ratio step until the engine is fixed or an exception is committed in bench/ratio-exceptions.json (an entry without accepted_by does not count). Evidence and the exception request: hq plan/p7/evidence/CI-46-perf.md.

Also found: on Linux the engine runs every job under ulimit -v from a 256 MB default demand, which kills go and node at start. The fixtures set requires.mem_mb to work around it.

Scope notes

  • Three files outside the Ticket write_scope are touched by lead ruling (same pattern as P7-CI-32): architecture/layers.yaml (rows for bench and bench/cmd/cibench), architecture/layers_test.go (os/exec exemption for bench/spawn.go only), internal/nodes/provision/single_exec_test.go (bench/spawn.go:spawn). The timing tool must not run inside the engine funnel whose overhead it measures.
  • perfbench check defaults to -min-n 20; the Ticket's N=10 needs -min-n 10 on that verify line. The workflow uses 20 runs and the default.

@acamarata
acamarata force-pushed the p7/P7-CI-46 branch 2 times, most recently from 525700e to 8b121a8 Compare October 19, 2026 04:06
@acamarata

Copy link
Copy Markdown
Contributor Author

Parked ready, not merged. Required check passes; independent Sonnet review (clean checkout) found no must-fix and its should-fix items are in. ci-bench is red by design: the relative budget is missed (V15 tiny median 4 to 10 times v1.4 on linux/amd64 and darwin/arm64; cause D-1008, evidence hq plan/p7/evidence/CI-46-perf.md) and a missed budget fails the Ticket unless the owner accepts a written exception. Merging now would turn ci-bench red on every free/ci PR. Next step: an accepted entry in free/ci/bench/ratio-exceptions.json (or the D-1008 fix), then merge. gitleaks is red on main too (e23f267).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant