[obvious-autobuild]

fleet-packet: burst10-bs-profile-evidence

Backend: run real CPU criteria on published capture artifacts; C2 report follows the default profile

What changed and why

Capability. The criteria stage's published-inputs seam (PR #279): two records for running the real CPU criteria stage over published capture artifacts, plus the tests that prove them. This packet adds the third record: the top-level C2 report in criteria_report.json now describes the configured default profile — status, agreement, positions compared and vetoes together — under HSS_CRITERIA_C2_PROFILES=all.

The defect (Greptile r4290120811, Codex r4290131173 — both verified against head c815b43). With HSS_CRITERIA_C2_PROFILES=all, the profile loop overwrote the top-level c2_report on every A/B/C/S iteration (run.py:410-415), so after the loop it described S, the last profile. The post-loop selection (run.py:527-529) picked the default profile's counts for c2.sweep.json but corrected only status on the report. A reader — including server/result/compose.py, which publishes report["c2"]["vetoes"] to the device — could see S's count and agreement under C's status. Report-level selection must not depend on iteration order.

The fix. The loop records each profile's full comparison evidence (status, agreement_with_c1, positions_compared, vetoes, n_pass, n_spots) into c2_by_profile and never touches the top-level report. After the loop, the report's whole C2 document is selected from the configured default profile (c2_default_document); a default profile with no C2 entry keeps the named off document with agreement_with_c1: None — another profile's numbers are never promoted, and c2.sweep.json's first-profile substitution fallback is removed (c2_sweep_document now reports off counts with the per-profile evidence kept in by_profile). Preserved unchanged: the explicit no-coverage / skipped / no-observed-ground downgrades, #255's agreement=None for zero compared positions, the published-input stage contract, and POST /v1/placements (criteria stays authoritative for device eligibility; no evaluator or threshold changed).

Alternatives considered.

1. Keep the loop's overwrites and re-merge every field from the default's sweep entry after the loop. Rejected: two sources of truth for the same numbers — the loop still publishes a last-profile document every reader must know to ignore, and each future field needs the same triple-write dance. This is the shape both reviewers flagged.
2. Move the per-profile evidence to a new top-level report.c2.by_profile and drop the flat fields. Rejected: changes the published report contract consumed by result/compose.py and the eval gates — a wider blast radius than the defect justifies.

The chosen fix has one source of truth (the per-profile record), preserves the report's shape, and makes the top-level document order-independent by construction.

Fixture helper and bridge diagnostic. Unchanged and still pinned: backend/tests/fixtures/cpu_downstream/scenes.py (supported_wall_variants, original_scene) and test_bridge_boundary_diagnostic_on_unchanged_generator_geometry (the placement bridge's unchanged refusal of the generator's geometry: keyframes, then /coverage/grid).

How I verified it

• Working clone of obv/burst9-bs-criteria-published-inputs at start SHA c815b43 on the repo sandbox cmp_l3e2i2Fu (fresh directory, no other thread's tree touched).
• Failing-before: with only the production diff stashed, the new regressions fail against the old behavior (report count/agreement describe S; the sweep document substitutes the first profile; new per-profile evidence keys are absent). Restoring the fix turns them green. Details in the log block below.
• Strict regression (real path): test_c2_report_evidence_follows_the_default_profile_not_the_iteration_order runs the real criteria stage — publication → Services.stage_context hydration → runner.run_stage(ctx, stubs=False) — over both committed synthetic variants (clean, hazard) with HSS_CRITERIA_C2_PROFILES=all, once as profiles A,B,C,S with default C, once reordered S,B,A,C with default S. It asserts every profile's evidence is kept, the report's whole C2 document equals the configured default's sweep entry (for C and for a non-C default), the same profile's evidence is byte-identical across both iteration orders, and C's and S's real evidence differs on both variants — so the selection cannot pass by coincidence.
• Named statuses (real path): test_c2_report_names_off_absent_default_skipped_and_no_coverage_without_fabricated_agreement covers C2 off, a default profile absent while A/S really ran C2 (no substitution), a budget-bounded sweep (every profile named skipped, C1 passes downgraded), and a scene that declares no coverage grid (no_coverage_grid, downgraded, agreement_with_c1: None) — each through the real stage with no fabricated agreement.
• Selection unit pins in server/criteria/tests/test_criteria.py: no substitution when the default is absent; the whole default comparison selected verbatim, including fix(backend): review-page links, packet still paths, C2 join agreement, and hydrated reads/scale #255's ok-with-zero-positions agreement_with_c1: None.
• Zero-comparison-ok is additionally pinned at the join level by fix(backend): review-page links, packet still paths, C2 join agreement, and hydrated reads/scale #255's existing test_join_without_matches_reports_no_comparison.
• Baseline identified against the starting head instead of hidden: the full backend suite's 5 failures are the known pre-existing set (SSR path hygiene + 4 expired SD-96 gates), matching the failures recorded on c815b43. Never skipped or extended.
• mypy reported honestly against the repo's existing non-clean baseline (numbers below).
• make lint, make schemas, full backend pytest, and the root solver suite (make server, pinned uv) — results below.

Verification results (fix commit 89d7de6d = 89d7de6, author Obvious <obvious@obvious.ai>, Co-authored-by: Sam Gu trailer; branch head 563855d1 = merge of the advanced base obv/burst7-bs-capture-placement-runner, whose positions_compared additions to the old-style top-level documents are subsumed by the per-profile records — conflict resolved to the default-profile selection):

Gate	Result
Failing-before (start SHA c815b43f, fix stashed, final tests)	4 failed / 2 passed in 140.42s — both real-stage regressions and both unit pins fail; both preserved tests pass (log below)
Focused suite at fix head	11 passed in 239.62s
Full backend pytest at fix head	5 failed / 1279 passed / 17 skipped in 485.84s — the 5 failures are exactly the documented pre-existing baseline (test_ssr_overlap_check.py + 4 x test_gates.py SD-96), identical on the starting head
ruff check / ruff format --check / lint-imports	All checks passed / 479 files already formatted / Contracts: 5 kept, 0 broken
make schemas	passed (export_schemas --check)
mypy server/criteria/run.py	33 errors in 8 files at the fix head and identically 33 errors in 8 files with the production change stashed — no new type errors
Root solver suite make server (pinned uv)	ruff clean, 312 passed in 68.20s

Failing-before detail (old code, final tests): the strict regression failed on the substituted report; the named-statuses regression first failed at the absent-default assert expecting off/None/0 and receiving ok/1.0/12; test_c2_sweep_document_never_substitutes_another_profile failed assert doc["status"] == "off" receiving 'ok'; test_c2_default_document_selects_the_whole_default_comparison failed on ImportError because c2_default_document did not exist pre-fix; test_c2_sweep_counts_the_default_profile and #255's test_join_without_matches_reports_no_comparison passed untouched.

Test logs (inline)

$ uv run --frozen --extra api pytest -q tests/test_criteria_published_inputs.py     server/criteria/tests/test_criteria.py::test_c2_sweep_counts_the_default_profile     server/criteria/tests/test_criteria.py::test_c2_sweep_document_never_substitutes_another_profile     server/criteria/tests/test_criteria.py::test_c2_default_document_selects_the_whole_default_comparison     server/criteria/tests/test_criteria.py::test_join_without_matches_reports_no_comparison

# --- with the production fix stashed (old code, start SHA c815b43f) ---
FAILED tests/test_criteria_published_inputs.py::test_c2_report_evidence_follows_the_default_profile_not_the_iteration_order
FAILED tests/test_criteria_published_inputs.py::test_c2_report_names_off_absent_default_skipped_and_no_coverage_without_fabricated_agreement
FAILED server/criteria/tests/test_criteria.py::test_c2_sweep_document_never_substitutes_another_profile
FAILED server/criteria/tests/test_criteria.py::test_c2_default_document_selects_the_whole_default_comparison
4 failed, 2 passed in 140.42s (0:02:20)

# --- with the fix restored, formatted tree ---
11 passed in 239.62s (0:03:59)

$ uv run --frozen --extra api mypy server/criteria/run.py
# fix head:      Found 33 errors in 8 files (checked 1 source file)
# fix stashed:   Found 33 errors in 8 files (checked 1 source file)  <- identical baseline

$ uv run --frozen --extra api pytest -q          # full backend suite, fix head
5 failed, 1279 passed, 17 skipped, 7 warnings in 485.84s (0:08:05)
FAILED tests/test_ssr_overlap_check.py::test_repo_has_no_ssr_shingle_overlap          <- pre-existing
FAILED server/eval/tests/test_gates.py::test_committed_fixtures_are_scored_against_their_labels   <- pre-existing
FAILED server/eval/tests/test_gates.py::test_offline_table_skips_the_engine_and_passes_the_unpatched_reference  <- pre-existing
FAILED server/eval/tests/test_gates.py::test_route_over_the_bar_is_known_until_expiry  <- pre-existing
FAILED server/eval/tests/test_gates.py::test_refresh_rewrites_pins_after_a_committed_edit  <- pre-existing

$ ruff check . / ruff format --check . / lint-imports
All checks passed! / 479 files already formatted / Contracts: 5 kept, 0 broken.

$ make schemas
uv run --frozen python -m server.contracts.export_schemas --check   # exit 0

$ make server
cd server && uv run ruff check .         All checks passed!
cd server && uv run ruff format --check .  29 files already formatted
cd server && uv run pytest -q            312 passed, 1 warning in 68.20s

What is not verified

• iOS, Swift, server/ (root placement service source) and deployment files are untouched by this packet and not exercised here.
• A real-path zero-comparison-ok report (ok status with 0 compared positions on the default profile) is not producible from the two committed variants without authoring new fixture geometry — it is pinned at the join level (fix(backend): review-page links, packet still paths, C2 join agreement, and hydrated reads/scale #255 regression, real code) and at the report-selection level (unit + the real-path invariant assert), not by an end-to-end zero-position stage run.
• mypy is not clean on this repo's baseline; the numbers below are the honest comparison, not a green claim.
• No cloud, paid calls, real user data, dataset/weight downloads or GPU were used; prewarm and webhook delivery are off in the tests by construction.

Assumptions

• "Unavailable default-profile" means the default profile has no C2 entry (C2 disabled entirely, or the default profile is not among the configured profiles); its named status is the pre-existing off, and the report carries no agreement rather than another profile's.
• The sweep document's by_profile entries gaining comparison keys (agreement_with_c1, positions_compared, vetoes) is additive: nothing consumes c2.sweep.json beyond the published-inputs tests, and compose.py reads only report["c2"]["vetoes"], whose semantics now correctly describe the default profile.
• Running the stage with a default profile outside HSS_CRITERIA_PROFILES is a misconfiguration worth reporting honestly (off, no substitution) rather than one to mask with another profile's numbers.
• Work done on the shared cmp_l3e2i2Fu sandbox in the fresh clone ~/work/burst10-bs-profile-evidence; no branches checked out in other directories' trees; ios: fix review debt on the walk, review and AR result screens #241/chore(backend): port house-scanning-server into the monorepo #247 untouched; no merge, no default-branch push, no force-push, no auto-merge.

Human author: Sam Gu (sgu07966@gmail.com)

🔗 Obvious Project · 🧵 Obvious Thread · 🔀 Obvious PR Artifact

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: f1785951-d452-4f3d-aec8-7f4b3e6b5e13

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-16T02:02:21.221808Z	c815b43	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[obvious-autobuild]

Verification record (head c815b43f)

• Focused: backend/tests/test_criteria_published_inputs.py — 5 passed (~74 s) through the real path (runner.run_stage(ctx, stubs=False), stubs off).
• Existing suites: server/criteria/tests + server/orchestrator/tests — 281 passed, 1 skipped.
• Full backend suite: 5 failed / 1275 passed / 17 skipped in 6 m 55 s. The failure set is exactly the known pre-existing baseline — tests/test_ssr_overlap_check.py::test_repo_has_no_ssr_shingle_overlap (SSR hygiene) plus the four expired SD-96 gates in server/eval/tests/test_gates.py. No new failures.
• Root solver suite (make scoring, pinned uv 0.11.15 per experiments/scoring/pyproject.toml): 372 passed; ruff check and ruff format --check clean.
• make lint — ruff check, ruff format --check (479 files), lint-imports (5 contracts kept, 0 broken). make schemas — green.
• PR state: MERGEABLE against obv/burst7-bs-capture-placement-runner (base advanced to c255a235, [obvious-fleet] Merge origin/alpha into the capture-API placement-case replay runner (resolve stale early-#271 conflicts) #274's head, after this branch was cut — normal stacking; this PR merges after [obvious-fleet] Merge origin/alpha into the capture-API placement-case replay runner (resolve stale early-#271 conflicts) #274).

Interface blocker check from the task: none — the criteria stage consumed the published inputs end to end (hydration, input validation, C1, C2, C3, verdict publication all ran against the published bytes).

[codemetrics-ai]

Note

Review skipped

Codemetrics is installed on this repository, but it hasn't been added to your team yet.

To enable AI code reviews for this repo, go to your dashboard and add BaseScanning to your team's repositories.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; the previous report-selection issue is fixed, and no new actionable issues were found.

Summary

The PR adds tests that run real CPU criteria over published capture artifacts and fixes C2 report selection.

• The C2 report now follows the configured default profile.
• Criteria tests now run from published capture artifacts.

Previous finding #1 is fully addressed. No new actionable issues were found.

_{Reviews (2) · Last reviewed commit: "Merge branch 'obv/burst7-bs-capture-plac..." · Reviewed by Greptile}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c815b43fb5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Report C2 comparisons for the default profile

When HSS_CRITERIA_C2_PROFILES=all, this assignment overwrites the top-level c2_report on every profile iteration, so after the A,B,C,S loop positions_compared (and agreement_with_c1) describe profile S rather than the configured default profile C. The later c2_sweep_document only corrects the top-level status/count fields, leaving this newly exposed comparison count inconsistent with the report's default-profile semantics whenever profiles produce different matches.

Useful? React with 👍 / 👎.