[Task][RFC]: Qualify the integrated benchmark study upload and dashboard contract #5203

Activity

1. huangruiteng

   added

   help wantedExtra attention is neededExtra attention is needed

   direction/architecture-evolutionArchitecture evolution and research-incubator work.Architecture evolution and research-incubator work.

   on Sep 27, 2026
2. karenchuu commented on Sep 28, 2026

   

   karenchuu

   on Sep 28, 2026

   Contributor

   More actions

   Claiming the criterion-4 slice of §7 (the PR 2: Read-only dashboard row in §6) on this task.

   Measured on main at 6643f3670, source/PR inspection only — no new test run yet:

   • apps/presentation/dashboard/public/benchmark-study.example.json:32 shows "runtime_classification_counts": { "running_qualified": 2, "terminal_qualified": 4 }, but terminal_qualified is not a member of BenchmarkRuntimeClassification (loopx/capabilities/benchmark_toolkit/runtime_observation.py:26-36), and the projection builds those keys directly from observed classifications (loopx/capabilities/benchmark_toolkit/study_projection.py:1304-1311).
   • The same fixture claims "factorial_contrast_count": 1 at :29-30 while "factorial_contrasts": [] at :88, although the reducer defines that count as len(board["factorial_contrasts"]) (study_projection.py:1341).
   • grep over tests/ and examples/ finds no case that uploads record_kind="runtime_observation" through build_benchmark_upload_envelope, so one of the five kinds in study_projection.py:47-53 never reaches the projection the page renders from (apps/presentation/dashboard/src/views/benchmark-study-page.tsx:154-156, default source ${BASE_URL}benchmark-study.example.json at :293).

   What I intend to change, exact files:

   1. tests/capabilities/test_benchmark_study_projection.py — one new case that uploads a manifest, a board row, a case insight, a runtime observation and a behavior finding through the existing envelope/simulate path, then asserts the campaign block's classification keys are real BenchmarkRuntimeClassification values and that factorial_contrast_count == len(factorial_contrasts).
   2. apps/presentation/dashboard/public/benchmark-study.example.json — regenerate from that reducer output so every number the default dashboard shows is derivable, plus the matching apps/presentation/dashboard/smoke/benchmark-study-contract-smoke.ts expectation and the loopx/web/chat/bundle-manifest.json digest if the smoke requires it.

   No schema change, no scoring decision, no transport. One open question I will raise instead of deciding for you: if the current fixture values are intentional illustrative data rather than a defect, say so and I will stop before replacing them; my default is to make the shipped default payload consistent with what the reducer can emit.
3. karenchuu commented on Sep 28, 2026

   

   karenchuu

   on Sep 28, 2026

   Last edited by karenchuu

   Contributor

   More actions

   Acceptance mapping done against 6643f3670, and one slice delivered: PR #5255 (test(benchmark): pin the runtime-observation path into the study campaign block).

   Per §7, item by item — what I could point at in the shipped tree:

   §7 item	Evidence on main today
   one manifest for simple and factorized studies	study_projection.py:194-262 types arms as factor assignments; test_benchmark_study_projection.py:305-310 normalizes 2-arm, 4-arm and 5-arm designs
   board rows round-trip without losing meaning	test_benchmark_study_projection.py:326-335 asserts envelope output equals the row exactly
   reducers expose campaign/arm/case/run with denominators, no new scoring	study_projection.py:1329-1360, authority block :1361-1371; tests at :639, :870-883
   dashboard renders native partial/binary/guardrail/effort/runtime-health, no benchmark-specific core fields	page renders roles at benchmark-study-page.tsx:107-109, :145-158, :177-186; no benchmark names inside the reducer. Coverage was missing for the runtime-health half, and the default payload has the defect below
   invalid identity / unknown record kind / silent terminal rewrite / forbidden evidence fail closed	study_projection.py:502-506, :579-583, :694-709; :497-498 with :478; :737-748 via preview_benchmark_experiment_board_upsert and experiment_board.py:24-30; :424 with :66-70, :382-402 — tests at :338-356, :395-437, :460-501, :504-515, :518-587
   external transport optional, provider-owned, separately authorized	study_projection.py:770-776 (execute: bool = False) and the receipt marks external_write_performed: false; benchmark_study.py:108-116 needs explicit --store and --execute; no network client in the toolkit

   Two things came out of the mapping. I deliberately did not decide the second one:

   1. Delivered in test(benchmark): pin the runtime-observation path into the study campaign block #5255. No case on main uploaded record_kind="runtime_observation", although study_projection.py:47-53 accepts it and :1304-1311 derives the campaign's runtime-health counts from it. The new case uploads three observations through the same envelope builder the CLI uses and pins the derived counts. Three mutations of the product — dropping that record kind, hardcoding a contrast count, dropping the classification aggregation — all leave the existing 20 cases green and all fail the new one.

   2. Yours to call. apps/presentation/dashboard/public/benchmark-study.example.json, which is the dashboard's default payload, carries two values this reducer cannot produce:

      • :32 lists "terminal_qualified": 4, and terminal_qualified is not a member of BenchmarkRuntimeClassification (runtime_observation.py:30-36);
      • :29 claims factorial_contrast_count: 1 while :88 is "factorial_contrasts": [], although the count is defined as len(board["factorial_contrasts"]) (study_projection.py:1341).

      Repairing it changes user-visible numbers on the study page and needs a re-hashed entry in loopx/web/chat/bundle-manifest.json, and GH-C101 keeps the dashboard surface with you, so I am asking instead of submitting: are those intentional illustrative values, or should I send the regenerated payload as a follow-up?

   One note for the fourth checkbox: the RFC has no delivery-checkpoint section to reconcile — its status line is a bare Accepted, and there is no docs/architecture/rfcs/ledger/benchmark-study-upload-dashboard-v0/ directory — so the table above answers it directly rather than through a doc edit.