Injection quality: run a fresh calibrated eval — committed results predate all 14 quality features #95

Activity

1. scbrown commented on Sep 16, 2026

   

   scbrown

   on Sep 16, 2026

   CollaboratorAuthor

   More actions

   Both harness blockers under this issue are now FIXED and merged

   Attempting this eval surfaced two defects that would have made a fresh study worthless. Both are on
   main:

   fix	commit	what it was
   #98	5fd5d41	the harness moved the shared ~/.claude/settings.json aside host-wide for every agent run
   #99	16300d3	a run with zero tests executed scored PASS (passed = exit_code == 0, and cargo output parsed as pytest)

   The second one matters directly here: any study run before 16300d3 produced a pass-rate that
   could not fail on a task whose test filter missed. That includes the archived results this issue
   exists to replace, so they should be re-checked against their stored outputs before being used as a
   comparison baseline.

   Measured inputs for the scale decision

   A partial run was made before it was stopped — 2 runs of the 8 planned, task ruff-001, one attempt
   per arm:

   no-bobbin     $1.0573   294.6s   48 turns   2,086,969 in    9,644 out
   with-bobbin   $0.8382   262.8s   37 turns   1,553,471 in    8,200 out
   

   Both produced a real, semantically-correct fix. Notes on reading these:

   • Cost and token figures are sound — they come from the agent result, not the scorer.
   • The PASS verdicts from that run are NOT — they were produced by the pre-16300d3 scorer, i.e.
     they are the vacuous signal itself. Do not cite "2/2 passed" from it.
   • with-bobbin was cheaper, not dearer — 21% less, 11 fewer turns. The intuition that injection
     costs more because it enlarges the prompt is backwards: every turn resends accumulated history, so
     turn count dominates prompt size, and injection buys fewer exploration turns than it adds context.
   • Do not extrapolate the full study cost from this. It is N=1 per arm on a single task.

   For reference, the full study as scripted is 96 agent runs — 12 no-bobbin + 12 with-bobbin + 72
   ablations (4 tasks × 6 variants × 3 attempts), each with a $5.00 per-run cap.

   One thing to check before running it

   run-baseline-study.sh passes --force-budget, which disables the harness's own pre-flight check.
   That check is a token-expiry guard (it aborts only if the OAuth token has under an hour left) —
   it is not a spend cap, and nothing in the harness caps spend. Worth knowing before choosing a
   scale, since the flag's name suggests otherwise.

   Comparability caveat

   flask-001..005 are now in eval/tasks/_quarantined/, and the archived ablation-report-final.md
   lists flask tasks among its data. Old-vs-new comparisons must be restricted to the non-quarantined
   set.
2. scbrown commented on Sep 16, 2026

   

   scbrown

   on Sep 16, 2026

   CollaboratorAuthor

   More actions

   Adding a concrete instance to the archived-baseline caveat already noted here, because it is more
   specific than "needs re-checking" and it names the artifact a reader would actually cite.

   eval/results/ablation-report-final.md (committed 2026-03-02) carries a Pass% column, and its
   targets include cargo-built ones — ruff is the most-referenced target in the file and is itself a
   Rust project, with cargo appearing throughout. That report predates the zero-tests fix by roughly
   six months.

   The issue is not that those numbers are known wrong. It is that nothing in the report
   distinguishes a real pass from a vacuous one, because the signal that produced the column was
   later found unable to fail for runs whose output parsed as a suite that never executed. A file named
   -final with a Pass% column is precisely the artifact that gets cited later without anyone
   re-deriving it.

   Same applies to the other committed results carrying cargo/Rust targets:
   baseline-study-20260302-163649.log, ablation-study-20260302-170130.log,
   interim-ablation-report.md, fresh-report.md.

   Suggested handling, in preference order:

   1. Do not re-run them to "confirm" — a re-run on the fixed harness produces a new number, which
      is a different study, not a validation of the old one.
   2. Annotate the artifacts in place with the date of the fix and one line saying the Pass% column
      was produced by a signal since shown unable to fail. That is cheap, survives, and reaches the
      reader who finds the file rather than this thread.
   3. Treat F1 separately from Pass% — F1 does not come from the exit-code path and is not implicated
      by the same defect, so blanket-invalidating the whole report would discard sound numbers.

   Flagging the asymmetry that makes this worth doing rather than noting: an over-cautious annotation
   costs a line of text, while an uncaveated Pass% in a file named final costs whoever builds on it.
3. scbrown commented on Oct 12, 2026

   

   scbrown

   on Oct 12, 2026

   CollaboratorAuthor

   More actions

   Current-source triage at main 94bfc9353d9d5f7a256e673c489aa691c3155b6a: the next evaluation path is now the v2 harness rather than rerunning the archived v1 study unchanged.

   • runner.cli l0 runs offline retrieval with no agent or LLM calls. It invokes the production injection hook, supports all ablation/baseline arms, and writes per-arm JSONL results.
   • The preregistration fixes gold pre-image hunks, paired comparisons, and primary hunk recall/density at a 300-line budget. L0 is approved; L1 is a paired 2×2 pilot, with a separate decision required before a full study.
   • The L0 environment strips inherited BOBBIN_* settings and uses an empty XDG configuration so host configuration cannot redirect retrieval to a shared server.
   • A new GPU pilot/continuation also needs the reviewed runtime library lock; historical archive-only pins are not accepted as proof of the extracted runtime.

   The concrete next step is a frozen current-source L0 run and ablations with task/config/binary provenance, followed by the preregistered analysis. Preserve the earlier warning about archived v1 Pass%: a rerun is a new study, not validation of the old scorer's verdicts. This source check provides no new evaluation result and does not clear the evidence gate for #96 or #97.
4. scbrown commented on Oct 13, 2026

   

   scbrown

   on Oct 13, 2026

   CollaboratorAuthor

   More actions

   A bounded v2 L0 run is complete for go-001: one task, 11 arms, 300-line budget, not the full 40-task study. This used the production injection path with no agent or LLM calls and isolated retrieval configuration.

   Provenance: harness commit 94bfc9353d9d5f7a256e673c489aa691c3155b6a, Bobbin 0.27.3, and the parent of Go fixing commit 6aef900af4eeb4a35d3ff1237a41cf3f63c56c24.

   All 11 arms produced context; none returned an error or skip. Gold coverage for this task is one file and one pre-image line, src/runtime/metrics/sample.go:45.

   Result	Observation
   Hunk recall and density	0 for every arm
   File recall	BM25: 1; every other arm: 0
   Full pipeline	32 injected lines
   No-semantic arm	106 injected lines
   Other baselines	rg/BM25/repomap: 300 lines; embed: 154 lines

   A scoring positive control covering the exact gold span returns file recall, hunk recall and density of 1. That is an instrument control, not a retrieval result.

   Both index commands completed successfully, but the last progress count was 10164/10165 files. Complete file coverage is therefore not claimed. The guarded run reported no changes to the watched user configuration or process resources.

   This is one task's observed miss, not a general quality comparison or a completed study. The remaining preregistered task matrix is still needed before drawing aggregate conclusions. The earlier caveat about archived v1 Pass% remains unchanged.
5. scbrown

   added a commit that references this issue on Oct 16, 2026

   docs(eval): retain capped L0 result for #95 (#193)

   ...

   e48be5b
6. scbrown commented on Oct 17, 2026

   

   scbrown

   on Oct 17, 2026

   CollaboratorAuthor

   More actions

   The bounded fresh L0 run and its exact partial results are now published in PR #193, merged as e48be5bc7eafa9e66fa284ac891ffa5e67c2e234.

   The run hit its six-hour hard deadline (exit 124). It produced 330 of 1,320 planned cells across 10 of 40 tasks: 213 injected outcomes, 83 skips, and 34 errors. The report, sanitized rows, and frozen provenance are committed. These partial measurements do not establish full-cohort retrieval quality, and this issue remains open.

   No second campaign or silent continuation has been started. The next decision is whether a separately bounded indexing/reuse redesign can complete the preregistered matrix without increasing the resource/time envelope. Model and chunk-boundary decisions in #96/#97 remain gated on usable complete evidence; this partial run does not justify either change.
7. scbrown commented on Oct 21, 2026

   

   scbrown

   on Oct 21, 2026

   CollaboratorAuthor

   More actions

   2026-10-21 owner disposition: retain open; repair the measurement contract before another campaign

   Re-read the committed partial report at current main. It remains 330/1,320 cells across 10/40 tasks, with 213 injected, 83 skipped and 34 errors, stopped at the six-hour cap. This is not a complete calibrated baseline and does not release #96/#97.

   The concrete next source work is to discriminate the 34 no parseable chunk headers errors and reconcile the 23 dedup-worded skips against the preregistered --no-dedup contract, with positive/negative fixtures. Then prepare bounded indexing/reuse that preserves exact task, arm, source, model and configuration provenance within the existing resource envelope. Do not silently enlarge the cap or resume the old output stream.

   Owner: Gennaro. Resume the evaluation step only when those source/fixture contracts and the bounded reuse plan are actually delivered; freeze their identities before a separately bounded run. No new campaign, agent/model call, GPU work or shared index change was started by this triage. The historical v1 Pass% caveat remains; a new run cannot retroactively validate those old scorer verdicts. Existing partial results remain intact and this issue stays open.
8. scbrown commented on Oct 23, 2026

   

   scbrown

   on Oct 23, 2026

   CollaboratorAuthor

   More actions

   2026-10-23 triage — owner: Gennaro. Keep open; the next deliverable is measurement-contract repair, not another evaluation campaign.

   New source delivery since the October 21 disposition: #204 merged as 3d9073b41ca6dffc15583852a74911f20396d29a. Its identity-sealed opt-in scorer qualifies native workspace paths and refuses ambiguous, linked or unreadable inputs without numeric recall. That is a source/instrument repair; it neither rescored the historical rows nor produced new retrieval-quality evidence.

   At frozen current main fddcdb7b1bdf885528dbf0b432958125afe9c29d, the committed report remains 330/1,320 cells across 10/40 tasks: 213 injected, 83 skipped, 34 errors. Next I will reconcile the 34 header errors and 23 dedup-worded skips against the production-output and preregistered no-dedup contracts, accounting for #204's opt-in scoring convention. Deliverable: paired positive/refusal fixtures and an explicit disposition for each outcome family, then a bounded indexing/reuse plan with frozen task/arm/config/model/scoring identities. Preserve original rows; a changed scoring convention requires separate output.

   A further campaign starts only after those contracts and its bounded execution envelope are accepted. This triage starts no build, indexing, agent/model call or GPU run, raises no cap, and does not release #96/#97. Historical v1 Pass% remains separately caveated.
9. scbrown commented on Oct 24, 2026

   

   scbrown

   on Oct 24, 2026

   CollaboratorAuthor

   More actions

   2026-10-24 status — owner: Gennaro. The measurement-contract repair has started and its source changes have landed; a new calibrated campaign has not started.

   Since #204 (merge 3d9073b4, qualified scoring paths and unreadable-input refusal), #210 merged as e010faaf on October 23. Its opt-in independent-hook-outcomes/v1 contract treats contamination, contradictory/unsupported output and unexplained silence as errors with null efficacy metrics and quarantine, rather than skips or numeric zero recall. Valid headers and explicit operational/gate skips retain their meanings. Validation recorded 623 evaluation tests and 24 focused contract controls, including inverse controls.

   The reconciliation accounts for all 330 historical cells: 213 injected, 34 header errors, 23 dedup-worded skips, 42 operational skips and 18 gate skips. The scalar archive lacks the stdout needed to determine the exact cause of the 34 header errors; that cause remains unknown. Original rows and legacy scoring remain unchanged. No new retrieval-quality conclusion follows, and #96/#97 remain gated.

   Next: prepare the bounded-run readiness packet: enroll an exact current native runtime in sealed reuse (it currently admits exact 0.28.0 bytes), measure aggregate storage, and obtain acceptance of the existing six-hour/4-CPU/6-GiB/no-swap/no-GPU envelope. The proposed 32-GiB storage ceiling is neither accepted nor proven sufficient. ETA: readiness disposition by October 25, 18:00 UTC; campaign completion has no defensible ETA until those gates are resolved. No cap increase or silent continuation of the old campaign.