[maxwellpeterson]

WIP -- Not ready for review

[ask-bonk]

LGTM!

github run

[github-actions]

Preview: pr896-mpeterson-con-f3f115f7

https://pr896-mpeterson-con-f3f115f7-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[github-actions]

Eval results

Verdict: ⚪ Unchanged. No task moved beyond what these runs can tell apart from noise.

Task	Score	Δ score	Fisher test	Cache hits	Avg min	Avg steps
change-calendar (gpt-6-luna)	100% → 90%	−10 pp	p = 1.00	95% 0 pp	4.5 → 4.3	30.5 → 26.4
chess (gpt-6-luna)	60% → 70%	+10 pp	p = 1.00	98% 0 pp	8.2 → 7.7	50.9 → 50.1
compaction (claude-sonnet-5-5)	100%	0 pp	p = 1.00	90% 0 pp	2.0 → 1.8	43.0
compaction (gpt-6-luna)	100%	0 pp	p = 1.00	93% 0 pp	1.7 → 1.5	43.0
incident-desk (gpt-6-luna)	100% → 90%	−10 pp	p = 1.00	95% → 96% +1 pp	4.3 → 3.8	26.1 → 28.2
ticket-triage-agent (gpt-6-luna)	100%	0 pp	p = 1.00	96% 0 pp	0.1 → 0.2	1.6 → 2.0
worker-logs (gpt-6-luna)	100%	0 pp	p = 1.00	95% 0 pp	4.2 → 5.3	26.1 → 27.2

Failed checks

Task	Check	Failed
change-calendar (gpt-6-luna)	t1 agent.timedOut	0/10 → 1/10
chess (gpt-6-luna)	t1 agrees-with-the-oracle-on-the-hard-positions	3/10
chess (gpt-6-luna)	t1 agrees-with-the-oracle-on-perft-positions	1/10 → 3/10
chess (gpt-6-luna)	t1 agrees-with-the-oracle-through-random-games	1/10 → 2/10
chess (gpt-6-luna)	t1 the-game-is-shared-across-connections	0/10 → 2/10
chess (gpt-6-luna)	t1 starts-from-the-standard-position	0/10 → 1/10
chess (gpt-6-luna)	t3 repeats-a-position-only-with-the-same-rights	1/7 → 0/7
chess (gpt-6-luna)	t3 rules-still-agree-with-the-oracle-after-draws	1/7 → 0/7
incident-desk (gpt-6-luna)	t1 opens-acknowledges-and-resolves-in-order	0/10 → 1/10
incident-desk (gpt-6-luna)	t1 simultaneous-acknowledges-yield-exactly-one-owner	0/10 → 1/10

Run · trajectories and raw results

[ask-bonk]

🔬 Eval runs review

Performance

Against main, chess passed 6/10 → 7/10, change-calendar and incident-desk each fell 10/10 → 9/10, and all other task/model rows remained fully passing; comparison.json finds no pass-rate or prompt-cache movement beyond noise. Costs moved both ways: chess averaged approximately $0.0494 → $0.0456/run, while worker-logs rose $0.0217 → $0.0254/run and 4.2 → 5.3 minutes; the single Sonnet compaction run remained the costliest individual run ($1.77 → $1.82). Chess accounted for the most failed runs (4 → 3) and model steps (50.9 → 50.1/run), with both sides spending steps on handwritten rules, PGN/draw extensions, and edit retries.

⚪ VERDICT: NO REGRESSION FROM THIS PR

The Context observer-tracking migration leaves the failing code paths unchanged, Context skill reads succeed in both trajectories, and neither the outcome changes nor the mixed cost changes establish a PR-caused effect.

Triage

Failure modes

• Incorrect chess rules · chess 2/10 · model error · this PR: no — Trial 1, turn 1, wrote reversed rook relocation for castling; executeCode returned the incorrect FEN, but the final reply still claimed verification. Trial 8’s turn-1 writeFile limited every move to two files horizontally, dropping legal sliding-piece moves; its smoke tests missed this. The sharing check’s failure in that run was also a move-set mismatch, not lost shared state.
• Invalid generated JavaScript · chess 1/10 · model error · this PR: no — Trial 2, turn 1, wrote a malformed nested ternary in client.js and finished without calling executeCode; verification could not start the gadget Worker because of Unexpected token ':'.
• Activation stalled before completion · change-calendar 1/10 · harness bug · this PR: no — Trial 3, turn 1, ended after three model steps and successful writeFile returns, with no subsequent reply before the 420-second activation timeout. The underlying stall is not logged; this happened before Skills was enabled or any Context read occurred.
• Internal call violates its own argument contract · incident-desk 1/10 · model error · this PR: no — Trial 3’s turn-1 writeFile defined incident({ id }) but called this.incident(id) from acknowledgment and resolution. Later editFile calls retained that mismatch, so existing incidents returned UNKNOWN_INCIDENT, including losing acknowledgment attempts; the agent never exercised those RPCs before declaring completion.

Tool errors

• editFile: No matching text was found · change-calendar 4 → 0, chess 12 → 13, incident-desk 0 → 1, worker-logs 2 → 0 · model error — Agents supplied text differing from the current file, especially escaped regexes; candidate chess trial 5 wasted six calls on over-escaped matches before correcting them.
• editFile: Validation failed · change-calendar 2 → 4, chess 2 → 3, incident-desk 1 → 3, worker-logs 2 → 4 · model error — Calls omitted required fields or supplied code/batch objects instead of edit parameters. For example, candidate calendar trial 6 supplied only the workpiece and filename, then recovered with a complete call.
• editFile: Multiple matches were found · chess 2 → 0, incident-desk 2 → 2 · model error — Replacement spans were not unique. Candidate incident-desk trial 5 matched both SQL projections, then recovered by including the surrounding method.
• executeCode: Failed to start Worker · chess 3 → 0 · model error — Baseline trial 6 submitted .}, . n, and .foo as executable modules, then retried with valid JavaScript; these were malformed inputs, not Context failures.
• executeCode: This step already made code changes · change-calendar 0 → 1, worker-logs 0 → 1 · tool design — The description does not advertise the edit-step barrier. Both candidate runs batched edits with execution, received the recovery instruction, and succeeded by executing in the next step.

What to do

• Optional, outside this PR: in packages/workshop-backend/src/agent.ts, add verification guidance to exercise gadget RPCs, compare returned state with expectations, and clean up test data before claiming completion.
• Optional, outside this PR: in the same file’s EXECUTE_CODE_TOOL_DESCRIPTION, explicitly say that execution must occur in a separate model step after file edits.

github run