[Divide-By-0]

Sandbox fixes for the agentic solver, plus the traces that motivated them. Targeted at our fork, not upstream.

Changes

change	why
skip prompt attachments when solver=agentic	files are already copied into the sandbox; the duplicate is ~600k tokens on seqqa2 and hard-fails on Vertex (client.files.upload() is Developer-API only)
python_session replaces stock python()	94 of 125 tool errors were NameError from state loss between calls
pydoc for pydna / Bio.Restriction / primer3 baked into /opt/docs	agents were reading pydna's source and web-searching for its API
web search scoped to single lookups in the prompt	one sample spent 18 of its last 22 calls on DuckDuckGo, then hit the limit
network_mode: "none"	container had egress; the dataset's ideal answers are public on HuggingFace
tools/patch_broken_cloning_tasks.py	repairs three tasks unsolvable for environment reasons

Measured effect of the session change on one task: errored calls 23% → 2%, NameError 94 → 0.

Reading the traces

uv run inspect view --log-dir experiments/traces --recursive --port 7575   # http://127.0.0.1:7575
uv run inspect log dump experiments/traces/<file>.eval                      # JSON for jq

trace	run	result
seqqa2_6tasks_2epochs.eval	6 never-solved seqqa2 design tasks, 2 epochs	11/12
seqqa2_primer_design_rpoC.eval	single task, first sandbox pass	PASS
cloning_11tasks.eval	all gibson + golden-gate + hardest restriction-ligation, 2M	3/11
cloning_2tasks_2epochs.eval	2 never-solved cloning tasks	0/4
cloning_sorcs2_after_session_fix.eval	fb8fc27d post-fixes	fail
cloning_6tasks_rerun_6M.eval ⚠️	6 truncated tasks re-run at 6M	2/6 — see caveat

Results

seqqa2 design tasks — solved

gibson_primers had 0 passes in 119 graded published attempts across every model and mode, and went 6/6 here on the three longest genes. primer_design went 6/7. Those subtypes were never hard — 77% and 71% of published attempts never reached the validator at all.

cloning at 2M — the authorised configuration

verdict	tasks
pass	61e4b666 (patched), 5cf2e092, 4fb34135
genuine failure	0a4f4de7 (digest mismatch), 21e4def0 (accuracy), fb8fc27d (accuracy)
truncated at 2M — no verdict	dff28bd4, 3a6704ab, 31d22b22, a4bf037c, bc918101

# a genuine failure — ran to completion, wrong construct
uv run inspect log dump experiments/traces/cloning_11tasks.eval \
  | jq '.samples[] | select(.metadata.id|startswith("0a4f4de7")) | {limit, scores}'
# the truncations
uv run inspect log dump experiments/traces/cloning_11tasks.eval \
  | jq '.samples[] | select(.limit != null) | {id: .metadata.id, limit}'

⚠️ The 6M re-run is not valid at 2M. Five of its six samples exceeded 2M total, including both of its passes (bc918101 2.92M, a4bf037c 2.62M). Only fb8fc27d (1.76M) holds at 2M, and it failed. Excluded from the table above.

fb8fc27d (gibson Sorcs2) is the cleanest hard task in the benchmark: 0 passes in 23 graded published attempts and 0 across all 13 of our samples, always running to completion. No environment excuse.

Patched tasks — not comparable to published numbers

task	defect	fix	result
61e4b666	insert template is an 11-record FASTA the loader rejects (#16)	keep record 0, ENST00000612748.1	PASS — was 0/27
31d22b22	reference omits the BlastR cassette its prompt requires (#32)	splice the 454 bp BSD region in after the IRES	truncated
a4bf037c	filename contains a space; the DSL cannot express it (#34)	drop the browser (1) suffix, keep the Addgene ids	truncated at 2M

Record 0 is provably the template for 61e4b666: it sits in the reference at offset 2868 with exactly one mismatch, position 295 C→A — the C295A mutation the prompt asks for. That patch alone turned a task nothing had ever solved into a pass.

---

gpt-5.6-sol runs (added later)

trace	run	result
gpt56sol_smoke_2tasks_hinted.eval	max effort, original prompts, 2 tasks	0/2
gpt56sol_8tasks_method_blind.eval	max effort, method hint stripped, 8 tasks	1/8

Budgeted with --cost-limit 0.1 against experiments/novel_token_costs.yaml — cache priced at $0 so the limit is 100,000 novel tokens. All 8 ran to completion; zero truncations.

	gemini-3.7-flash	gpt-5.6-sol
novel tokens / sample	347k – 661k	22k – 76k (mean 46k)
tool calls / sample	46 – 59	10 – 31
truncated	6 of 11	0 of 8

~10× cheaper, and it finishes tasks Gemini never could at any budget we tried.

Removing the method hint changed nothing

All 7 failures still chose the correct method from the biology alone — gibson( for the four gibson tasks, goldengate( for the three golden-gate ones. The hint carried no signal; failures are downstream in junction and overhang design, not strategy.

⚠️ The tasks are only method-weaker, not method-blind — BsaI-MCS and BsmBI remain in the golden-gate prompts and are unmistakable Type IIS tells.

The single pass, a4bf037c, used restriction_assemble on a task whose reference is a Gibson assembly and still matched at ≥95% — the scorer compares the assembled product, not the route.

# which method each run actually chose
uv run inspect log dump experiments/traces/gpt56sol_8tasks_method_blind.eval \
  | jq -r '.samples[] | "\(.metadata.id[0:8]) \(.output.completion | capture("<protocol>(?<p>[^(]*)\\(").p)"'

Duplicate guard fired correctly

tools/check_run_guards.py flagged 0a4f4de7 and 3a6704ab as present in both runs — two results each (hinted and method-blind), which must not be pooled. Both failed in both conditions, so no conclusion changes.

fb8fc27d remains unsolved by anything: 0 across 23 published attempts, 13 Gemini samples, and gpt-5.6-sol at max effort.