Skip to content

Fix cloning simulators and add sequence review tooling - #2

Draft
ayu-atp wants to merge 29 commits into
fix/agentic-skip-redundant-attachmentsfrom
codex/cloning-sequence-visualization
Draft

ayu-atp wants to merge 29 commits into
fix/agentic-skip-redundant-attachmentsfrom
codex/cloning-sequence-visualization

Conversation

@ayu-atp

@ayu-atp ayu-atp commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator

This PR contains

Description

This PR adds an Inspect-native sequence-review workflow and fixes cloning
simulator/scorer defects found while reviewing the saved model traces.

Simulator and verifier changes

  • Add versioned v2 implementations for PCR, Golden Gate, Gibson, and sticky-end
    restriction assembly while retaining wrappers for the upstream behavior pinned
    at c028ecdcf144b55ffcd92b68be45081df5628c20.
  • Recover unique exact PCR amplicons that cross coordinate 0 on circular templates.
  • Build multi-fragment Golden Gate products one compatible junction at a time.
    Empty-vector/dropout reassembly remains a plausible candidate; products that
    restore Type IIS sites rank below site-free products.
  • Fix Gibson enumeration so a valid circle is not hidden by another possible
    extension, track fragments by input position rather than generated name, and
    deduplicate reverse-complement representations.
  • Fix restriction assembly so backbone self-ligation does not suppress a valid
    insert-containing product.
  • Score every physically enumerated top-level product rather than result[0].
    Nested multi-product operations fail explicitly as ambiguous because the DSL
    has no product-selection operator. The same candidate must pass both global
    similarity and any digest validator.
  • Repair missing reference circularity only when digest metadata proves it during
    normal scoring. The offline rescoring tool also supports the documented
    final-plasmid corpus policy.
  • Bump LAB-Bench 2 from 1-A to 2-A because these changes alter cloning scores.

The design rationale, audit findings, and modeling limits are documented in
src/lab_bench_2/CLONING_SIMULATOR_V2.md.

Inspect review tooling

  • Render aligned predicted/reference plasmid maps with annotations, readable
    coordinate ticks, topology, digest sites/fragments, sequence differences,
    FASTA views, and recursive sequence provenance.
  • Transfer repeated and boundary-modified GenBank features so constructs such as
    the five-copy mCherry reference remain visible.
  • Add tools to backfill visualizations and rescore existing traces without
    regenerating model answers. Original scores, explanations, task version, and
    aggregate metrics are preserved in metadata.

Rescored saved submissions

The six *_simulator_v2_reviewed.eval files reuse the recorded answers and include
the rendered comparison event. They have internally updated 2-A aggregate
metrics; the original 1-A summaries remain in metadata.

Run Original v2 Changed
cloning_11tasks 3/11 6/11 3
cloning_2tasks_2epochs 0/4 0/4 0
cloning_6tasks_rerun_6M 2/6 3/6 1
cloning_sorcs2_after_session_fix 0/1 0/1 0
gpt56sol_8tasks_method_blind 1/8 6/8 5
gpt56sol_smoke_2tasks_hinted 0/2 2/2 2

Across 32 stored submissions, 11 verdicts change from incorrect to correct. One
method-blind GPT-5.6-sol answer passes using Gibson candidate 3 of 14, directly
showing why the first candidate is not authoritative. The SORCS2 answer now forms
an amplicon but remains incorrect at 0.839761 sequence similarity.

Validation

  • make check: all 7 enforced checks pass. Advisory failures are the pre-existing
    private Inspect API import, generated-README dirty-tree heuristic, and a local
    markdownlint Node SSL bootstrap issue.
  • LAB-Bench 2 tests: 166 passed and 12 skipped. The two initially sandbox-blocked
    Inspect tests pass when INSPECT_TRACE_FILE is placed under /private/tmp.
  • Simulator/scorer/visualization regression tests: 76 passed.

Checklist

  • Are you adding a new eval?

  • Does this change affect existing eval(s)?

  • Does this change affect how future contributors write or submit evaluations (e.g. new required fields, changed tooling, updated conventions)?

@ayu-atp ayu-atp changed the title Add cloning sequence visualization, provenance, and topology-aware rescoring Fix cloning simulators and add sequence review tooling Aug 27, 2026
@ayu-atp
ayu-atp marked this pull request as draft August 27, 2026 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant