Repository navigation

Rebuild the eval engine on inspect_ai and retire the legacy runner - #19

Merged

rominf merged 77 commits into

amd:mainfrom

rominf:inspect-engine

Oct 22, 2026

Merged

Rebuild the eval engine on inspect_ai and retire the legacy runner#19
rominf merged 77 commits into
amd:mainfrom
rominf:inspect-engine

Commits on Sep 30, 2026
- Add inspect_ai engine behind --engine flag
  rominf
  committed
  760a7d3
- Record what a graded run spends, and compare engines
  rominf
  committed
  f06ee3e
- Add the Claude Code verification leg
  rominf
  committed
  5ba287b
- Count legacy model calls the same way in both commands
  rominf
  committed
  5693b4e
- Document the engine choice
  rominf
  committed
  9df5ac4
- Make the sandbox provider selectable, and stop it eating the skill's config
  rominf
  committed
  11340ae
- Say in the report whether the run was isolated
  rominf
  committed
  0f4011f
- Check the model answers before starting containers
  rominf
  committed
  6159330
- Stop a broken sandbox looking like an idle agent
  rominf
  committed
  bda1405
- Give a containerised case a working directory
  rominf
  committed
  10aa362
- Let the judge read the artifacts, and hear what the agent said
  rominf
  committed
  31e55a5
- Stop truncation hiding the action a check turns on
  rominf
  committed
  72f2c4f
- Use neutral values in the header-parsing fixture
  rominf
  committed
  125dcaa
- Exercise the inspect engine in CI, on both sandboxes
  rominf
  committed
  912653d
- Fix what CI found: platform-blind tests and a misplaced fixture root
  rominf
  committed
  d3cfe84
- Do not pin a mock model to opus under CI
  rominf
  committed
  94a3e43
- Stop the wiring run paying for a mock that cannot finish
  rominf
  committed
  0e53d7b
- Say what to install when a sandbox provider will not resolve
  rominf
  committed
  f6278a6
- Grade the answer the agent submitted, not the sentence introducing it
  rominf
  committed
  6cca129
- Document what podman actually needs
  rominf
  committed
  bd15fd7
- Resolve a declared compose file beside machine.yml
  rominf
  committed
  776a0b1
- Contain a skill that cannot be run to that skill
  rominf
  committed
  b1d48bd
- Show the judge everything the agent said, not just the last thing
  rominf
  committed
  92cc9e3
- Run the real CLI inside inspect's framework
  rominf
  committed
  3f75fe2
- Let the benchmark compare any two engines
  rominf
  committed
  26e56a1
- Run Claude Code where the scorers look
  rominf
  committed
  5ae3c79
- Name the engine that was actually asked for
  rominf
  committed
  0959415
- Send claude-cli to the engine it asked for
  rominf
  committed
  13ea9b9
- Launch the CLI on Windows, where it is not an executable
  rominf
  committed
  812b89c
- Give the CLI the skill it is supposed to be testing
  rominf
  committed
  3843dbc
- Let the shell probe survive a guest with no bash
  rominf
  committed
  59ba8d5
- Drop the live sample buffer on Windows
  rominf
  committed
  94a9ac5
- Make a timeout say where it got to
  rominf
  committed
  daefd7b
- Bound the model reachability probe
  rominf
  committed
  158f39d
- Stop routing reporting a sandbox it never started
  rominf
  committed
  1d265c7
- Show what a sandbox declaration actually looks like
  rominf
  committed
  b4fc2c7
- Drop the harness-independent engine
  rominf
  committed
  797cb72
- Rename claude-cli to claude-code-no-sandbox
  rominf
  committed
  1275c5e
- Name the no-sandbox driver after the engine it implements
  rominf
  committed
  d3aec79
- Close the gaps the removal left behind
  rominf
  committed
  56c27bc
- Refuse claude-code on a sandbox that shares the host filesystem
  rominf
  committed
  f386de6
- Give routing a leg on each engine that drives the real CLI
  rominf
  committed
  d39d4e5
- Stop a guard test depending on an optional extra
  rominf
  committed
  3c94966
- Make the sandbox-guard tests test the sandbox guard
  rominf
  committed
  41107b6
- Correct what now claims routing runs on one engine
  rominf
  committed
  0d53215
- Make --case-timeout bind on the new routing legs
  rominf
  committed
  cfa9e52
- Bring routing termination up to legacy, and past it where it can go
  rominf
  committed
  ccacf1a
- Say what each routing leg can actually stop
  rominf
  committed
  8c8cab5
- Say when the provider failed, not the skill
  rominf
  committed
  419ca52
- Refuse to skip a skill's hooks rather than skipping them quietly
  rominf
  committed
  152a101
- Treat a quiet finish as a routing result, not a failed run
  rominf
  committed
  abd323e
- Read the transcript too, not just the adopted messages
  rominf
  committed
  25cc974
- Count the routing budget per case, and count it the way legacy does
  rominf
  committed
  c4d6290
- A case that hit a limit ran; grade it
  rominf
  committed
  4a337be
- Count each turn once, not once per record it appears in
  rominf
  committed
  8631eb0
- Fix the test helpers the de-duplication change outgrew
  rominf
  committed
  5b6d60d
- Say how much of a run the budget decided
  rominf
  committed
  7697930
- Fit the routing budget to the engine it judges
  rominf
  committed
  fcbf785
- Calibrate the routing budget on decisions, not on totals
  rominf
  committed
  573ccc0
- Stop the host routing leg at the decision, as legacy does
  rominf
  committed
  f4dc593
- Key the turn de-duplication on what survives the bridge
  rominf
  committed
  1f1f5d8
- Give the sandboxed legs the effort they were told to use
  rominf
  committed
  f752c8d
- Require an inspect-swe that accepts the effort argument
  rominf
  committed
  c7444cd
- Give the sandboxed legs a whole skill, not a third of one
  rominf
  committed
  e013b6d
- Retire the legacy engine
  rominf
  committed
  93d106c
- Require inspect_ai rather than offering it as an extra
  rominf
  committed
  fbe6045
- Restore _isolation_note, and render the reports in tests
  rominf
  committed
  25ef773
- Run evals/hooks.py setup and teardown again
  rominf
  committed
  2132b65
- Say how long the federated token lasts
  rominf
  committed
  85518b1
- Make the default engine work on a default install
  rominf
  committed
  573b934
- Keep the answer key out of the room, and notice if it gets back in
  rominf
  committed
  77a0041
Commits on Oct 1, 2026
- Fix what the legacy removal left behind
  rominf
  committed
  84b7781
- Point at the upstream issue behind the budget workaround
  rominf
  committed
  5cd4bce
- Give both engines the spend and session controls legacy had
  rominf
  committed
  c3bbbda
Commits on Oct 2, 2026
- Stop the cost limit from refusing the run it was meant to bound
  rominf
  committed
  9967cb5
- Get the default engine running again, and make the stop report what it kept
  rominf
  committed
  05c2587
Commits on Oct 5, 2026
- Read a stopped run's log back by a path Windows can open
  rominf
  committed
  ff33db4

Commits

Commits on Sep 30, 2026

Commits on Oct 1, 2026

Commits on Oct 2, 2026

Commits on Oct 5, 2026