Bootstrap an isolated daemon for real-model memory evals and always clean up

[Copilot]

Summary

• evals/memory/run_real_model.py now starts its own isolated tracedecay daemon run for each scenario and waits for it to be ready before init. Previously init failed with code_index_scheduler_unavailable on a fresh profile.
• Setup, daemon, agent and cancellation failures are written to report.json as error entries instead of calling sys.exit.
• The daemon, fixture and profile are removed on every exit path.

Motivation

On a fresh isolated profile, the runner called tracedecay init with no daemon running. run() then called sys.exit, which skipped the report. Because fixture was only assigned after build_fixture() returned, the fixture directory was also left behind.

Changes

• Daemon lifecycle (EvalEnvironment.start_daemon / stop_daemon):
  • It launches the selected binary through scripts/lib/portable_process.py (exec-session, wait-unix-socket, stop-group), the same helper scripts/with-isolated-tracedecay-daemon.sh uses.
  • The socket lives in a short private tempdir, because Unix socket paths are length-limited.
  • TRACEDECAY_DAEMON_SOCKET and TRACEDECAY_DATA_DIR point at the isolated profile, so the runner cannot reach the operator's daemon or storage.
• Fixture ownership:
  • The fixture is now created under the eval root, and build_fixture removes it itself if any setup step fails.
  • seed_fixture is split out of build_fixture.
  • build_fixture gains an optional parent argument.
• Error handling:
  • run() raises the new EvalSetupError instead of calling sys.exit.
  • The per-scenario body moved into run_scenario(), which returns (ok, cancelled).
  • SIGTERM is converted to KeyboardInterrupt. A cancelled run writes a cancelled report and exits 130.
• Cost gate (--agent-turn plus --i-understand-model-cost) is unchanged.
• evals/memory/README.md documents the lifecycle.

Test plan

• The affected Bazel targets pass. The new tests are in evals/memory/test_run_real_model.py and pass under python3 -m unittest (17 tests); I did not run them through Bazel.
• bazel build //... passes when the change crosses package boundaries
• Tested manually (describe below if applicable)

The new tests use a fake tracedecay binary and spend no model tokens. They cover:

• init and seeding against the isolated daemon, with the daemon process, socket and profile gone afterwards
• failed init leaving no fixture directory
• daemon startup failure
• agent failure
• cancellation

Not verified:

• No run against a Bazel-built shipping tracedecay. Binary resolution is unchanged (target/debug, then PATH); pinning to the shipping build is tracked separately.
• No live Hermes or cursor-agent turn.

Checklist

• CHANGELOG.md updated (under [Unreleased] if no version bump)
• No secrets, credentials, or .env files included
• Breaking changes documented (if any)

• Fixes Bootstrap an isolated daemon for real-model memory evals and always finalize cleanup #4205

[changeset-bot]

⚠️ No Changeset found

Latest commit: b7f9c08

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR