Skip to content

evals: make field leaves stale scored output and field_report.md on rerun #170

Open

Description

@SamGu-NRX

Follow-up from PR #12, review thread #12 (comment). This is the same class as #111, which covers the stale report.md on #20.

Problem. When make field reruns into an existing FIELD_OUT, evals.field score removes only the four row files (ROW_FILES, experiments/evals/evals/field.py). Two derived outputs can survive:

  • If a rerun without TRUTH follows a run with TRUTH, the old FIELD_OUT/scored output from the scoring harness stays.
  • If a rerun fails after the cleanup, the old field_report.md stays.

Either can be mistaken for the current capture's results.

What a fix needs. At the start of score, remove or stage every derived output (field_report.md, scored/ and measure-lab.json) together with the row files. Better, write the whole results folder to a staging folder and swap it in only on success, as models.run.publish does. Then test a rerun without TRUTH and a rerun that fails after cleanup.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions