evals: make field leaves stale scored output and field_report.md on rerun #170
Copy link
Copy link
Open
Description
Follow-up from PR #12, review thread #12 (comment). This is the same class as #111, which covers the stale report.md on #20.
Problem. When make field reruns into an existing FIELD_OUT, evals.field score removes only the four row files (ROW_FILES, experiments/evals/evals/field.py). Two derived outputs can survive:
- If a rerun without
TRUTHfollows a run withTRUTH, the oldFIELD_OUT/scoredoutput from the scoring harness stays. - If a rerun fails after the cleanup, the old
field_report.mdstays.
Either can be mistaken for the current capture's results.
What a fix needs. At the start of score, remove or stage every derived output (field_report.md, scored/ and measure-lab.json) together with the row files. Better, write the whole results folder to a staging folder and swap it in only on success, as models.run.publish does. Then test a rerun without TRUTH and a rerun that fails after cleanup.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity