From a full-repo review (line refs from ecb7fe0).
Per-user resume skips are keyed only on parquet existence, with no model/config identity or completeness check:
src/forecasting_evaluation/io/predict_result_writer.py:166-170 + evaluation/evaluator.py:278-284,381-382: re-running with a different forecaster in the same output_dir regenerates nothing — the previous model's predictions are re-scored under the new name.- A mid-run crash still finalizes a valid partial parquet via
finally: predict_writer.close(); the next run's skip treats it as complete, silently shrinking that user's evaluated windows.
metrics/offline/pipeline.py:313-321 has the same existence-only skip for metric outputs.
Suggested fix: stamp a sidecar (model name + config hash + expected window count) when a user file is finalized; on resume, skip only if the stamp matches and the row count is complete, else fail loud or regenerate. Mark files written during an aborted run (e.g. .tmp rename-on-complete) so partials are never mistaken for finished output.
From a full-repo review (line refs from
ecb7fe0).Per-user resume skips are keyed only on parquet existence, with no model/config identity or completeness check:
src/forecasting_evaluation/io/predict_result_writer.py:166-170+evaluation/evaluator.py:278-284,381-382: re-running with a different forecaster in the sameoutput_dirregenerates nothing — the previous model's predictions are re-scored under the new name.finally: predict_writer.close(); the next run's skip treats it as complete, silently shrinking that user's evaluated windows.metrics/offline/pipeline.py:313-321has the same existence-only skip for metric outputs.Suggested fix: stamp a sidecar (model name + config hash + expected window count) when a user file is finalized; on resume, skip only if the stamp matches and the row count is complete, else fail loud or regenerate. Mark files written during an aborted run (e.g.
.tmprename-on-complete) so partials are never mistaken for finished output.