[Fix] Honour val_begin=0 in EpochBasedTrainLoop - #1701
Open
dgexplores wants to merge 1 commit into
Open
dgexplores wants to merge 1 commit into
dgexplores wants to merge 1 commit into
Conversation
Set val_begin=0 to run a validation before the first epoch is trained. Previously the validation condition was only evaluated after run_epoch(), so val_begin=0 had no effect and the val loop first ran after the first epoch. The per-epoch validation timing is unchanged for every other value of val_begin. Fixes open-mmlab#1448
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Fixes #1448.
EpochBasedTrainLooponly ever evaluated its validation condition afterrun_epoch()had completed and incrementedself._epoch. Since_epochstarts at0and is bumped to1at the end of the first epoch, settingval_begin=0had no observable effect — the val loop first ran after the first epoch had finished training.This makes
val_begin=0unusable for the case it exists for: evaluating a model (e.g. a freshly added customMetric) before any training has happened.Modification
mmengine/runner/loops.py— inEpochBasedTrainLoop.run(), run the val loop once before the training loop whenval_begin <= 0:The
val_begindocstring is updated to document the0behaviour.Why not just move the validation block before
run_epoch()?That was suggested in the issue, but it changes the timing of validation for every existing user. Currently
val_begin=1(the default) validates after epoch 1, i.e. whenrunner.epoch == 1; moving the block ahead ofrun_epoch()would instead validate before epoch 2, andval_begin=1would no longer trigger any pre-training validation. The documented semantics (val_begin=2→ "start validation from the 2nd epoch",docs/en/tutorials/runner.md) would silently shift by one epoch.This PR instead adds an explicit epoch-0 validation, gated on
val_begin <= 0. Per-epoch validation timing is unchanged for all other values ofval_begin.BC-breaking (Optional)
No. For any
val_begin >= 1the sequence ofval_loop.run()calls is byte-identical. Only the previously-unreachableval_begin <= 0case changes behaviour — from "no pre-training validation" to "validate at epoch 0".IterBasedTrainLoophas the same structural gap (val_begin=0is likewise never honoured), but it is not touched here to keep this PR scoped to the reported bug.Use cases
Evaluate a model before training, e.g. to sanity-check a newly added custom
Metric:Validation now runs once at epoch 0, then every
val_intervalepochs as before.Validation
pytest tests/test_runner/test_runner.py -k val_begin_zero→ 1 passedAssertionError: Lists differ: [1, 2] != [0, 1, 2]pytest tests/test_runner/→ 68 passed, 3 failed, 3 skipped. The 3 failures (test_amp.py::TestAmp::test_autocast,test_runner.py::TestRunner::test_test,test_runner.py::TestRunner::test_val) are pre-existing — the identical 3 fail on unmodifiedmainin this environment (torch 2.14).pre-commit run --all-files→ all 17 hooks passedinterrogate -v --ignore-init-method --ignore-module --ignore-nested-functions --ignore-regex "__repr__" --fail-under 80 mmengine→ 80.7% (unchanged)New test
TestRunner::test_val_begin_zeroasserts both halves of the contract:val_begin0[0, 1, 2][0, 4, 8]1(default)[1, 2][4, 8]Note on CI
ci/circleci: lintis currently red for every open PR in this repository (#1696, #1699, #1700 as well as #1693), sopre-commit run --all-fileswas verified locally against the exact recipe in.circleci/test.ymlrather than via CI.