Skip to content

[Fix] Honour val_begin=0 in EpochBasedTrainLoop - #1701

Open
dgexplores wants to merge 1 commit into
open-mmlab:mainfrom
dgexplores:dgexplores/val-begin-zero
Open

dgexplores wants to merge 1 commit into
open-mmlab:mainfrom
dgexplores:dgexplores/val-begin-zero

Conversation

@dgexplores

Copy link
Copy Markdown

Motivation

Fixes #1448.

EpochBasedTrainLoop only ever evaluated its validation condition after run_epoch() had completed and incremented self._epoch. Since _epoch starts at 0 and is bumped to 1 at the end of the first epoch, setting val_begin=0 had no observable effect — the val loop first ran after the first epoch had finished training.

This makes val_begin=0 unusable for the case it exists for: evaluating a model (e.g. a freshly added custom Metric) before any training has happened.

Modification

mmengine/runner/loops.py — in EpochBasedTrainLoop.run(), run the val loop once before the training loop when val_begin <= 0:

def run(self) -> torch.nn.Module:
    """Launch training."""
    self.runner.call_hook('before_train')

    # `val_begin` is 1-based for the validation performed after an epoch.
    # Allowing it to be 0 additionally validates at epoch 0, i.e. before
    # any training step has been taken.
    if self.val_begin <= 0 and self.runner.val_loop is not None:
        self.runner.val_loop.run()

    while self._epoch < self._max_epochs and not self.stop_training:
    ...

The val_begin docstring is updated to document the 0 behaviour.

Why not just move the validation block before run_epoch()?

That was suggested in the issue, but it changes the timing of validation for every existing user. Currently val_begin=1 (the default) validates after epoch 1, i.e. when runner.epoch == 1; moving the block ahead of run_epoch() would instead validate before epoch 2, and val_begin=1 would no longer trigger any pre-training validation. The documented semantics (val_begin=2 → "start validation from the 2nd epoch", docs/en/tutorials/runner.md) would silently shift by one epoch.

This PR instead adds an explicit epoch-0 validation, gated on val_begin <= 0. Per-epoch validation timing is unchanged for all other values of val_begin.

BC-breaking (Optional)

No. For any val_begin >= 1 the sequence of val_loop.run() calls is byte-identical. Only the previously-unreachable val_begin <= 0 case changes behaviour — from "no pre-training validation" to "validate at epoch 0".

IterBasedTrainLoop has the same structural gap (val_begin=0 is likewise never honoured), but it is not touched here to keep this PR scoped to the reported bug.

Use cases

Evaluate a model before training, e.g. to sanity-check a newly added custom Metric:

train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=10, val_begin=0)

Validation now runs once at epoch 0, then every val_interval epochs as before.

Validation

  • pytest tests/test_runner/test_runner.py -k val_begin_zero → 1 passed
  • Verified the new test fails without the source change: AssertionError: Lists differ: [1, 2] != [0, 1, 2]
  • pytest tests/test_runner/ → 68 passed, 3 failed, 3 skipped. The 3 failures (test_amp.py::TestAmp::test_autocast, test_runner.py::TestRunner::test_test, test_runner.py::TestRunner::test_val) are pre-existing — the identical 3 fail on unmodified main in this environment (torch 2.14).
  • pre-commit run --all-files → all 17 hooks passed
  • interrogate -v --ignore-init-method --ignore-module --ignore-nested-functions --ignore-regex "__repr__" --fail-under 80 mmengine → 80.7% (unchanged)

New test TestRunner::test_val_begin_zero asserts both halves of the contract:

val_begin epochs validated at iters at validation
0 [0, 1, 2] [0, 4, 8]
1 (default) [1, 2] [4, 8]

Note on CI

ci/circleci: lint is currently red for every open PR in this repository (#1696, #1699, #1700 as well as #1693), so pre-commit run --all-files was verified locally against the exact recipe in .circleci/test.yml rather than via CI.

Set val_begin=0 to run a validation before the first epoch is trained.
Previously the validation condition was only evaluated after
run_epoch(), so val_begin=0 had no effect and the val loop first ran
after the first epoch.

The per-epoch validation timing is unchanged for every other value of
val_begin.

Fixes open-mmlab#1448
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Setting EpochBasedTrainLoop.val_begin=0 does not work!

1 participant