Integrate depth-dependent soil extraction and optional multi-year spin-up

CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## Repeated calendar-block spin-up — 2026-10-20
+
+- Add complete, consecutive `spinup.source_years` blocks to both solver backends, calibration and native workflows, retaining leap days and the original output calendar.
+- Count repetitions including the retained source block and exclude that block from scoring. Keep memory independent of repetition count by discarding each column's extra warm-up trajectories.
+- Retain mandatory convergence gates by default; support explicitly labelled fixed-duration experiments that preserve failed diagnostics without claiming convergence.
+- Validate compact histories against expanded histories and verify TOML/API integration; 480 offline tests passed in Gadi job 181730321.
+
 ## Daily two-parameter SWB — 2026-10-18
 
 - Use one conservative update per daily forcing record; retain donor/receiver limits, physical bounds and separate actual/unmet E/T budgets.

README.md

@@ -61,7 +61,9 @@ The five stages share one implementation in `pysweb.workflow`:
 | `calibrate` | Fit two domain parameters to reference surface soil moisture | Enabled explicitly; local reference or configured Earth Engine asset |
 | `run` | Simulate independent soil columns | Prepared forcing and hydraulic properties |
 
-Select stages with `--stages preprocess,calibrate,run`. The default workflow preserves source download grids, fits on native GSSM cells, and writes both final products on one selected Landsat 30 m UTM grid. It runs only Landsat 8/9 by default, for dates from 2013 onwards. Earlier missions require explicit `run.sensors = ["LT05", "LE07", "LC08", "LC09"]`; no second sensor pathway is launched automatically. For an 8/9 run beginning in 2013, calibration starts on 2014-01-01 by default, reserving the available 2013 record for initialisation. Set `calibration.start` to override this choice. For both calibration and normal runs, the SWB makes one conservative update per day, starts at physical field capacity and carries the available 2013 warm-up state into 2014. Resolved plans and final products identify the warm-up; NetCDF `is_warmup` flags those dates. No years are repeated unless an optional `[spinup]` table is explicitly supplied. Verify forcing coverage; this initialization assumes field capacity rather than claiming complete independence from the starting profile. Calibration is skipped when `calibration.enabled = false`. Its scoring window lies within the continuous simulation window. Set `run.holdout_start` to protect an independent test period. See [native grids and output contracts](docs/native-grids.md). Optional [configured acceptance protocols](docs/configured-protocols.md) integrate validated L7 donors, development-only spin-up, paired support, benchmark timing and site-specific evaluation. See [the workflow guide](docs/workflows.md) for offline operation, HPC execution and individual APIs.
+Select stages with `--stages preprocess,calibrate,run`. The default workflow preserves source download grids, fits on native GSSM cells, and writes both final products on one selected Landsat 30 m UTM grid. It runs only Landsat 8/9 by default, for dates from 2013 onwards. Earlier missions require explicit `run.sensors = ["LT05", "LE07", "LC08", "LC09"]`; no second sensor pathway is launched automatically. For an 8/9 run beginning in 2013, calibration starts on 2014-01-01 by default, reserving the available 2013 record for initialisation. Set `calibration.start` to override this choice. For both calibration and normal runs, the SWB makes one conservative update per day, starts at physical field capacity and carries the available 2013 warm-up state into 2014. Resolved plans and final products identify the warm-up; NetCDF `is_warmup` flags those dates. Repeated spin-up is disabled by default; set `[spinup] enabled = true` to enable it, or `enabled = false` to retain the settings without repetition. Verify forcing coverage; this initialization assumes field capacity rather than claiming complete independence from the starting profile. Calibration is skipped when `calibration.enabled = false`. Its scoring window lies within the continuous simulation window. Set `run.holdout_start` to protect an independent test period. See [native grids and output contracts](docs/native-grids.md). Optional [configured acceptance protocols](docs/configured-protocols.md) integrate validated L7 donors, development-only spin-up, paired support, benchmark timing and site-specific evaluation. See [the workflow guide](docs/workflows.md) for offline operation, HPC execution and individual APIs.
+
+For repeated multi-year initialization, see [calendar-block spin-up](docs/multiyear-spinup.md). A complete 2014–2015 block can be repeated five or ten times without expanding forcing grids or spatial output calendars. Repetition counts include the retained source block; scoring starts afterward. Convergence is checked separately from numerical conservation.
 
 Every configured run writes `resolved-config.json`. Model output manifests record input SHA-256 checksums, configuration, code identity and output checksums. `skip_existing` validates the model output against this identity before reuse. Unmanifested older outputs require regeneration. Generated data and environments stay outside version control.
 
@@ -124,4 +126,4 @@ The code is distributed under the [MIT licence](LICENSE). Dataset licences remai
 
 Component outputs use `E_s` (soil evaporation), `E_t` (transpiration) and `f_t` (transpiration fraction). See the [notation migration and legacy-input policy](docs/migration.md#component-notation-migration-11-october-2026).
 
-The [conservative two-parameter SWB formulation](docs/swb-formulation.md) calibrates diffusivity and root beta, uses fixed versioned GSSM references and records separate actual/unmet E/T with conservative water budgets. Existing four-parameter calibrations require recalibration.
+The [conservative two-parameter SWB formulation](docs/swb-formulation.md) calibrates diffusivity and root beta, uses fixed versioned GSSM references and records separate actual/unmet E/T with conservative water budgets. V3 applies the GSSM/WP dry ratio to extraction thresholds in 0–15 cm, a half adjustment in 15–30 cm, and hydraulic WP below 30 cm. Continuous three-day relaxation and physical zero-to-porosity bounds are retained. Existing v2 and four-parameter calibrations require recalibration; immutable GSSM reference artifacts remain compatible.

docs/configured-protocols.md

@@ -22,12 +22,17 @@ The padding table contains `primary_bounds`, `comparison_bounds`, their `crs`, `
 
 ## Paired support and initialization
 
-The usual Landsat 8/9 workflow runs alone. `forcing.peer_runs` is optional and should be set only for an explicitly requested comparison. For runs beginning in 2013, the default calibration scoring start is 2014-01-01, so available 2013 forcing supplies initialisation. Both calibration and normal runs (including `calibration.enabled = false`) start at physical field capacity, run the available 2013 record once and carry its state into 2014 without resetting. Resolved plans record the initialization and analysis windows; final SWB products preserve the actual calendar and mark warm-up days with `is_warmup`. Configured evaluation defaults to the analysis start. Omit `[spinup]` for this default pathway: there are no artificial years or mandatory repeated-cycle gates. Incomplete calendar-year coverage is acceptable when the available daily prefix is complete from the chosen start. This is an initial-state assumption, not a claim of convergence from arbitrary profiles. See [the actual-2013 warm-up example](../examples/2013-warmup.toml). The repeated-year protocol below is an additional explicit option with its own complete-year and convergence requirements.
+For complete multi-year blocks, use the [calendar-block spin-up interface](multiyear-spinup.md).
+It keeps source forcing and output dates unexpanded. The `source_year` interface
+below remains available with its original artificial-prefix semantics.
+
+The usual Landsat 8/9 workflow runs alone. `forcing.peer_runs` is optional and should be set only for an explicitly requested comparison. For runs beginning in 2013, the default calibration scoring start is 2014-01-01, so available 2013 forcing supplies initialisation. Both calibration and normal runs (including `calibration.enabled = false`) start at physical field capacity, run the available 2013 record once and carry its state into 2014 without resetting. Resolved plans record the initialization and analysis windows; final SWB products preserve the actual calendar and mark warm-up days with `is_warmup`. Configured evaluation defaults to the analysis start. Omit `[spinup]` or leave `spinup.enabled = false` for this default pathway: there are no artificial years or mandatory repeated-cycle gates. Incomplete calendar-year coverage is acceptable when the available daily prefix is complete from the chosen start. This is an initial-state assumption, not a claim of convergence from arbitrary profiles. See [the actual-2013 warm-up example](../examples/2013-warmup.toml). The repeated-year protocol below is an additional explicit option with its own complete-year and convergence requirements.
 
 `forcing.peer_runs` lists the other run directories for a paired comparison. **Preprocess both runs before calibrating either.** Each run writes its own complete development forcing/physical-soil mask on both preparation grids. Initialization intersects corresponding masks with exact coordinates and CRS, preserving missing holdout forcing without using it to select cells. OpenLandMap's optional `soil.maximum_organic_matter_percent` also rejects hydraulic estimates outside the configured pedotransfer applicability domain.
 
 ```toml
 [spinup]
+enabled = true
 source_year = 2014
 repeats = 18
 tolerance = 0.005

docs/migration.md

@@ -1,5 +1,29 @@
 # Migration to the maintained package
 
+## Explicit repeated-spin-up switch (21 October 2026)
+
+Repeated forcing is disabled by default. Set `[spinup] enabled = true` to enable
+the existing single-year or multi-year protocol. An omitted or false `enabled`
+key leaves repetition disabled even when source years and a repetition count
+are present. This allows the settings to remain in a study configuration while
+the experiment is switched off. Actual run dates, physical field-capacity
+initialisation and explicit calibration windows remain unchanged.
+
+Update earlier repetition configurations by adding `enabled = true` when using
+this branch. Direct column calls likewise use
+`spinup={"enabled": True, "source_years": [2014, 2015], "repeats": 10}`.
+The existing component CLI/API `spinup_source_years` option is itself an explicit
+request to enable repetition. See [usage and disabled behaviour](multiyear-spinup.md).
+The daily physical equations and formulation identity are unchanged by this edit.
+
+## Depth-dependent extraction thresholds (21 October 2026)
+
+`swb-daily-depth-extraction-2p-v3` applies the frozen GSSM-to-WP dry ratio to extraction in 0–5 and 5–15 cm, uses the midpoint of adjusted and hydraulic WP in 15–30 cm, and retains WP below 30 cm. The continuous three-day supply relaxation, physical field-capacity initialisation, conservative flux update and zero-to-porosity physical bounds are retained. See the [equations, upper-bound assessment and limitations](swb-formulation.md).
+
+Use a new output directory and refit diffusivity and root beta. V2 fitted coefficients are rejected because the extraction equation changed. Existing immutable GSSM p01/p99 artifacts remain accepted without rewriting their hashes. Output diagnostics now include `below_reference_days_layer_*` for every layer, and calibration/product identities include `extraction_threshold_profile` and `physical_upper_bound`. The reference remains an empirical surface constraint; its depth transfer does not establish physiological validity or improved predictive skill.
+
+The depth-dependent physics also applies when repeated spin-up is enabled; the switch controls initialization only.
+
 ## September 2026 baseline
 
 GitHub and Gadi shared commit `e96879a`; the local checkout was four commits behind. Gadi additionally contained notebook/PBS improvements. Those two files were committed as `3fb0f23`, then GitHub, local and Gadi were fast-forwarded to that baseline before this update. This preserves the latest development rather than replacing it with an older checkout.

docs/multiyear-spinup.md

@@ -0,0 +1,96 @@
+# Repeated calendar-block initialization
+
+Repeated spin-up is disabled by default. Set `spinup.enabled = true` to use a
+complete block at the beginning of the forcing record for repeated
+initialisation. For example:
+
+```toml
+[run]
+start = "2014-01-01"
+end = "2025-12-31"
+holdout_start = "2021-01-01"
+state_timing = "start"
+
+[calibration]
+enabled = true
+start = "2016-01-01"
+end = "2020-12-31"
+
+[spinup]
+enabled = true
+source_years = [2014, 2015]
+repeats = 10
+tolerance = 0.005
+minimum_fraction = 0.99
+convergence_mode = "wet_dry"
+convergence_action = "require"
+```
+
+This example integrates the 730-day block ten times (twenty simulated years).
+The count includes the retained 2014–2015 pass. The solver first integrates nine
+extra passes, discarding their trajectories, then continues through the actual
+2014–2025 record without resetting the soil state. Output dates remain unchanged.
+All 2014–2015 outputs are initialization and are excluded from fitting and
+evaluation. Calibration observations are never repeated.
+
+To disable repetition while retaining the other settings for later use, set:
+
+```toml
+[spinup]
+enabled = false
+```
+
+An omitted table, an empty table or an omitted `enabled` key also disables
+repetition. `enabled` must be a boolean. Disabled settings do not require source
+years, complete calendar blocks or convergence diagnostics. The ordinary run
+starts at physical field capacity and processes its actual dates once. Explicit
+calibration dates and any actual-forcing prefix remain in effect; the switch
+does not remove ordinary initialisation or rewrite the requested date range.
+
+Configurations created before the explicit switch need `enabled = true` to
+reproduce their repeated-year experiment with this version. Preserve their
+original source snapshot when reproducing an archived accepted run.
+
+`source_years` must be consecutive integer years forming a complete initial daily
+forcing block. Leap years retain every day. Calibration must begin after the
+block. The native workflow requires a development boundary before the holdout.
+The existing single-year `source_year` interface retains its earlier semantics.
+Do not specify both interfaces.
+
+The reference and Numba solvers share this implementation. Their public
+`soil_water_balance_1d` functions accept
+`spinup={"enabled": True, "source_years": [2014, 2015], "repeats": 10}`.
+Passing `None`, an empty mapping or `{"enabled": False}` disables repetition;
+omitting `enabled` in a supplied mapping also leaves it disabled.
+Gridded run and calibration
+APIs accept `spinup_source_years` and `spinup_repeats`; the command-line flags are
+`--spinup-source-years 2014 2015 --spinup-repeats 10`. Direct calibration also
+requires `--spinup-days 730` to exclude the retained block.
+Supplying `--spinup-source-years` explicitly enables repetition in these
+component APIs; omit the two repetition flags for the default disabled path.
+`--spinup-days` alone excludes an actual forcing prefix from calibration and
+does not repeat any forcing.
+
+Only one column's current cycle is allocated during initialization. Neither
+forcing grids nor spatial warm-up outputs are expanded with repetition count.
+Runtime still increases with the number of simulated days.
+
+## Interpreting convergence
+
+`wet_dry` compares empty and saturated initial states at the scoring boundary.
+`field_capacity_cycles` compares field-capacity starts with N and 2N total passes;
+passing it does not establish independence from other initial states. Each test
+requires every layer to meet the tolerance in the requested fraction of admitted
+cells. Default and newly calibrated parameters are checked separately.
+
+The default `convergence_action = "require"` stops on a failed diagnostic.
+An explicitly labelled fixed-duration research experiment may use `"record"`.
+This retains a failed receipt, emits a warning and records the interpretation in
+product initialization metadata. It does **not** turn the diagnostic into a pass
+or establish readiness for regional inference. Numerical conservation and
+calibration reproduction remain mandatory in either mode.
+
+Repeated forcing can reduce sensitivity to assumed initial water storage. It
+does not correct bias in forcing, hydraulic properties, rooting assumptions or
+the observation operator, and a longer initialization need not converge for
+every parameter setting.