[mj023]

What problem do you want to solve?

GridSearch averages the next-period value over every joint lottery node at every
state–action candidate. A Markov or process draw whose weights read no state or action
other than the drawn state itself, and whose value no law reads, only selects a slice of
V′. Its expectation is therefore the same for every candidate. This PR sums such draws
into V′ once per kernel call. The continuation then reads one row at the current node
instead of mapping over the draw.

On ACA's benchmark grids, spousal income, the wage residual (Rouwenhorst) and health are
contracted. The warm solve drops from 11.67 s to 7.96 s (1.47×), with values equal to
2.65e-15.

Summary

∑_{z'} P(z'|z) · interp(V'(·, z'), y)  =  interp(Ṽ(·, z), y),   Ṽ(·, z) = ∑_{z'} P(z'|z) V'(·, z')

Linear interpolation is linear in the node values, so the swap is exact in real
arithmetic.

 _ComputeCE.__call__  (per state–action candidate)
   for each target
     next_values = next_regime_to_V_arr[target]
+    next_values = continuation.with_contracted_draws(values=next_values, ...)  # once per call: Ṽ
     next_V(**coordinates, next_V_arr=next_values)   # contracted draws: no node axis,
                                                     # read at the current node

 src/_lcm/
 ├── regime_building/
+│   ├── lottery_contraction.py   # eligibility, rewritten laws/plans, the contraction
 │   ├── Q_and_F.py               # _contract_target_lotteries; field on _TargetContinuation
 │   └── processing.py            # solve site passes LotteryContractionSource for GridSearch
 └── solution/
     └── grid_search.py           # GridSearch(lottery_contraction: bool = True)

A draw next_<state> toward a target is contracted only when all of these hold:

• Linear expectation. The regime's certainty equivalent is the linear expectation.
• Ordinary reader. The continuation is not gated, collective, landing-counting or
  participation-querying, and the target is not reached through a participation check,
  which rebinds the reader later.
• Persisted state. The draw is a persisted-state lottery onto the target's grid, with
  no original (fixed-component) layout.
• Pure index. No law of the target reads the drawn value, directly or through helpers.
• No cell inputs in the weights. The weights read no source state or action other
  than <state>. Period, age and parameters are allowed.
• Static source grid. The source grid is a discrete grid with codes 0..n-1, or a
  stochastic process with static, finite nodes. Runtime-parameterized processes keep
  their axis.

The table is indexed by the source grid, so cross-grid draws (ACA health, 3 → 2
categories at 65) are covered. When the weights don't read <state> (IID draws), all
rows coincide: the continuation reads row 0 and no current value. That keeps the
existing broadcast of unread IID states.

Numerics: each row uses zero_safe_average over scaled_joint_weight coefficients,
like the per-node path. The mean is held between its smallest and largest live value,
because summing one draw on its own at near-maximal values could otherwise round past
the largest finite number.

Evidence

• Oracle: tests/regime_building/test_lottery_contraction.py checks a literal
  joint-node expectation, with contraction on and off, for these cases:

  • same-grid and cross-grid targets;
  • a Markov process and an IID process;
  • draws that must stay enumerated: weights reading another state, weights reading an
    action, and a draw read by next wealth.

  The planner tests cover runtime-node processes and the IID law reading no state.
  test_continuation_expectation_oracle.py (including the near-float-max cases) passes.

• Full CPU suite, float64: Marvin, 48 workers, compared with the base commit
  2ca4c2c05.
  Before (first version of this branch): 104 failed against 32 on the base, so 72
  were caused by contraction. The causes:

  • all-NaN values for runtime-parameterized processes;
  • the IID broadcast broken;
  • participation readers rebinding a contracted continuation;
  • fingerprint pins.

  After (head 64c8314ad, job 28686948): 32 failed, 29853 passed, 410 skipped,
  9 xfailed. The 32 failures are exactly the base's 32; none is new.

• ACA benchmark grids (RTX 3050, float64, warm median of 3): off 11.67 s, on 7.96 s,
  1.47×. 184 value arrays: max relative difference 2.65e-15, no finiteness mismatches.
  Measured on the pre-rebase base; to re-measure on Keep solved values on the host or on disk (#591) #633.

• Production-grid memory, compile-only, 2 preference types: one V′-sized table per
  target, temporaries about 4× V′ (largest 37.6 MiB → 150 MiB).

Merge Danger

Door: two-way. GridSearch(lottery_contraction=False) restores per-node enumeration.

Blast Radius: numerical.

• Default on: every GridSearch solve with an eligible draw changes by rounding.
  Stored-bytes regressions and bitwise parity checks see that. The broadcast and
  fixed-component parity tests now switch contraction off.
• Fingerprints change for every model (GridSearch gained a field). The pins are
  regenerated, and the base reproduces the old ones.
• Certificate: module-surface pins of Q_and_F.py, processing.py and
  grid_search.py re-anchored, byte seals refreshed, verify.py passes.
• Memory: the contraction runs inside the compiled period core, so its tables and
  temporaries are part of the compiler's memory report that budgeted width selection
  admits against. On the test model the core's temporaries drop from 1.87 MB to
  0.41 MB with contraction on, at the same widths. The table is V′-sized and does not
  shrink with narrower widths.

Notes

• Where the switch lives: lottery_contraction is the first option on GridSearch;
  execution switches such as donate_buffers live on ExecutionConfig. It sits on the
  solver because it changes how GridSearch computes the expectation, can be set per
  regime and means nothing to other solvers, like DCEGM's method options.
• Per-tile rebuild on GPU: the contraction is traced inside the cell- and
  action-tile loop body. XLA:CPU hoists it out of the loop; XLA:GPU does not, so on GPU
  the table is rebuilt once per (cell tile, action tile): 12 times per core call at cell
  width 64 on the test model. Values and memory are unaffected, and the ACA speedup
  above already includes this repeated work. We do not hoist the contraction here,
  because building it once before the tiles would need structural changes to the chunk
  loop.
• CI manifest: registering the test file re-derives the weighted shard split, so many
  files move between shards in ci-workloads.json. The tool does this by design; base
  and branch both pass --check.

Todo

• Rerun the full CPU suite (float64) on the final head.
• Run the full CPU suite in float32.
• Re-measure the ACA benchmark on the Keep solved values on the host or on disk (#591) #633 base.
• Decide whether contraction should default to on.

🤖 Generated with Claude Code

[read-the-docs-community]

Documentation build overview

📚 pylcm | 🛠️ Build #35666240 | 📁 Comparing df375f5 against latest (f57269f)

  🔍 Preview build  

38 files changed · + 1 added · ± 37 modified

+ Added

• shocks/index.html

± Modified

• age-specialized/index.html
• architecture/index.html
• architecture-transition-ledger/index.html
• benchmarking/index.html
• beta-delta/index.html
• categories-with-characteristics/index.html
• certification/index.html
• choosing-a-solver/index.html
• collective-regimes/index.html
• collective-regimes-1/index.html
• and 27 more...

[codecov]

Codecov Report

❌ Patch coverage is 73.93617% with 49 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (feat/591-value-storage@628c38b). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/_lcm/regime_building/lottery_contraction.py	74.54%	42 Missing ⚠️
src/_lcm/regime_building/Q_and_F.py	65.00%	7 Missing ⚠️

Additional details and impacted files

@@                    Coverage Diff                    @@
##             feat/591-value-storage     #636   +/-   ##
=========================================================
  Coverage                          ?   94.10%           
=========================================================
  Files                             ?      663           
  Lines                             ?   111924           
  Branches                          ?        0           
=========================================================
  Hits                              ?   105321           
  Misses                            ?     6603           
  Partials                          ?        0           

Flag	Coverage Δ
cpu-python	94.10% <73.93%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Benchmark comparison (main → HEAD)

Comparing c8430c80 (main) → b80f7a2c (HEAD)

Benchmark	Statistic	before	after	Ratio	Alert
ACA (reduced)	warm solve + simulate (reuses compiled code)	43.816 s	317.155 s	7.24	❌
	cold solve + simulate (first run, includes compilation)	791.91 s	2192.00 s	2.77	❌
	peak CPU mem	7.54 GB	11.89 GB	1.58	❌
Mahler-Yum	warm solve + simulate (reuses compiled code)	5.853 s	7.589 s	1.30	❌
	cold solve + simulate (first run, includes compilation)	18.37 s	35.04 s	1.91	❌
	peak CPU mem	1.69 GB	2.28 GB	1.35	❌
	peak GPU mem: automatic solve + simulate	1.61 GB	436 MB	0.27
	peak GPU mem: load saved solution + simulate	1.89 GB	837 MB	0.44
	peak GPU mem: solve + save all persistable artifacts	571 MB	571 MB	1.00
Precautionary Savings - Solve	execution time	259.9 ms	85.0 ms	0.33
	first call (including compilation)	1.55 s	1.71 s	1.10	❌
	peak CPU mem	1.44 GB	1.45 GB	1.01
Precautionary Savings - Simulate	execution time	900.0 ms	997.7 ms	1.11	❌
	peak GPU mem	324 MB	640 MB	1.98	❌
	first call (including compilation)	2.91 s	3.99 s	1.37	❌
	peak CPU mem	1.59 GB	1.58 GB	0.99
Precautionary Savings - Solve & Simulate	execution time	866.3 ms	843.7 ms	0.97
	peak GPU mem	1.50 GB	3.74 GB	2.49	❌
	first call (including compilation)	4.07 s	5.54 s	1.36	❌
	peak CPU mem	1.54 GB	1.55 GB	1.00
Precautionary Savings - Solve & Simulate (irreg)	execution time	997.5 ms	972.7 ms	0.98
	peak GPU mem	2.20 GB	3.74 GB	1.70	❌
	first call (including compilation)	4.20 s	5.80 s	1.38	❌
	peak CPU mem	1.54 GB	1.55 GB	1.00
Iskhakov et al. (2017) - GridSearch Solve	execution time	363.4 ms	197.5 ms	0.54
	peak GPU mem	120 MB	120 MB	1.00
	first call (including compilation)	2.81 s	2.66 s	0.95
	peak CPU mem	1.57 GB	1.63 GB	1.04
Iskhakov et al. (2017) - DC-EGM Solve	execution time	651.1 ms	827.1 ms	1.27	❌
	first call (including compilation)	6.13 s	6.76 s	1.10	❌
	peak CPU mem	1.85 GB	1.96 GB	1.06
Iskhakov et al. (2017) - GridSearch Simulate	execution time	1.011 s	1.058 s	1.05
	peak GPU mem	892 MB	890 MB	1.00
	first call (including compilation)	6.54 s	7.30 s	1.12	❌
	peak CPU mem	1.70 GB	1.80 GB	1.06
Iskhakov et al. (2017) - DC-EGM Simulate	execution time	1.115 s	1.625 s	1.46	❌
	peak GPU mem	891 MB	891 MB	1.00
	first call (including compilation)	6.84 s	10.02 s	1.47	❌
	peak CPU mem	1.92 GB	2.03 GB	1.06
Collective Household - Construct	execution time	735.8 ms	1.512 s	2.06	❌
	peak CPU mem	1.40 GB	1.43 GB	1.02
Collective Household - Solve	execution time	481.2 ms	5.473 s	11.37	❌
	first call (including compilation)	2.88 s	44.68 s	15.50	❌
	peak CPU mem	1.46 GB	2.75 GB	1.88	❌
Collective Household - Simulate (1000)	execution time	1.189 s	39.990 s	33.64	❌
	first call (including compilation)	14.59 s	601.97 s	41.26	❌
	peak CPU mem	1.48 GB	27.69 GB	18.71	❌
Collective Household - Simulate (100000)	execution time	1.220 s	nan µs	nan
	peak GPU mem	189 MB	nan MB	nan
	first call (including compilation)	15.33 s	nan s	nan
	peak CPU mem	1.56 GB	nan MB	nan
Reference Chain - Solve (1)	execution time	217.7 ms	40.4 ms	0.19
	first call (including compilation)	1.76 s	1.79 s	1.02
	peak CPU mem	1.12 GB	1.19 GB	1.06
Reference Chain - Solve (8)	execution time	492.7 ms	271.9 ms	0.55
	first call (including compilation)	5.80 s	6.44 s	1.11	❌
	peak CPU mem	1.23 GB	1.33 GB	1.09
SimulationDispatch ('dissolution', 'off')	warm simulation time per period and regime (ms)	45.77 ms	88.00 ms	1.92	❌
	backend compile requests on second simulation call		0
SimulationDispatch ('dissolution', 'progress')	warm simulation time per period and regime (ms)	44.76 ms	87.11 ms	1.95	❌
	backend compile requests on second simulation call		0
SimulationDispatch ('multi_regime', 'off')	warm simulation time per period and regime (ms)	49.92 ms	65.56 ms	1.31	❌
	backend compile requests on second simulation call		0
SimulationDispatch ('multi_regime', 'progress')	warm simulation time per period and regime (ms)	48.43 ms	62.05 ms	1.28	❌
	backend compile requests on second simulation call		0