optimizer: preserve plan semantics through dispatch and prove realized value

[frahlg]

Problem

The optimizer's core equations and Core-side replay are sound, but the full path from config and solver output to device dispatch does not preserve several decisions and hard limits. Further solver tuning cannot fix plans that runtime changes or metrics that cannot attribute optimizer value.

This issue records a read-only audit of checkout 8ca77904 and fetched origin/master c9c7919e. The relevant logic was the same on both.

Confirmed failures

1. Per-storage plans are validated, then discarded

The worker returns storage_power_w and storage_energy_wh, and Core validates each storage. mpc.SlotDirective then carries only aggregate fleet energy. Control creates a new split by mode, capacity, limits and PV locality.

A deterministic reproduction planned a feasible 100 W + 1900 W = 2000 W split. Runtime produced 100 W + 1000 W = 1100 W.

Impact:

• planned and delivered SoC paths differ;
• cost and terminal value use energy runtime may not deliver;
• a heterogeneous fleet may hit driver clamps although the optimizer plan passed validation.

2. Explicit zero power limits change meaning

MPC treats batteries.<id>.max_charge_w: 0 as "forbid charge". driverLimitsFrom only copies positive battery limits. Control treats zero as unset and falls back to MaxCommandW = 5000 W.

A config-to-control reproduction produced a +5000 W charge target from an explicit zero charge limit. Discharge has the same fault.

Related: #816 covers a separate slew/non-following-device failure that also exposes inconsistent zero/default handling.

3. PV-only export can exceed the declared hard ceiling

The fuse/export guard can scale battery discharge. If PV alone causes the overage, it returns unchanged targets and depends on a planner curtail directive. ComputePVCurtail does not react directly to fresh meter export above min(fuse, MaxExportW).

4. Full PV curtailment cannot be represented

pv_limit_w = 0 means both:

• no curtailment; and
• an active cap of zero watts.

Python, Go fallback, Core validation and driver dispatch all collide on this sentinel.

Reproduction: PV -5000 W, load 0 W, full battery, export price -100 ore/kWh. Python chooses zero grid flow and serializes a zero PV limit. Core interprets that as no cap, replays -5000 W grid flow and rejects the plan. Go fallback also cannot emit the full-curtail action.

5. An older replan can replace newer operator intent

Service.replan takes a params snapshot and always publishes its result. It has no request generation or latest-wins guard.

A deterministic test started a blocked self_consumption solve, completed a later SetMode(arbitrage) solve, then released the old solve. The old plan became active last.

6. Solver deadlines and worker queue ownership disagree

Core sends 0.8 * optimizer timeout. Python gives that full limit separately to service and economic solves. Total solver time can approach 1.6 * Core timeout, before model build and queue time.

The Unix worker starts one thread per connection but serializes solves with one global lock. A request that Core has already abandoned can finish and block newer requests.

7. USER_LIMIT is accepted as an optimal status

CVXPY USER_LIMIT can return ok: true without a proven service optimum. Core catches many physical inconsistencies, but a physically valid incumbent may still miss a feasible EV target or be much more expensive.

8. Mode-specific economic terms leak across modes

The Go contract applies min_arbitrage_spread_ore_kwh only to arbitrage modes. Python adds it to discharge cost in every mode.

Reproduction in self_consumption, 2 kW load and 80% SoC:

• spread 0: about 2050 W discharge;
• spread 100: no discharge.

Explicit relaxed can also allow simultaneous charge/discharge under unsafe tariffs and then fail replay.

9. Planner config can give Python and Go fallback different physics

Config validation does not reject:

• soc_min_pct >= soc_max_pct;
• efficiency above 1;
• invalid per-battery SoC windows;
• invalid weights.

Python can reject the request while Go DP accepts and uses the same bad values. batteries.<id>.weight is currently not wired to control.

10. Current savings and backtest output cannot prove optimizer value

/api/savings/daily compares actual grid cost with a no-PV/no-battery baseline. It combines PV value, battery value, export revenue and EV timing. It is not incremental optimizer value.

The production cost integrator also applies the next sample across arbitrarily long telemetry gaps. A separate, currently unused savings integrator already rejects gaps over 20 minutes and reports coverage.

The optimizer backtest:

• sums overlapping plan horizons;
• may select a plan produced after a measured bucket;
• assumes planned battery power was delivered;
• ignores PV curtailment in realized grid replay;
• does not link plan revision, clamps, command results and measured per-device power.

Input and model gaps to measure after correctness fixes

• forecast.solar uses the maximum sample in an hour as the full-hour power;
• every forecast price gets fixed confidence 0.6 although the model stores sample counts and MAE;
• PV scenarios use fixed probabilities 0.60/0.25/0.15, one absolute spread, identical load and no price paths;
• local household and Nordic market time features use UTC buckets across CET/CEST;
• per-device efficiency, degradation cost and capability are not fed through the live planner contract.

Order of work

PR 1: preserve explicit zero battery limits

• Keep unset and explicit zero distinct through config, PowerLimits and every clamp.
• Add config-to-dispatch tests for charge and discharge.
• Keep Slew anchored on actual output + MaxCommandW default pins the controller into commanding more charge than it wants, indefinitely, when a device stops following #816's separate slew fault out of scope.

PR 2: make export protection and curtail state explicit

• React to fresh PV-only export above the effective ceiling.
• Separate curtail active from limit_w.
• Validate and dispatch an active zero cap without converting it to release.
• Define driver support for a true zero cap or a documented minimum safe cap.
• Test partial cap, full cap, release, unsupported PV and residual uncontrollable export.

PR 3: make replans and solver work latest-wins

• Add a monotonic replan generation and commit only the newest request.
• Snapshot reason and mode with the request.
• Use one absolute deadline across queue, model build, service solve, economic solve and response.
• Drop expired/superseded work before it enters the solve lock.
• Do not accept USER_LIMIT without a validated feasible incumbent and service proof.

Follow-up: execute the per-storage plan

• Carry per-device Wh budgets through SlotDirective.
• Use runtime redistribution only for a device failure or hard clamp.
• Feed delivered per-device energy back into the next replan.

Follow-up: prove realized value

Persist and link:

• plan and model revision;
• forecast issue/source/revision;
• Core-validated target;
• clamp reason;
• driver command result;
• measured grid, PV, load, EV, battery power and SoC;
• data coverage.

Run leak-free walk-forward scoring one executed first action at a time. Compare no battery, simple self-consumption, Go DP, active Python/challenger and perfect-foresight lower bound with the same exogenous data and terminal energy.

Acceptance gates

• No configured hard limit changes meaning across layers.
• The plan Core validates is the plan control attempts to deliver.
• Older work cannot replace newer operator intent.
• Timed-out work cannot block a newer solve.
• Every accepted plan passes independent physical replay and required service checks.
• Optimizer gain is reported separately from PV and hardware value.
• Missing data lowers coverage instead of creating energy.
• A deterministic full-stack test runs optimizer -> validator -> dispatch -> driver simulator -> measured SoC/cost.

Current evidence

• 66 optimizer tests passed.
• go test -race ./internal/mpc ./internal/control passed outside the sandbox.
• 611 golden dispatch records passed.
• Temporary targeted tests reproduced the fleet split, explicit-zero and stale-replan faults.
• No existing test runs the complete optimizer-to-realized-cost path.

Existing work with right of way

• feat(web): turn a driver's write path on from Settings #769 touches go/cmd/ftw/main.go and go/internal/control/dispatch.go.
• perf(optimizer): use direct HiGHS for shared storage #883 owns direct HiGHS and optimizer/ftw_optimizer/model.py; its active review thread covers shared post-curtailment baseline parity.
• feat(roofmodel): optional shadow-aware irradiance via vostok #736/feat(roofmodel): derive PV array geometry from Lantmäteriet open geodata #735/feat(weather): add STRÅNG as an irradiance source and declare source coverage #734 touch go/internal/mpc/external_optimizer.go; they are draft/dirty.
• fix(config): reject >3 fuse phases and duplicate site meters #844 and older PRs own current config.go work.
• fix(control): keep autonomous defaults ahead of stale commands #798 owns the stale-driver autonomous-default path and still needs its separate human safety review.

Implementation PRs must stack on or wait for those owners rather than replace their changes.

[frahlg]

Implementation has started in three review-sized draft PRs:

• fix(control): preserve explicit zero battery limits #887 preserves explicit one-sided 0 W battery limits through config and every runtime clamp.
• fix(control): mitigate PV-only export over limits #888 adds partial reactive PV-only export mitigation and exposes any reduction that safe positive caps cannot cover.
• fix(mpc): keep the newest replan #889 makes replan publication latest-wins and keeps plan mode/reason/diagnostics paired.

#887 and #888 are stacked on the exact reviewed head of #769 because they overlap its Core files. Retarget them to master after #769 lands. #889 targets current master.

Still open here: an explicit active-vs-release PV curtail contract with driver zero/minimum capability, solver cancellation and one shared deadline, per-storage dispatch, and the closed-loop measurement work.

[frahlg]

Implementation update: four more review-sized draft PRs are now open.

• fix(optimizer): enforce result and mode semantics #890 fixes optimizer mode semantics and rejects unproven CVXPY USER_LIMIT results. It targets master, is mergeable, and CI is green.
• fix(optimizer): bound queued solves to one deadline #893 bounds queued and active solver work to one monotonic worker deadline. It is stacked on fix(optimizer): enforce result and mode semantics #890 because it needs the same solver files. Review its incremental diff, then retarget it to master after fix(optimizer): enforce result and mode semantics #890 lands. The custom stack base does not trigger GitHub checks; local make verify, make verify-all, MPC race tests, Core↔Python tests and 112 Python tests pass.
• fix(savings): reject long telemetry gaps #892 stops telemetry gaps over 20 minutes from creating energy, cost or savings; adds weighted history/price coverage; and labels saved_ore as total site value rather than optimizer gain. It targets master, is mergeable, and CI including Core/full-stack is green.
• fix(optimizer): make backtest first actions causal #891 makes backtest first-action scoring causal and non-overlapping. It excludes unmodeled PV curtailment and labels the result as an independent grid-cost counterfactual, not delivered savings. It targets master, is mergeable, and CI is green.

#883 has landed, so its former file-ownership block is resolved through #890/#893.

Suggested review order:

1. fix(optimizer): enforce result and mode semantics #890, then fix(optimizer): bound queued solves to one deadline #893.
2. fix(savings): reject long telemetry gaps #892 and fix(optimizer): make backtest first actions causal #891 independently.
3. Keep fix(control): preserve explicit zero battery limits #887/fix(control): mitigate PV-only export over limits #888 stacked behind feat(web): turn a driver's write path on from Settings #769 as already documented.
4. After fix(mpc): keep the newest replan #889 and the dispatch stack settle, carry per-storage plans into runtime and add the plan → validation → clamp → command-result → measured-power linkage.

Still not solved: a transport-independent absolute deadline field across every Core transport, explicit active-zero PV curtail capability in the canonical drivers, per-storage execution, and true closed-loop optimizer value attribution. No PR has been merged by this implementation pass.

[frahlg]

Post-merge and beta status

#890–#894 are merged. The final master commit is f9297c06073836cca2a629cfa775ded9d6984731:

• fix(optimizer): enforce result and mode semantics #890: mode and USER_LIMIT semantics
• fix(optimizer): make backtest first actions causal #891: causal, non-overlapping first-action backtest
• fix(savings): reject long telemetry gaps #892: telemetry-gap rejection and coverage
• fix(optimizer): bound queued solves to one deadline #893: one bounded queue/solve deadline
• perf(app): reduce uplink, history, and background work #894: app/uplink performance, including bounded dashboard poll reads

Two prereleases were published from that exact commit:

• Core v1.16.1-beta.19, run 31468877785
  • Core digest: sha256:513be2b6b4b11cb7ce2cef98c827bd15826681ce98ce3ce9d41248edb039dc53
  • Updater digest: sha256:993114f6c56bd01f2bcd687160b506dd9ec84d0938862ae124b3ebb89ef2f85f
• Optimizer optimizer-v1.4.0-beta.2, run 31469463012
  • Optimizer digest: sha256:fe92b7cf965c9380dba3de00980f4f05d2c272876709606526e7cb3d23aee890

The fixed tags and :beta aliases match. The Core and updater canonical/compatibility mirrors also match. All images contain linux/amd64 and linux/arm64 manifests whose revision labels resolve to f9297c0.

Verification passed on the exact release commit:

• local make verify-all: 117 Python tests, all Go packages, Core↔Optimizer contract tests, compose migration, e2e, vet/build, and amd64/arm64/Windows cross-builds;
• fresh PR checks before each merge;
• optimizer release: Python suite, self-update version contract, Core↔Optimizer multi-device contract, and multi-arch image build.

Master push run 31468510104 is red only because the registry job runs gh pr view "" during a push event. Core Go test/vet, web, full stack and the Device Support contract are green; the final umbrella job only propagates that context error. Registry blobs also match the merged webapp. This is a workflow context fault, not a code regression.

#824 was not merged or used. It remains open and blocked. No stable promotion happened. Publication is verified here; real-site beta validation is still outstanding.

Next:

• update draft fix(mpc): keep the newest replan #889 onto current master, rerun full checks and review latest-wins publication;
• keep fix(control): preserve explicit zero battery limits #887/fix(control): mitigate PV-only export over limits #888 behind feat(web): turn a driver's write path on from Settings #769 until its hardware/driver pairing is approved;
• then add superseded-replan cancellation on top of fix(mpc): keep the newest replan #889;
• retain the remaining open gates: active-zero PV curtail capability, transport-independent deadline propagation, config physics validation, per-storage execution, and plan → validation → clamp → command result → measured power/SoC linkage.

Issue #885 stays open until those acceptance gates and real-site beta checks pass.

[frahlg]

Latest-wins Core work is merged

• fix(mpc): keep the newest replan #889 merged as 6e23450f47df23e5a023477d034089c4ab8ed741. It pairs mode, reason, plan and diagnostics under one generation guard, so an older solve cannot replace newer operator intent.
• fix(mpc): cancel superseded replans #895 merged as 5213dd5916a5a1513114cb2676e96a1793562375. A newer generation or Stop now cancels Core's active replan request, skips stale Go fallback/shadow/publication/diagnostic work, rejects new work during shutdown, and waits for all accepted replans before Optimizer.Close.

#895 keeps caller cancellation separate from service cancellation. Caller deadlines still reach the external optimizer and retain the prior safe Go-DP fallback. Supersession and Stop set the service-cancel flag before canceling the context, so stale work cannot take that fallback path.

Verification on the exact #895 head:

• five controlled concurrency tests passed 20 consecutive runs with the race detector; independent audit repeated them 50 times;
• full MPC, API and process-wiring race suites passed;
• make verify-all passed: 117 Python tests, all Go tests, Core to Optimizer contracts, compose migration, end-to-end tests, vet/build, and Linux arm64/amd64 plus Windows amd64 cross-builds;
• all PR checks passed, with no review comments or inline threads; three independent final reviews found no P0-P3 issue.

Post-merge run 31474732641 has green Core, Optimizer contract, full-stack and Device Support jobs. Its only primary failure is the known push-context bug: the registry job runs gh pr view "" on master; the umbrella go test + vet job only propagates that failure. This is not a code failure.

Remaining deadline/cancellation boundary: Core now stops waiting and doing stale follow-on work, but an already active native solve in the Unix optimizer sidecar continues while Python holds SOLVE_LOCK, until its existing solver deadline. A newer request can still wait behind it. The next solver-lifecycle slice needs worker-side active-solve supersession or an isolated solve process that can be stopped safely, plus an end-to-end test proving request 2 starts before request 1's old deadline.

Issue #885 remains open. #824 remains unmerged and was not used.

[frahlg]

Active cancellation follow-up merged

#896 is merged as 16ddcf88940020767c9274a826d37a74323ffd5e.

This closes the next bounded part of the stale-work problem:

• Core sends a request-scoped cancel_request frame when a sent Unix optimizer request is superseded, canceled or reaches its deadline.
• The worker can wake and remove queued work before it starts.
• The current primary direct-HiGHS path uses HiGHS user interrupts for active solves.
• Canceled work cannot fall through to CVXPY or write a stale response.

Verification before merge:

• all PR checks passed;
• make verify-all passed;
• 134 optimizer tests passed;
• all Go packages, Core↔Optimizer contract, Compose migration, full-stack e2e, vet, build and cross-builds passed;
• the new Unix transport cases passed 20 times under race testing;
• the cancel wire test passed 100 times;
• worker and direct-HiGHS cancel tests each passed 50 times.

Post-merge run 31478951607 used the exact merge SHA. Core, optimizer, full stack and Device Support contract jobs passed. The run is red only because the existing registry job executes gh pr view "" on a master push; the umbrella job then reports that failed dependency. This is a workflow-context bug, not a code or test regression.

Scope remains narrow: active native interruption is implemented for direct HiGHS. CVXPY/CLARABEL still stop only at existing checkpoints/deadlines, while stale result suppression remains in force. Full CVXPY cancellation needs process isolation and stays open as a separate item. No release or real-site validation was performed for this merge.