fix(control): a driver that cannot actuate gets its autonomous default

[frahlg]

Summary

AGENTS.md states the law plainly: a failed/stale driver receives its autonomous default mode. FTW does the stale half well. The failed half was not done at all.

Two cases reached no default. They are the same condition seen from opposite ends of the wire, so they get one mechanism and one latch.

(a) A device fault never reached the default. A driver that flags DeviceFault keeps polling, so WatchdogScan — which keys on LastSuccess alone — saw nothing wrong and produced no transition. The driver left onlineBats, received no command as a result, and held its last accepted setpoint for as long as the fault lasted. Before this PR, DeviceFault had no reader anywhere in go/cmd/ftw outside home_link.go. DeviceFault means "cannot actuate", which is precisely the condition the autonomous default exists for. Three shipped drivers raise it: ferroamp.lua, sungrow.lua, pixii.lua.

(b) A command-refusing driver stayed "healthy". A v1 driver that answers every poll and rejects every command had its error logged and discarded by sendDriverCommand. It kept Status=ok, stayed inside onlineBats and inside the MPC fleet, and the power the plan counted on but never got silently became grid import. #741 gave the registry a per-command restoreAfterCommand, so the device does now get a default write after each refusal — but nothing told the rest of the system to stop counting on it, so the loop ran on forever: dispatch, refuse, default, dispatch, refuse, default, once per control tick.

First commit: one offline surface, not two

Asked separately, before building anything on top.

• DriverHealth.SetOffline() — dead in production. Zero runtime callers; WatchdogScan set h.Status = StatusOffline inline and only tests reached the method. Wired, so staleness has one writer, with a doc comment naming the caller that owns it.
• Store.DriverHealthMut() — not dead. It is the documented test-setup escape hatch ("Runtime code should use the Store RecordDriver* helpers"), used by ~50 tests. Kept.

No behaviour change in that commit.

The mechanism

driverActuationTracker (go/cmd/ftw/driver_failure_default.go) copies the shape of staleSiteDefaultTracker: send the declared default exactly once per transition, re-arm on recovery. Both (a) and (b) are inputs to it.

Exclusion needs no new path. Both conditions land in telemetry as DeviceFault, so IsOnline() — the predicate ComputeDispatch and onlineFleetParams already share — drops the driver from dispatch and from the plan, and every existing operator surface (/api/health "fault", the driver inventory, the support report) reports it without a line changed.

DeviceFault becomes derived from two sources that cannot overwrite each other. This is the one piece of new structure, and it earns its place: a driver re-asserts its own view on every poll — sungrow.lua calls set_device_fault(false, "") each time it does not see a fault — so a single shared boolean would have let a driver that believes the device is fine clear a core-set fault on its next poll, flipping the driver in and out of the MPC fleet for as long as the refusals lasted. driverFault is the driver's verdict, commandFault is core's; DeviceFault is their OR, and the driver's reason wins when both are set because it saw the device.

Refusals: three in a row, matching RecordError's existing degrade threshold — one rejected Modbus write is a normal event on a busy device, three in a row at control cadence is not. One command is let through every five minutes so an inverter that rejects writes through a firmware restart comes back without an operator; a single fresh refusal puts it straight back out.

Not counted as refusals, because each is a fault another mechanism already owns and double-booking it would push a merely slow driver out of control: ErrObserveOnly (the registry refused on the driver's behalf and never touched the device), ErrControlBlocked (the registry is already holding the driver in its default and retrying with backoff), and deadline/cancel (a wedged driver stops emitting telemetry too, and the staleness watchdog walks it to its default — the same reasoning sendDriverCommand's timeout handling already carries from #791).

What this trades

New commands now reach hardware on a path that previously sent none. A driver reporting a device fault, or one that has refused three commands, now receives driver_default_mode where before it received nothing at all.

That is the intended direction, and the trade is explicit: one write of the driver's own declared safe state, against a battery holding a stale setpoint into a fault indefinitely. A Ferroamp in Fault Mode that was last told to discharge at 5 kW keeps that setpoint until somebody restarts something; a Sungrow that stopped taking writes keeps whatever it last accepted while the plan spends energy it is not delivering. The default is the driver's own declaration, not core's guess at one, and observe-only drivers receive nothing — not even this.

A driver that provably rejects the default release is reported as default held, not as a failure (set_self_consumption in sungrow.lua returns true for an inverter that was never under control). Nothing here escalates on it: only dispatch commands feed the refusal counter, never the default release itself.

Not in scope

A driver stuck returning ErrControlBlocked — because its own default keeps failing — is still counted by the MPC fleet while the registry retries it with backoff. Same "silently counted" shape through a third door. Left out deliberately: the registry owns that state machine, and #798 is currently changing exactly that path. Worth a follow-up once #798 lands.

Verification

• go test ./internal/control/ -run TestGoldenCorpus — no record moved.
• make verify clean (pre-commit gate on both commits); go test -race on cmd/ftw, telemetry, control, mpc, drivers.
• Each new test was checked against a mutation of the code it covers: removing the exclusion fails the (b) tests, removing the latch fails the (a) test, collapsing the two fault sources fails the poll/command-fault test.
• No overlap with any open PR's files. fix(control): keep autonomous defaults ahead of stale commands #798 touches only registry.go; it decides how a default gets through, this decides when one is owed.

Tests

• driver_failure_default_test.go — device fault reaches the default once per transition and re-arms; a stale driver is left to the watchdog; observe-only receives nothing; refusals exclude and default once; the retry window re-admits, re-excludes on a fresh refusal, and clears on an accepted command; non-refusal errors never exclude; a removed driver is forgotten and never resurrected.
• internal/control — a battery that refuses commands gets no dispatch target and the load is reallocated to its healthy sibling.
• internal/mpc — a refusing battery is dropped from onlineFleetParams: its capacity and its SoC leave the plan.
• internal/telemetry — a driver's own poll cannot clear a core-set command fault; the two sources are independent; a command fault never creates a health record.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9b72b7aae4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Count refusals that also report blocked recovery

When a command is refused and the registry's automatic default restoration also fails, restoreAfterCommand returns a joined error that matches both the underlying refusal and ErrControlBlocked. This early case therefore discards the initial refusal, and every later dispatch returns bare ErrControlBlocked and is discarded as well. If the driver keeps polling, it never reaches the refusal threshold, remains online in dispatch and MPC, and the plan continues relying on a device for which control is blocked indefinitely.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Track refusal outcomes from EV and PV dispatch too

The tracker is wired only around the storage finalTargets loop. In the inspected control tick, loadpoint commands still go through lpController.TickWithDispatch using reg.Send, and PV curtailment still calls sendDriverCommand directly, so chargers and PV-only drivers that answer polls but reject every command remain online and continue contributing to plans indefinitely. Route those dispatch results through the same tracker (or expose an equivalent outcome hook) so the advertised unable-to-actuate behavior applies to every controlled driver.

Useful? React with 👍 / 👎.