Skip to content

Commit 604ad59

Browse files
committed
docs: describe MQTT health management in the device architecture
The health management section described local connection recovery as the HealthManager and stated that A01/B01 devices don't need one. Split it into local connection recovery and MQTT health management, covering the restart conditions and pacing and the mqtt_health diagnostics.
1 parent 0cc13e1 commit 604ad59

1 file changed

Lines changed: 43 additions & 7 deletions

File tree

‎docs/DEVICES.md‎

Lines changed: 43 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -496,18 +496,54 @@ For each V1 command:
496496
| **Channel Class** | `V1Channel` | `MqttChannel` directly |
497497
| **RPC Abstraction** | `RpcChannel` with strategies | Helper functions |
498498
| **Strategy Pattern** | ✅ Multi-strategy (Local → MQTT) | ❌ Direct MQTT only |
499-
| **Health Manager** | ✅ Tracks local/MQTT health | ❌ Not needed |
499+
| **MQTT Health Reporting** | ✅ MQTT requests | ✅ A01 requests; ❌ B01 not yet |
500500
| **Code Location** | `v1_channel.py` | `a01_channel.py`, `b01_q7_channel.py` |
501501

502-
#### Health Management (V1 Only)
502+
#### Local Connection Recovery (V1 Only)
503503

504-
Each V1 RPC strategy can have a `HealthManager` that tracks success/failure:
504+
`V1Channel` keeps the local connection alive in the background:
505505

506-
- **Exponential Backoff**: After failures, wait before retrying
507-
- **Automatic Recovery**: Periodically attempt to restore failed connections
508-
- **Network Info Refresh**: Refresh local IP addresses after extended periods
506+
- **Exponential Backoff**: After a failed local connection, wait before retrying (1 to 10 minutes)
507+
- **Automatic Recovery**: Check the local connection periodically and reconnect when it drops
508+
- **Network Info Refresh**: Refresh the device's local IP address after 12 hours
509509

510-
A01/B01 devices don't need health management since they only use MQTT (no fallback).
510+
#### MQTT Health Management
511+
512+
The broker can stop delivering one device's messages while the MQTT connection itself stays up and
513+
other devices on the shared session keep working. The client keepalive can't detect this, so the
514+
outcome of MQTT requests is tracked instead. Subscribing to the device's topic again on the same
515+
connection doesn't bring its messages back; a new connection does.
516+
517+
The session's `HealthManager` receives the result of every MQTT request through a per-device
518+
`DeviceHealthManager`, and restarts the whole session (a new connection with all subscriptions
519+
re-established) when:
520+
521+
- **One device fails**: 3 of its requests time out within 30 minutes. A single response in between
522+
doesn't reset the count, because a failing connection may still deliver some of them. The device
523+
counts as recovered after 3 responses in a row, or at its first response once its timeouts are
524+
older than 30 minutes.
525+
- **The session fails**: 3 requests in a row time out across more than one device.
526+
527+
Only timeouts on the current connection count: every new connection, whether forced or after the
528+
session reconnected on its own, starts counting from zero.
529+
530+
Restarts are paced so that a device that is really offline doesn't keep interrupting the others:
531+
532+
- At most one restart every 30 minutes.
533+
- While the same device keeps failing, the interval between the restarts it triggers doubles
534+
(30 minutes, 1, 2, 4 hours, then every 6 hours). It is reset once the device answers 3 requests
535+
in a row, so a device that answers once after every restart and then fails again still backs off.
536+
- Timeouts while the session is already reconnecting don't restart it.
537+
- A forced restart reconnects right away; other disconnects keep the exponential backoff.
538+
539+
V1 devices answer most requests over the local connection, so their MQTT topic may carry no traffic
540+
for hours. A broken topic is then noticed when the map (only available over MQTT) is requested
541+
again, and the session is restarted after the first few map requests time out.
542+
543+
Each device's diagnostics include an `mqtt_health` section: when it started failing, its last
544+
success and timeout, request counts, and the last 10 health events (newest first): `timeouts_started`,
545+
`session_restart` (with `cause`: `this_device`, `other_device` or `several_devices`) and `recovered`.
546+
B01 devices don't report their requests yet, so they have no `mqtt_health` section.
511547

512548
### Protocol Versions
513549

0 commit comments

Comments
 (0)