Skip to content

DEBUGGING.md

Latest commit

 

History

History
153 lines (119 loc) · 7.58 KB

DEBUGGING.md

File metadata and controls

153 lines (119 loc) · 7.58 KB
Raw

Debugging Runbook

A task-oriented index for debugging a pod in the field: symptom → first checks → where the authoritative detail lives. This does not restate the ADRs and topic docs — it points at them. The desktop Diagnostics console (/debug) surfaces most of these signals live.

Getting onto a pod

Pods run sshd on port 8822 (key-only), not 22.

ssh -p 8822 root@<pod-ip>
journalctl -u sleepypod.service -n 200 --no-pager   # core app
journalctl -u frank -n 200 --no-pager               # firmware: what the DAC actually did

For a support handoff, sp-bundle-logs includes the firmware journal, a combined firmware/app timeline, and pod model, firmware revision/build date, and SleepyPod build metadata. sp-info prints that compact context; sp-status and sp-logs include it too. The model comes from the local device-status API; when hardware is unreachable it is reported as unknown, so include the model manually.

To capture an install incident even after a reboot, select its UTC time window:

sp-bundle-logs --since '2026-10-05 19:50:00 UTC' --until '2026-10-05 20:05:00 UTC'
sp-logs --firmware --since '2026-10-05 19:50:00 UTC' --until '2026-10-05 20:05:00 UTC'

Explicit windows search all retained boots. Without a window, bundles capture the current boot and sp-logs --firmware follows app and firmware logs together. On older installs without these options, use:

journalctl -u frank.service -u sleepypod.service --since '2026-10-05 19:50:00 UTC' --until '2026-10-05 20:05:00 UTC' --no-pager

SSH setup and hardening: scripts/README.md, docs/DEPLOYMENT.md.

Data paths

What Path Authoritative doc
App database /persistent/sleepypod-data/ docs/DEPLOYMENT.md
RAW frames (hot, tmpfs) /persistent/biometrics/*.RAW docs/adr/0018-tmpfs-raw-frames.md
RAW frames pinned for archiving /persistent/biometrics/.pending/*.RAW (hard links) docs/adr/0018-tmpfs-raw-frames.md
RAW archive (cold, eMMC) /persistent/biometrics-archive/*.RAW.gz docs/adr/0012-biometrics-module-system.md
DAC socket /persistent/deviceinfo/dac.sock (Pod 5) · /deviceinfo/dac.sock (Pod 3/4) docs/DEPLOYMENT.md

RAW_DATA_DIR (default /persistent) overrides where readers look for RAW frames — see src/streaming/piezoStream.ts and src/server/routers/raw.ts.

Biometrics empty / "nothing is being written"

The Biometrics page in the console shows a live data-flow banner — if it reads red/amber while the bed is occupied, the ingest pipeline has stalled.

  1. Open Settings → Device → Sensor data source first. It names the firmware generation (NATS JetStream vs the .RAW variants), which transport the core picked at startup, and the age of the last frame. A note under the rows calls out a fallback or an env override. This is the in-app equivalent of the biometrics pipeline section of sp-status (docs/nats-frame-readers.md → "Surfacing the selection in the app"). On NATS firmware, stop here: there are no .RAW files to inspect, and an empty /persistent/biometrics is expected.
  2. Confirm RAW_DATA_DIR matches the tmpfs hot dir (/persistent/biometrics, per ADR-0018). A mismatch makes readers see an empty directory while frank writes fine — the classic silent failure (docs/sleep-detector.md §9).
  3. ls -la /persistent/biometrics/*.RAW — are files fresh (mtime moving)?
  4. journalctl -u frank — is the firmware writing at all?

Deep detail: docs/adr/0012-biometrics-module-system.md, docs/adr/0018-tmpfs-raw-frames.md.

RAW archive is empty / archived=0 on every run

Past nights can only be replayed (e.g. against the piezo-processor beat detector) if frames reached /persistent/biometrics-archive/. The firmware rotates *.RAW every ~15 min and unlinks the finished frame within a second, so the archive depends on the linker pinning each frame with a hard link first (ADR-0018).

  1. sp-status — biometrics-archiver units must show sleepypod-biometrics-linker.timer active, and .RAW pinned (.pending) should be 1 while the firmware is writing.
  2. journalctl -u sleepypod-biometrics-linker -n 50 --no-pager — the linker is quiet on a no-op run; linked= lines appear once per rotation. dropped= means pinned frames were discarded because the archiver is not draining.
  3. journalctl -u sleepypod-biometrics-archiver -n 50 --no-pager — expect archived=1 roughly once per rotation. failed= means gzip could not write to eMMC (check df -P /persistent); a stuck pending= with archived=0 means frames are pinned but not being released.
  4. ls -la /persistent/biometrics /persistent/biometrics/.pending — one live frame with a moving mtime, plus its pin.

Deep detail: docs/adr/0018-tmpfs-raw-frames.md ("Rotation deletes the finished frame").

Pump stalled but the side reads as "powered"

The Thermal page now plots per-side target / bed / water trends. A stalled pump shows as water + bed flatlining away from target while the side still reports powered. The guard logic lives in src/hardware/pumpStallGuard.ts (surfaced as the stalled verdict via health.thermal).

HomeKit “leak” notification immediately after startup

The HomeKit Pod pump left/right accessories expose pump-stall notices through LeakSensor; the Home app's leak wording does not identify a water leak. Check journalctl -u sleepypod.service for [pumpStallGuard] and compare the firmware's pump duty / on-off commands in journalctl -u frank.

On startup, saved device_state.isPowered and a retained non-neutral target with a zero countdown are insufficient to establish a running pump. The guard waits for per-side live evidence: a positive session countdown with a non-neutral target, positive pump duty or observed RPM >= 50 outside priming/spin-down with a fresh non-neutral firmware status, or a successfully completed non-neutral temperature command with a positive duration. Unknown-state frames do not accumulate stall dwell. Confirmation is captured when each frame arrives, so queued frames cannot inherit later evidence and wall-clock corrections cannot revoke confirmation. A once-per-side “waiting for live pump/session evidence” log explains this state; missing evidence is not reported as a stall. Unresolved incidents remain blocked, and pending hardware cutoffs still retry.

Web UI down, service failed or restarting in a loop

If sleepypod.service has failed and journalctl -u sleepypod.service ends with No production build at .next/standalone/server.js (older installs restart in a loop with a pnpm permission error instead), the installed build is gone, almost always because next build / pnpm build was run on the pod. Next empties .next before compiling, and the pod's 2GB of RAM can't finish the build. Recover by reinstalling a pre-built bundle:

sp-update                    # on the pod: latest release
./scripts/deploy <pod-ip>    # from a computer: your local checkout

Never build on the pod; next.config.mjs refuses to unless SP_ALLOW_POD_BUILD=1 is set. See docs/DEPLOYMENT.md → Why Build Off-Device.

Database or native-module errors on the pod

The systemd unit hardcodes /usr/local/bin/node — an ad-hoc node on PATH is often the wrong ABI for better-sqlite3 and will fail to load. The installer symlinks the correct binary (scripts/install); if you run scripts by hand, use /usr/local/bin/node explicitly.

Service / firmware reference

Service units, null-routed firmware domains, and the frank/Capybara split are documented in docs/DEPLOYMENT.md.