diff --git a/.github/workflows/python-modules.yml b/.github/workflows/python-modules.yml index 426e2c237..1704d5073 100644 --- a/.github/workflows/python-modules.yml +++ b/.github/workflows/python-modules.yml @@ -8,6 +8,7 @@ on: - 'modules/**/pyproject.toml' - 'modules/**/uv.lock' - 'modules/.coveragerc' + - 'scripts/**/*.py' - '.github/workflows/python-modules.yml' pull_request: types: [opened, synchronize, reopened] @@ -16,6 +17,7 @@ on: - 'modules/**/pyproject.toml' - 'modules/**/uv.lock' - 'modules/.coveragerc' + - 'scripts/**/*.py' - '.github/workflows/python-modules.yml' workflow_dispatch: @@ -76,6 +78,13 @@ jobs: target: . cov_source: . deps: "" + # Stdlib-only repo scripts; tests load them by path from + # scripts/tests/. + - name: scripts + workdir: scripts + target: tests + cov_source: . + deps: "" steps: - name: Checkout @@ -100,7 +109,9 @@ jobs: uv sync --locked --no-install-project else uv venv .venv - uv pip install --python .venv/bin/python ${{ matrix.module.deps }} + if [ -n "${{ matrix.module.deps }}" ]; then + uv pip install --python .venv/bin/python ${{ matrix.module.deps }} + fi fi # shellcheck source=/dev/null . .venv/bin/activate diff --git a/docs/research/sleep-stage-calibration/PLAN.md b/docs/research/sleep-stage-calibration/PLAN.md new file mode 100644 index 000000000..3813f08d7 --- /dev/null +++ b/docs/research/sleep-stage-calibration/PLAN.md @@ -0,0 +1,99 @@ +# Sleep-stage calibration against Apple Watch — plan + +Status: draft 2026-10-10. Owner: Jonathan. Each workstream below is scoped so one +agent can own it end to end. Workstreams A–D are independent; E depends on C. + +## 1. Recorded findings + +Source: Health export `/Users/ng/Desktop/apple_health_export/export.xml` joined per +minute with `vitals` + `movement` from the Pod 5 `biometrics.db` +(192.168.1.88:8822, `/persistent/sleepypod-data/biometrics.db`, timestamps in +seconds). `scripts/sleep-stage-dataset.py` does the join and reproduces the +first three table rows within 1 point with `--summary` (it labels by exact +Watch segment times, so 10-08 prints 52/32/68); `feat2.py` here runs the windowed +features and leave-one-night-out tree on its CSV. See `README.md`. The +720-combo grid search lived in the original `eval.py` (removed; in git history). + +Usable nights: 2026-10-08/09 and 2026-10-09/10, left side, 587 Watch-labeled vitals samples (about one per minute). +(2026-10-01/02 has Watch only against the right side, HR MAD 12 bpm — excluded.) + +| Variant | 10-08 | 10-09 | +|---|---|---| +| Server as deployed (movement-only, `calibrationQuality` defaults to 0) | 53% | 55% | +| iOS rule set (offline simulation, `calibrationQuality`=1) | 33% | 31% | +| Always "light" | 69% | 57% | +| Best of 720 threshold combos (pooled) | 55% | 57% | + +Why the rules fail: +- Per-minute pod HR ratio for Watch-deep is 0.95–1.04, never < 0.92. The "deep = low HR" rule fires on noise. +- Pod HRV relative to night median: Watch-deep 0.56–0.78×, Watch-REM 1.1–1.4×. The rule "REM = low HRV" is inverted; it labels deep as REM. +- `movement.total_movement` is exactly 0 for 97% of epochs on 10-09 (39% nonzero on 10-08). Wake rules cannot fire. +- Both callers omit `calibrationQuality`, so the server never emits deep/REM: `src/server/routers/biometrics.ts:1217`, `src/components/Sleep/sleepData.ts:255`. +- 126 of 782 `sleep_records` have `sleep_duration_seconds` = 57600 (the `MAX_SESSION_S` 16 h cap) with 0 exits. Vitals stop when the bed empties (0 rows 11:05→18:04 on 10-10), so the cap pads the stage timeline, not the vitals averages. +- Pod HRV is an index of successive differences of 10 s-window IBIs (docs/wiki/topics/piezo-processing.md), not SDNN. Watch SDNN is 3 spot readings/night. No scalar correction is valid. + +Stages are **not stored**. `classifySleepStages` runs at query time over `vitals` + `movement`. A classifier change therefore applies to every past night automatically. + +## 2. Backfill policy (what can and cannot be repaired) + +| Layer | Stored? | Backfill | +|---|---|---| +| Sleep stages | No (derived at read) | Automatic once classifier changes. Nothing to migrate. | +| `sleep_records` 16 h cap | Yes | One-off repair, `scripts/repair-capped-sessions.py` (PR #804), for records with `sleep_duration_seconds`=57600. Occupancy evidence is the side's `vitals` only; `movement` is written every minute while a session is open, so it runs to the cap. Vitals are split into runs at gaps > 5 min; occupancy ends at the end of the last run lasting ≥ 30 min. `left_bed_at` = that end + `ABSENCE_TIMEOUT_S` (120 s), never later than the original; duration recomputed, intervals clipped, exits recounted. Records with no qualifying run are left unchanged and listed. Dry run by default; `--apply` first writes `.bak.` with the SQLite backup API. | +| `movement` zeros | Yes | Not recoverable: raw `.RAW` is a rolling single file on tmpfs (Pod 5), nothing archived (`archive-push-staging` empty). `cap_sense_frames` holds ~2 days of per-window zone sums (69k rows, 10-08→10-10) — only those days could be re-derived, not worth it. Forward-only fix. | +| `vitals` HR/HRV/BR | Yes | Not recomputable (raw gone). Keep as is; treat HRV as an index. | +| HealthKit samples written by the iOS app ("sleepypod" source) | In HealthKit | iOS app must delete its own `HKCategoryTypeIdentifierSleepAnalysis` samples for affected nights and rewrite after the server returns new stages. HealthKit permits deleting samples your app wrote. Out of this repo; track in sleepypod-ios. | +| Watch labels | In HealthKit | iOS app can read Watch `SleepAnalysis` and POST to the pod (see D) so the dataset accrues without manual exports. | + +## 3. Workstreams + +### A. Sleep detector: stuck presence / 16 h cap (modules/sleep-detector, python) +Why: 126/782 sessions hit `MAX_SESSION_S`; the stage timeline runs to 17:30 on an empty bed. +What: find why presence stays true after the bed empties on Pod 5 (adaptive baseline, exit fraction, or capSense2 sentinel handling); fix; add the repair script from §2 as `scripts/repair-capped-sessions.py` (dry-run default). +Acceptance: +- Replay test with recorded capSense2 frames ending in an empty bed closes the session within `ABSENCE_TIMEOUT_S`. +- Repair script on a copy of the pod DB rewrites every capped record that has a qualifying vitals run (52 of 126 on the 2026-10-10 copy) and lists the rest unchanged; new `left_bed_at` ≤ end of the last qualifying run + 120 s and ≤ the original; writes `.bak` first; a re-run finds nothing. +- `docs/sleep-detector.md` updated. + +### B. Sleep detector: movement all-zero (modules/sleep-detector, python) +Why: 97% zero epochs on 10-09 makes wake undetectable. +What: determine which stage of the PIM pipeline (pump gate, P5 baseline subtraction, median filter, scale factor) collapses the signal on Pod 5; fix so a quiet night yields the documented distribution (70–80% epochs 0–50, not 97% exactly 0). +Acceptance: +- Unit test with a synthetic still-then-roll sequence produces nonzero scores for the roll and a nonzero floor during stillness. +- One live night on Pod 5 shows ≥ 30% nonzero epochs and ≥ 1 epoch > 200 per bed exit. + +### C. Dataset builder (scripts/, typescript or python) +Why: calibration needs ≥ 20 labeled nights; today the join lives in three scratch scripts. +What: `scripts/sleep-stage-dataset.py --side left --out dataset.csv` emitting one row per pod vitals sample (about one per minute): `night, side, ts, watch_stage, hr, hrv, br, movement, hr_quality` plus `watch_hr` when within 60 s. Reuse `parse.py`/`eval.py` logic. Also `--summary` printing per-night agreement for the current classifier and the always-light baseline. +Acceptance: +- Reproduces the first three rows of the table in §1 for the two nights (±1 point). +- Rejects nights where pod-vs-Watch HR MAD > 8 bpm with a warning (side mismatch guard). +- README in this folder documents how to pull a consistent copy of the DB (SQLite backup API on the pod, or `scp -P 8822 root@:/persistent/sleepypod-data/biometrics.db{,-wal}` while the writers are quiet). +- Python tests run in CI (Python Modules workflow). + +### D. Comparison-data ingestion from the app (src/server, tRPC + iOS stub) +Why: manual Health exports do not scale to 20 nights. +What: tRPC mutation `biometrics.importReferenceStages({side, source:'apple_watch', segments:[{start,end,stage}]})` writing a new `reference_stages` table (migration in `src/db/biometrics-migrations`), idempotent on (side, source, start). The iOS app posts the Watch's `SleepAnalysis` samples each morning. Dataset builder (C) reads this table when no export is given. +Acceptance: +- Migration + Drizzle schema; mutation validates input at the boundary; duplicate posts are no-ops. +- `sleep-stage-dataset.py --from-db` yields the same rows as the export input (the default; there is no `--from-export` flag) for 10-09, once the iOS app has posted that night. + +### E. Classifier v2 (src/lib/sleep-stages.ts) — blocked on C, needs ≥ 20 nights +Why: the current rule family cannot beat always-light. +What: replace `classifyEpoch` with windowed features: 15-min HR std, 15-min mean HRV / night median HRV, breathing-rate std, fraction of night elapsed, movement (once B lands). Start with a small, inspectable model (depth-3 tree or hand thresholds derived from it), same post-processing (smoothing, transition constraints). Pass real `calibrationQuality` from `calibration_profiles` at both call sites; fall back to light/wake-only when below 0.3. +Acceptance: +- Leave-one-night-out agreement ≥ always-light + 10 points on every held-out night, reported in `docs/research/sleep-stage-calibration/RESULTS.md`. +- Cohen's κ reported alongside accuracy. +- Existing `sleep-stages` tests updated; quality score unchanged in shape. +- UI copy: HRV card labels the pod value "HRV index", not ms-SDNN. + +### F. iOS app follow-ups (sleepypod-ios, not this repo) +- Align the comparison window to the intersection of Watch coverage and pod session; show "compared" over that intersection only. +- Delete and rewrite the app's own HealthKit stage samples after E ships. +- Post Watch stages to D each morning. + +## 4. Delegation notes +- A and B share `modules/sleep-detector`; acquire `ygg lock acquire modules/sleep-detector` and sequence B after A, or give both to one agent. +- C and D can run in parallel; D must not change C's CSV schema. +- Do not touch E until C reports ≥ 20 nights with HR MAD ≤ 8 bpm. +- Pod access: ssh port 8822, no `sqlite3` binary on the pod — copy the DB and query locally. diff --git a/docs/research/sleep-stage-calibration/README.md b/docs/research/sleep-stage-calibration/README.md new file mode 100644 index 000000000..718162ea2 --- /dev/null +++ b/docs/research/sleep-stage-calibration/README.md @@ -0,0 +1,132 @@ +# Sleep-stage calibration dataset + +Builds a per-minute dataset that pairs Apple Watch sleep stages with the pod's +vitals and movement, so the stage classifier in `src/lib/sleep-stages.ts` can be +measured and recalibrated. Background and findings: [PLAN.md](PLAN.md). + +## 1. Pull the pod database + +The pod has no `sqlite3` binary, so copy the database off the pod and query +locally. sshd listens on port 8822. + +The pod's services write to the database all the time, so copying +`biometrics.db` and its `-wal` file one after the other is not an atomic +snapshot. A checkpoint between the two reads can leave the copy inconsistent. +Take the snapshot on the pod with SQLite's backup API through the pod's +`python3`, then copy that one file: + +```sh +ssh -p 8822 root@ python3 - <<'EOF' +import sqlite3 +src = sqlite3.connect('file:/persistent/sleepypod-data/biometrics.db?mode=ro', uri=True) +dst = sqlite3.connect('/persistent/sleepypod-data/biometrics.snapshot.db') +src.backup(dst) +dst.close() +EOF +mkdir -p poddb +scp -P 8822 root@:/persistent/sleepypod-data/biometrics.snapshot.db poddb/biometrics.db +ssh -p 8822 root@ rm -f /persistent/sleepypod-data/biometrics.snapshot.db +``` + +The snapshot includes the rows still in the WAL. If you copy the live files +instead (`scp -P 8822 root@:/persistent/sleepypod-data/biometrics.db{,-wal} .`), +do it while the writers are quiet, such as when no one is in bed, and copy +both files in one command. Without the `-wal` file the last few hours are +missing. The script opens the DB read-only. SQLite still reads a WAL if one is +present and may create a `-shm` file beside it. + +## 2. Export Apple Health + +On the iPhone: Health → profile picture → Export All Health Data. AirDrop the +zip to the Mac and unzip it. The script needs `apple_health_export/export.xml` +(or the directory). The file is parsed incrementally with constant memory, so +a 100+ MB export takes a few seconds. + +Only samples whose source name matches `--source` (default `Watch`) are used: +`SleepAnalysis` stages (Core→light, Deep, REM, Awake; InBed and Unspecified are +ignored) and `HeartRate`. + +## 3. Build the dataset + +```sh +scripts/sleep-stage-dataset.py ~/Desktop/apple_health_export poddb/biometrics.db \ + --side left --out dataset.csv --summary +``` + +The shebang runs under `uv`; plain `python3 scripts/sleep-stage-dataset.py …` +works too (stdlib only, Python ≥ 3.9). + +One row per pod vitals sample (about one per minute) inside each night's +window. A window is the Watch's stage coverage padded by `--pad-min` (default +10) on each side. Windows never overlap: sleep that runs past the noon +rollover stays with the night it started in. + +| column | meaning | +|---|---| +| `night` | local date the night started (sample time minus 12 h) | +| `side` | `left` or `right` | +| `ts` | pod vitals timestamp, unix seconds | +| `watch_stage` | `wake`/`light`/`deep`/`rem` of the Watch segment containing `ts`, empty in the padding | +| `hr`, `hrv`, `br` | raw pod vitals, not outlier-filtered. HRV is a pod index, not SDNN | +| `movement` | nearest `movement.total_movement` within 60 s | +| `hr_quality` | `vitals_quality.quality_score` for that vitals row | +| `watch_hr` | nearest Watch heart-rate sample within 60 s | + +Options: + +- `--nights 2026-10-08,2026-10-09` keeps only those nights. +- `--max-hr-mad 8` is the side-mismatch guard. A night where the mean absolute + difference between pod HR and Watch HR exceeds it is skipped with a warning, + because it means the Watch wearer slept on the other side. Nights with fewer + than 10 HR matches are kept with a warning that the guard could not run. +- `--from-db` reads Watch stages from the `reference_stages` table that + `biometrics.importReferenceStages` fills, instead of an export: + `scripts/sleep-stage-dataset.py --from-db poddb/biometrics.db --side left --out dataset.csv`. + That table has no Watch heart rate, so `watch_hr` is empty and the guard + cannot run. Night labels use the machine's local time zone; set `TZ` if it + differs from the pod's. + +## 4. Summary + +`--summary` prints, per night, the agreement with the Watch over Watch-labeled +vitals samples for the ported classifier as deployed (`calibrationQuality` 0, +movement-only), with the iOS rule set (`calibrationQuality` 1), and for an +always-light baseline. Samples arrive about once a minute, so agreement is +weighted by sample, and the per-stage distribution below the table counts +samples, not minutes. The classifier runs over the night window, so its +average HR differs slightly from the server's, which uses the whole sleep +record. + +Output for the two usable nights on 2026-10-10. It is within 1 point of +PLAN.md section 1, which labeled whole minutes instead of exact segment times: + +``` +night side rows labeled HR-MAD deployed(cq=0) iOS(cq=1) always-light +2026-10-08 left 212 202 2.3 52% 32% 68% +2026-10-09 left 405 385 5.4 55% 31% 57% +``` + +The 2026-10-01 night is rejected by the guard on the right side +(HR MAD 11.8 bpm) and has no left-side vitals. + +## 5. Exploration + +`feat2.py` reads the CSV and prints windowed features per Watch stage and a +leave-one-night-out tree and forest: + +```sh +docs/research/sleep-stage-calibration/feat2.py dataset.csv +``` + +It needs numpy and scikit-learn; its shebang pulls them through `uv`. + +## Tests + +```sh +python3 -m unittest scripts/tests/test_sleep_stage_dataset.py +``` + +The Python Modules CI workflow runs them on every PR that touches +`scripts/**/*.py`. The tests use small synthetic fixtures, not a real export. When +`src/lib/sleep-stages.ts` changes, update the port in the script and the +classifier cases in the test, which mirror `src/lib/tests/sleep-stages.test.ts`. diff --git a/docs/research/sleep-stage-calibration/feat2.py b/docs/research/sleep-stage-calibration/feat2.py new file mode 100755 index 000000000..96a8fb1c5 --- /dev/null +++ b/docs/research/sleep-stage-calibration/feat2.py @@ -0,0 +1,109 @@ +#!/usr/bin/env -S uv run --with numpy --with scikit-learn --quiet --no-project +"""Windowed features per Watch stage + leave-one-night-out tree, from dataset.csv. + +Input is the CSV written by scripts/sleep-stage-dataset.py. Exploratory only; +classifier v2 (PLAN.md workstream E) builds on this once >= 20 nights exist. + +Usage: docs/research/sleep-stage-calibration/feat2.py dataset.csv +""" +import csv +import sys +from collections import Counter, defaultdict + +import numpy as np + +STAGES = ('wake', 'light', 'deep', 'rem') +KEYS = ['hr_std', 'hr_rel', 'hrv_rel', 'br_std', 'tso'] + + +def num(v): + return float(v) if v != '' else np.nan + + +def load(path): + nights = defaultdict(list) + with open(path, newline='') as f: + for r in csv.DictReader(f): + nights[(r['night'], r['side'])].append(r) + for rows in nights.values(): + rows.sort(key=lambda r: int(r['ts'])) + return dict(sorted(nights.items())) + + +def windowed(rows): + """15-min centered window features for every row of one night. + + Windows and elapsed fraction use timestamps, not row indices, so gaps in + the vitals do not stretch a window across hours.""" + ts = np.array([int(r['ts']) for r in rows]) + hr = np.array([num(r['hr']) for r in rows]) + hrv = np.array([num(r['hrv']) for r in rows]) + br = np.array([num(r['br']) for r in rows]) + # A nonpositive median would make the ratios infinite; treat it as missing. + medhr, medhrv = (m if m > 0 else np.nan for m in (np.nanmedian(hr), np.nanmedian(hrv))) + lo = np.searchsorted(ts, ts - 450, side='left') + hi = np.searchsorted(ts, ts + 450, side='right') + span = max(ts[-1] - ts[0], 1) + out = [] + for i, r in enumerate(rows): + s = slice(lo[i], hi[i]) + out.append({'w': r['watch_stage'], 'hr_std': np.nanstd(hr[s]), 'hr_rel': np.nanmean(hr[s]) / medhr, + 'hrv_rel': np.nanmean(hrv[s]) / medhrv, 'br_std': np.nanstd(br[s]), + 'tso': (ts[i] - ts[0]) / span}) + return out + + +def main(path): + nights = load(path) + print('=== movement health per night (nonzero fraction, p50/p90/p99)') + for (n, side), rows in nights.items(): + mv = [float(r['movement']) for r in rows if r['movement'] != ''] + if mv: + print(f' {n} {side}: n={len(mv)} nonzero={sum(1 for x in mv if x > 0) / len(mv):.0%} ' + f'p50={np.percentile(mv, 50):.0f} p90={np.percentile(mv, 90):.0f} p99={np.percentile(mv, 99):.0f}') + + print('\n=== windowed features per Watch stage (15-min centered)') + groups = defaultdict(list) + for (n, side), rows in nights.items(): + f = [x for x in windowed(rows) if x['w']] + if f: + groups[n].extend(f) + print(f' night {n} {side}') + for s in STAGES: + g = [x for x in f if x['w'] == s] + if g: + m = {k: np.nanmedian([x[k] for x in g]) for k in KEYS} + print(f" {s:5} n={len(g):3} hr_std={m['hr_std']:.2f} hr_rel={m['hr_rel']:.3f} " + f"hrv_rel={m['hrv_rel']:.2f} br_std={m['br_std']:.2f} tso={m['tso']:.2f}") + + if len(groups) < 2: + print('\nleave-one-night-out needs >= 2 Watch-labeled nights') + return + try: + from sklearn.ensemble import RandomForestClassifier + from sklearn.tree import DecisionTreeClassifier + except ImportError as e: + print('no sklearn', e) + return + + def X(g): + return np.nan_to_num(np.array([[x[k] for k in KEYS] for x in g]), nan=0.0, posinf=0.0, neginf=0.0) + + def y(g): + return [x['w'] for x in g] + + print('\n=== leave-one-night-out, windowed features, small tree / forest') + for night, te in groups.items(): + tr = [x for n, g in groups.items() if n != night for x in g] + for name, clf in (('tree d3', DecisionTreeClassifier(max_depth=3, class_weight='balanced', random_state=0)), + ('forest', RandomForestClassifier(200, max_depth=4, class_weight='balanced', random_state=0))): + clf.fit(X(tr), y(tr)) + pr = clf.predict(X(te)) + acc = np.mean(pr == np.array(y(te))) + print(f' test night {night} {name:8} acc={acc:.0%} pred={dict(Counter(str(p) for p in pr))} truth={dict(Counter(y(te)))}') + + +if __name__ == '__main__': + if len(sys.argv) != 2: + sys.exit(__doc__) + main(sys.argv[1]) diff --git a/scripts/sleep-stage-dataset.py b/scripts/sleep-stage-dataset.py new file mode 100755 index 000000000..44d9999ea --- /dev/null +++ b/scripts/sleep-stage-dataset.py @@ -0,0 +1,517 @@ +#!/usr/bin/env -S uv run --quiet --no-project +""" +Sleep-stage dataset builder: joins Apple Watch sleep stages with pod vitals. + +Emits one CSV row per pod vitals sample (about one per minute) for every night +that has Watch stage labels: + + night,side,ts,watch_stage,hr,hrv,br,movement,hr_quality,watch_hr + +- night local date the night started (sample time minus 12 h) +- ts pod vitals timestamp, unix seconds +- watch_stage wake|light|deep|rem of the Watch segment containing ts, empty if none +- hr,hrv,br raw pod vitals (not outlier-filtered) +- movement nearest movement.total_movement within 60 s, empty if none +- hr_quality vitals_quality.quality_score for that vitals row, empty if none +- watch_hr nearest Watch heart-rate sample within 60 s, empty if none + +A night's window is the Watch stage coverage padded by --pad-min on each side. +Nights whose pod-vs-Watch mean absolute HR difference exceeds --max-hr-mad are +skipped with a warning: that pattern means the Watch wearer slept on the other +side. + +--summary prints per-night agreement with the Watch for the classifier in +src/lib/sleep-stages.ts (ported below), as deployed (calibrationQuality 0, +movement-only) and with the iOS rule set (calibrationQuality 1), plus the +always-light baseline. + +Usage: + scripts/sleep-stage-dataset.py export.xml biometrics.db --side left --out dataset.csv + scripts/sleep-stage-dataset.py apple_health_export/ biometrics.db --summary + scripts/sleep-stage-dataset.py --from-db biometrics.db --side left --out dataset.csv + +--from-db reads Watch stages from the reference_stages table that +biometrics.importReferenceStages writes, instead of a Health export. That source +carries no Watch heart rate, so the side-mismatch guard cannot run. + +See docs/research/sleep-stage-calibration/README.md for pulling the pod DB. +Stdlib only; runs under python3 >= 3.9 directly or via the uv shebang. +""" + +import argparse +import bisect +import csv +import math +import os +import re +import sqlite3 +import sys +import xml.etree.ElementTree as ET +from collections import Counter +from datetime import datetime, timedelta +from pathlib import Path + +STAGES = ('wake', 'light', 'deep', 'rem') +CSV_FIELDS = ('night', 'side', 'ts', 'watch_stage', 'hr', 'hrv', 'br', 'movement', 'hr_quality', 'watch_hr') + +# HKCategoryValueSleepAnalysis* -> stage. InBed, Asleep and AsleepUnspecified +# carry no stage and are ignored. +HK_STAGE = { + 'HKCategoryValueSleepAnalysisAwake': 'wake', + 'HKCategoryValueSleepAnalysisAsleepCore': 'light', + 'HKCategoryValueSleepAnalysisAsleepDeep': 'deep', + 'HKCategoryValueSleepAnalysisAsleepREM': 'rem', +} +HK_SLEEP = 'HKCategoryTypeIdentifierSleepAnalysis' +HK_HR = 'HKQuantityTypeIdentifierHeartRate' + +JOIN_TOLERANCE_S = 60 +MIN_HR_MATCHES = 10 +NIGHT_ROLLOVER = timedelta(hours=12) + + +def warn(msg): + print(f'warning: {msg}', file=sys.stderr) + + +def night_of(dt): + """Local date the night started; dt must be timezone-aware local time.""" + return (dt - NIGHT_ROLLOVER).date().isoformat() + + +# ── Health export ────────────────────────────────────────────────────────── + +def parse_export(path, source_pattern='Watch'): + """Stream export.xml and return (stage_segments, hr_samples) for sources + matching source_pattern. + + stage_segments: [(start_dt, end_dt, stage)] in file order (aware datetimes) + hr_samples: [(unix_s, bpm)] sorted by time + """ + if os.path.isdir(path): + path = os.path.join(path, 'export.xml') + source_re = re.compile(source_pattern) + segments, hr = [], [] + + def add(a): + kind = a.get('type') + if kind not in (HK_SLEEP, HK_HR) or not source_re.search(a.get('sourceName', '')): + return + try: + start = datetime.strptime(a['startDate'], '%Y-%m-%d %H:%M:%S %z') + end = datetime.strptime(a['endDate'], '%Y-%m-%d %H:%M:%S %z') + except (KeyError, ValueError): + return + if kind == HK_SLEEP: + stage = HK_STAGE.get(a.get('value', '')) + if stage and end > start: + segments.append((start, end, stage)) + return + try: + bpm = float(a['value']) + except (KeyError, ValueError): + return + if math.isfinite(bpm): + hr.append((int(start.timestamp()), bpm)) + + # Incremental parse, so record layout (line breaks, attribute wrapping) + # does not matter. Clearing the root after each record keeps memory flat. + events = ET.iterparse(path, events=('start', 'end')) + _, root = next(events) + for event, elem in events: + if event == 'end' and elem.tag == 'Record': + add(elem.attrib) + root.clear() + hr.sort() + return segments, hr + + +def segments_from_db(conn, side, source='apple_watch'): + """Read Watch stage segments from reference_stages (sleepypod-core-116). + + Timestamps are unix seconds (Drizzle mode 'timestamp'); millisecond values + are tolerated. Returns segments as local aware datetimes, like parse_export. + """ + exists = conn.execute( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name='reference_stages'").fetchone() + if not exists: + raise SystemExit( + 'error: reference_stages table not found in the DB. It is created by the ' + 'biometrics.importReferenceStages migration; pass a Health export instead.') + + def to_dt(v): + v = float(v) + if v > 1e11: + v /= 1000.0 + return datetime.fromtimestamp(v).astimezone() + + rows = conn.execute( + 'SELECT start, "end", stage FROM reference_stages WHERE side = ? AND source = ? ORDER BY start', + (side, source)).fetchall() + return [(to_dt(s), to_dt(e), st) for s, e, st in rows if st in STAGES and float(e) > float(s)] + + +def stage_lookup(segments): + """Return stage_at(ts) -> stage of the segment whose [start, end) contains + unix second ts, else None. Where segments overlap, the one starting latest + wins, so export file order and DB order give the same labels.""" + segs = sorted((s.timestamp(), e.timestamp(), st) for s, e, st in segments) + starts = [s for s, _, _ in segs] + reach, m = [], float('-inf') + for _, e, _ in segs: + m = max(m, e) + reach.append(m) + + def stage_at(ts): + j = bisect.bisect_right(starts, ts) - 1 + # reach[j] is the latest end among segments 0..j; once it is <= ts no + # earlier segment can contain ts. + while j >= 0 and reach[j] > ts: + if segs[j][1] > ts: + return segs[j][2] + j -= 1 + return None + + return stage_at + + +def group_nights(segments, pad_min): + """{night: (window_start_s, window_end_s)} from Watch coverage + padding. + + Windows are disjoint: a window that touches the previous night's (sleep + spanning the noon rollover, or padding) is merged into that night, so no + vitals row is emitted under two nights.""" + spans = {} + for start, end, _ in segments: + n = night_of(start) + s, e = start.timestamp(), end.timestamp() + if n in spans: + spans[n] = (min(spans[n][0], s), max(spans[n][1], e)) + else: + spans[n] = (s, e) + pad = pad_min * 60 + out, prev = {}, None + for n, (s, e) in sorted(spans.items(), key=lambda kv: kv[1][0]): + s, e = int(s - pad), int(e + pad) + if prev is not None and s <= out[prev][1]: + out[prev] = (out[prev][0], max(out[prev][1], e)) + continue + out[n] = (s, e) + prev = n + return out + + +# ── Pod DB ───────────────────────────────────────────────────────────────── + +def load_pod(conn, side, start_s, end_s, join_pad_s=0): + """Return (vitals, movement) for one side and window, ascending. + + vitals: [(id, ts, hr, hrv, br, quality)] + movement: [(ts, total_movement)], over the window widened by join_pad_s + on each side so vitals at the edges can match their nearest row + """ + has_quality = conn.execute( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name='vitals_quality'").fetchone() + if has_quality: + q = ('SELECT v.id, v.timestamp, v.heart_rate, v.hrv, v.breathing_rate, ' + '(SELECT quality_score FROM vitals_quality WHERE vitals_id = v.id ORDER BY id DESC LIMIT 1) ' + 'FROM vitals v WHERE v.side = ? AND v.timestamp BETWEEN ? AND ? ORDER BY v.timestamp') + else: + q = ('SELECT id, timestamp, heart_rate, hrv, breathing_rate, NULL ' + 'FROM vitals WHERE side = ? AND timestamp BETWEEN ? AND ? ORDER BY timestamp') + vitals = conn.execute(q, (side, start_s, end_s)).fetchall() + movement = conn.execute( + 'SELECT timestamp, total_movement FROM movement WHERE side = ? AND timestamp BETWEEN ? AND ? ' + 'ORDER BY timestamp', (side, start_s - join_pad_s, end_s + join_pad_s)).fetchall() + return vitals, movement + + +def nearest(times, values, t, tol=JOIN_TOLERANCE_S): + """Value whose time is nearest t within tol (earlier wins ties), else None.""" + i = bisect.bisect_left(times, t) + best, best_d = None, None + for j in (i - 1, i): + if 0 <= j < len(times): + d = abs(times[j] - t) + if d <= tol and (best_d is None or d < best_d): + best, best_d = values[j], d + return best + + +def join_night(night, side, vitals, movement, stage_at, watch_hr): + """Build dataset rows (dicts keyed by CSV_FIELDS) for one night.""" + mov_t = [t for t, _ in movement] + mov_v = [v for _, v in movement] + hr_t = [t for t, _ in watch_hr] + hr_v = [v for _, v in watch_hr] + rows = [] + for _id, ts, hr, hrv, br, quality in vitals: + rows.append({ + 'night': night, + 'side': side, + 'ts': int(ts), + 'watch_stage': stage_at(ts), + 'hr': hr, + 'hrv': hrv, + 'br': br, + 'movement': nearest(mov_t, mov_v, ts), + 'hr_quality': quality, + 'watch_hr': nearest(hr_t, hr_v, ts), + }) + return rows + + +def hr_mad(rows): + """(mean |pod HR - Watch HR|, matched count) over rows with both values + finite.""" + diffs = [abs(r['hr'] - r['watch_hr']) for r in rows if r['hr'] is not None and r['watch_hr'] is not None] + diffs = [d for d in diffs if math.isfinite(d)] + return (sum(diffs) / len(diffs) if diffs else None), len(diffs) + + +# ── Classifier port of src/lib/sleep-stages.ts ───────────────────────────── +# Keep in lockstep with classifySleepStages / classifyEpoch / filterOutliers. + +def _js_round(x): + return math.floor(x + 0.5) + + +def filter_outliers(vitals): + """[(ts, hr, hrv, br)] -> same with outliers nulled (hard limits + ±2 HR median filter).""" + def ok_hr(h): + return h is not None and 45 <= h <= 130 + + out = [] + n = len(vitals) + for i, (ts, hr, hrv, br) in enumerate(vitals): + if hr is not None and (hr < 45 or hr > 130): + hr = None + if hrv is not None and (hrv < 1 or hrv > 300): + hrv = None + if br is not None and (br < 8 or br > 25): + br = None + if hr is not None: + window = [w[1] for w in vitals[max(0, i - 2):min(n - 1, i + 2) + 1] if ok_hr(w[1])] + if window: + median = sorted(window)[len(window) // 2] + mean = sum(window) / len(window) + std = math.sqrt(sum((h - mean) ** 2 for h in window) / len(window)) + if std > 0 and abs(hr - median) > 2 * std: + hr = None + out.append((ts, hr, hrv, br)) + return out + + +def classify_epoch(hr, hrv, mov, avg_hr, cq): + if cq < 0.3: + if mov is not None and mov > 200: + return 'wake' + return 'light' + if mov is not None and mov > 200: + return 'wake' + if hr is not None and hr > 0: + ratio = hr / avg_hr + if ratio < 0.92: + return 'deep' + if ratio >= 0.95: + if hrv is not None and hrv < 25 and mov is not None and mov < 30: + return 'rem' + if mov is not None and mov < 50 and hrv is not None and hrv < 40: + return 'rem' + if mov is not None and mov > 100: + return 'wake' + return 'light' + + +def classify_sleep_stages(vitals, movement, calibration_quality=0.0): + """vitals [(ts_s, hr, hrv, br)], movement [(ts_s, total)] -> [(ts_s, stage)]. + + Mirrors classifySleepStages: outlier filter, 5-min movement buckets + (last row wins), phase-1 rules, A-B-A smoothing, transition constraints. + """ + if not vitals: + return [] + v = filter_outliers(sorted(vitals, key=lambda r: r[0])) + + buckets = {} + for ts, total in movement: + buckets[_js_round(ts * 1000 / 300_000) * 300_000] = total + + valid = [hr for _, hr, _, _ in v if hr is not None and hr > 0] + avg_hr = sum(valid) / len(valid) if valid else 60 + + stages = [ + classify_epoch(hr, hrv, buckets.get(_js_round(ts * 1000 / 300_000) * 300_000), avg_hr, calibration_quality) + for ts, hr, hrv, _ in v + ] + for i in range(1, len(stages) - 1): + if stages[i - 1] == stages[i + 1] and stages[i] != stages[i - 1]: + stages[i] = stages[i - 1] + blocked = {('wake', 'deep'), ('deep', 'wake'), ('deep', 'rem'), ('rem', 'deep')} + for i in range(1, len(stages)): + if (stages[i - 1], stages[i]) in blocked: + stages[i] = 'light' + return [(ts, s) for (ts, *_), s in zip(v, stages)] + + +# ── Summary ──────────────────────────────────────────────────────────────── + +def agreement(rows, predicted): + """Percent of Watch-labeled rows where predicted[ts] equals watch_stage.""" + labeled = [r for r in rows if r['watch_stage']] + if not labeled: + return None + hit = sum(1 for r in labeled if predicted.get(r['ts']) == r['watch_stage']) + return 100.0 * hit / len(labeled) + + +def summarize_night(rows, movement): + vitals = [(r['ts'], r['hr'], r['hrv'], r['br']) for r in rows] + deployed = dict(classify_sleep_stages(vitals, movement, 0.0)) + ios = dict(classify_sleep_stages(vitals, movement, 1.0)) + light = {r['ts']: 'light' for r in rows} + return { + 'labeled': sum(1 for r in rows if r['watch_stage']), + 'deployed': agreement(rows, deployed), + 'ios': agreement(rows, ios), + 'light': agreement(rows, light), + 'watch_dist': Counter(r['watch_stage'] for r in rows if r['watch_stage']), + } + + +def print_summary(results, out=None): + out = out or sys.stdout + + def pct(x): + return ' n/a' if x is None else f'{x:5.0f}%' + + def mad(x): + return ' n/a' if x is None else f'{x:6.1f}' + + print('night side rows labeled HR-MAD deployed(cq=0) iOS(cq=1) always-light', file=out) + for r in results: + print(f"{r['night']} {r['side']:5} {r['rows']:5} {r['labeled']:7} {mad(r['mad'])} " + f"{pct(r['deployed']):>14} {pct(r['ios']):>9} {pct(r['light']):>12}", file=out) + print('\nWatch-labeled samples per night:', file=out) + for r in results: + dist = ', '.join(f'{s}={r["watch_dist"].get(s, 0)}' for s in STAGES) + print(f" {r['night']}: {dist}", file=out) + + +# ── Driver ───────────────────────────────────────────────────────────────── + +def build(conn, segments, watch_hr, side, pad_min=10, max_hr_mad=8.0, nights=None): + """Return [(night, rows, movement, mad, matched)] for nights passing the guard.""" + stage_at = stage_lookup(segments) + out = [] + for night, (start_s, end_s) in group_nights(segments, pad_min).items(): + if nights and night not in nights: + continue + vitals, near_mov = load_pod(conn, side, start_s, end_s, JOIN_TOLERANCE_S) + if not vitals: + warn(f'{night}: no {side} vitals in the Watch window; skipped') + continue + # Join candidates reach JOIN_TOLERANCE_S past the window; the + # classifier's movement input stays inside it. + hr_in = [(t, v) for t, v in watch_hr if start_s - JOIN_TOLERANCE_S <= t <= end_s + JOIN_TOLERANCE_S] + rows = join_night(night, side, vitals, near_mov, stage_at, hr_in) + movement = [(t, v) for t, v in near_mov if start_s <= t <= end_s] + mad, matched = hr_mad(rows) + if matched < MIN_HR_MATCHES: + warn(f'{night}: only {matched} pod/Watch HR matches; side-mismatch guard not applied') + elif mad > max_hr_mad: + warn(f'{night}: pod {side} vs Watch HR MAD {mad:.1f} bpm > {max_hr_mad:g} ' + f'({matched} matches); likely the other side, skipped') + continue + out.append((night, rows, movement, mad, matched)) + return out + + +def _fmt(v): + if v is None: + return '' + if isinstance(v, float): + if v != 0 and abs(v) < 0.0005: + # Three decimals would print it as 0. + return f'{v:.3g}' + return f'{v:.3f}'.rstrip('0').rstrip('.') + return v + + +def write_csv(nights, fh): + w = csv.writer(fh, lineterminator='\n') + w.writerow(CSV_FIELDS) + for _night, rows, *_ in nights: + for r in rows: + w.writerow([_fmt(r[k]) for k in CSV_FIELDS]) + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0]) + ap.add_argument('inputs', nargs='+', metavar='PATH', + help=' , or just with --from-db') + ap.add_argument('--from-db', action='store_true', + help='read Watch stages from the reference_stages table instead of an export') + ap.add_argument('--side', choices=('left', 'right'), default='left') + ap.add_argument('--out', help='CSV path (default: stdout unless --summary)') + ap.add_argument('--summary', action='store_true', help='print per-night classifier agreement') + ap.add_argument('--nights', help='comma-separated night dates to keep, e.g. 2026-10-08,2026-10-09') + ap.add_argument('--source', default='Watch', help='regex on export sourceName (default: Watch)') + ap.add_argument('--pad-min', type=int, default=10, help='minutes added around Watch coverage (default 10)') + ap.add_argument('--max-hr-mad', type=float, default=8.0, help='side-mismatch guard threshold in bpm (default 8)') + args = ap.parse_args(argv) + + expected = 1 if args.from_db else 2 + if len(args.inputs) != expected: + ap.error('expected with --from-db' if args.from_db + else 'expected ') + db_path = args.inputs[-1] + if not os.path.exists(db_path): + ap.error(f'{db_path} not found') + inputs = [db_path] + if not args.from_db: + export = args.inputs[0] + export_xml = os.path.join(export, 'export.xml') if os.path.isdir(export) else export + if not os.path.exists(export_xml): + ap.error(f'{export_xml} not found') + inputs.append(export_xml) + if args.out and os.path.exists(args.out) and any(os.path.samefile(args.out, p) for p in inputs): + ap.error(f'--out {args.out} is an input file; refusing to overwrite it') + # as_uri() percent-encodes ?, # and %, which SQLite would otherwise read + # as URI syntax and open a different file. + conn = sqlite3.connect(Path(db_path).resolve().as_uri() + '?mode=ro', uri=True) + + if args.from_db: + segments, watch_hr = segments_from_db(conn, args.side), [] + else: + try: + segments, watch_hr = parse_export(export_xml, args.source) + except ET.ParseError as e: + raise SystemExit(f'error: {export_xml} is not valid XML: {e}') + if not segments: + raise SystemExit('error: no Watch sleep-stage samples found') + + wanted = {n.strip() for n in args.nights.split(',') if n.strip()} if args.nights else None + nights = build(conn, segments, watch_hr, args.side, args.pad_min, args.max_hr_mad, wanted) + + if args.out: + with open(args.out, 'w', newline='') as fh: + write_csv(nights, fh) + print(f'wrote {sum(len(n[1]) for n in nights)} rows for {len(nights)} nights to {args.out}', file=sys.stderr) + elif not args.summary: + write_csv(nights, sys.stdout) + + if args.summary: + results = [] + for night, rows, movement, mad, _matched in nights: + s = summarize_night(rows, movement) + results.append({'night': night, 'side': args.side, 'rows': len(rows), 'mad': mad, **s}) + if results: + print_summary(results) + else: + print('no nights passed the filters', file=sys.stderr) + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/scripts/tests/test_sleep_stage_dataset.py b/scripts/tests/test_sleep_stage_dataset.py new file mode 100644 index 000000000..5d3bfb86c --- /dev/null +++ b/scripts/tests/test_sleep_stage_dataset.py @@ -0,0 +1,447 @@ +"""Tests for scripts/sleep-stage-dataset.py using small synthetic fixtures. + +Run: python3 -m unittest scripts/tests/test_sleep_stage_dataset.py + (or: uv run --with pytest --no-project pytest scripts/tests/test_sleep_stage_dataset.py) +""" + +import contextlib +import csv +import importlib.util +import io +import os +import sqlite3 +import tempfile +import time +import unittest +from datetime import datetime, timedelta, timezone + +HERE = os.path.dirname(os.path.abspath(__file__)) +_spec = importlib.util.spec_from_file_location( + 'sleep_stage_dataset', os.path.join(HERE, '..', 'sleep-stage-dataset.py')) +ds = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(ds) + +TZ = timezone(timedelta(hours=-7)) +# Night of 2026-10-08: Watch coverage 02:00-02:30 local on 10-09. +T0 = datetime(2026, 10, 9, 2, 0, tzinfo=TZ) +WATCH = "Jane’s Apple Watch" + + +def hk_time(dt): + return dt.strftime('%Y-%m-%d %H:%M:%S %z') + + +def sleep_record(start, end, value, source=WATCH): + return (f' \n' + f' \n \n') + + +def hr_record(at, bpm, source=WATCH): + return (f' \n') + + +# Segments: 02:00-02:10 light, 02:10-02:20 deep, 02:20-02:30 rem +SEGMENTS = [(0, 10, 'AsleepCore'), (10, 20, 'AsleepDeep'), (20, 30, 'AsleepREM')] + + +def write_export(path, hr_offset=0.0, with_hr=True): + parts = ['\n\n'] + for a, b, v in SEGMENTS: + parts.append(sleep_record(T0 + timedelta(minutes=a), T0 + timedelta(minutes=b), v)) + # Noise that must be ignored: other source, InBed, non-Watch HR. + parts.append(sleep_record(T0, T0 + timedelta(minutes=30), 'AsleepDeep', source='sleepypod')) + parts.append(sleep_record(T0, T0 + timedelta(minutes=30), 'InBed')) + parts.append(hr_record(T0, 150, source='sleepypod')) + if with_hr: + for m in range(30): + parts.append(hr_record(T0 + timedelta(minutes=m, seconds=20), pod_hr(m) + hr_offset)) + parts.append('\n') + with open(path, 'w', encoding='utf-8') as f: + f.write(''.join(parts)) + + +def pod_hr(minute): + return 60.0 + (minute % 3) + + +def make_db(path, side='left', minutes=range(-5, 35)): + conn = sqlite3.connect(path) + conn.executescript(''' + CREATE TABLE vitals (id INTEGER PRIMARY KEY AUTOINCREMENT, side TEXT, timestamp INTEGER, + heart_rate REAL, hrv REAL, breathing_rate REAL); + CREATE TABLE movement (id INTEGER PRIMARY KEY AUTOINCREMENT, side TEXT, timestamp INTEGER, + total_movement INTEGER); + CREATE TABLE vitals_quality (id INTEGER PRIMARY KEY AUTOINCREMENT, vitals_id INTEGER, + side TEXT, timestamp INTEGER, quality_score REAL); + ''') + base = int(T0.timestamp()) + for m in minutes: + ts = base + m * 60 + 5 + cur = conn.execute('INSERT INTO vitals (side, timestamp, heart_rate, hrv, breathing_rate) VALUES (?,?,?,?,?)', + (side, ts, pod_hr(m), 50.0 + m, 14.0)) + conn.execute('INSERT INTO vitals_quality (vitals_id, side, timestamp, quality_score) VALUES (?,?,?,?)', + (cur.lastrowid, side, ts, 0.5)) + conn.execute('INSERT INTO movement (side, timestamp, total_movement) VALUES (?,?,?)', + (side, base + m * 60 + 42, m * 10)) + conn.commit() + return conn + + +def add_reference_stages(conn, side='left'): + conn.execute('CREATE TABLE reference_stages (id INTEGER PRIMARY KEY AUTOINCREMENT, side TEXT, ' + 'source TEXT, start INTEGER, "end" INTEGER, stage TEXT, created_at INTEGER)') + stage = {'AsleepCore': 'light', 'AsleepDeep': 'deep', 'AsleepREM': 'rem'} + for a, b, v in SEGMENTS: + conn.execute('INSERT INTO reference_stages (side, source, start, "end", stage, created_at) VALUES (?,?,?,?,?,0)', + (side, 'apple_watch', int((T0 + timedelta(minutes=a)).timestamp()), + int((T0 + timedelta(minutes=b)).timestamp()), stage[v])) + conn.commit() + + +class Fixture(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.export = os.path.join(self.tmp.name, 'export.xml') + self.db_path = os.path.join(self.tmp.name, 'biometrics.db') + + def tearDown(self): + self.tmp.cleanup() + + def run_main(self, *argv): + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + ds.main(list(argv)) + return out.getvalue(), err.getvalue() + + +class ParseExportTest(Fixture): + def test_keeps_only_watch_stage_samples_and_hr(self): + write_export(self.export) + segments, hr = ds.parse_export(self.export) + self.assertEqual([s for _, _, s in segments], ['light', 'deep', 'rem']) + self.assertEqual(len(hr), 30) + self.assertTrue(all(55 <= v <= 65 for _, v in hr)) + + def test_accepts_export_directory(self): + write_export(self.export) + segments, _ = ds.parse_export(self.tmp.name) + self.assertEqual(len(segments), 3) + + def test_source_regex_selects_other_source(self): + write_export(self.export) + segments, hr = ds.parse_export(self.export, source_pattern='^sleepypod$') + self.assertEqual([s for _, _, s in segments], ['deep']) + self.assertEqual(hr, [(int(T0.timestamp()), 150.0)]) + + def test_record_layout_does_not_matter(self): + # Attributes wrapped over lines, and two records on one line. + wrapped = sleep_record(T0, T0 + timedelta(minutes=10), 'AsleepCore').replace('" ', '"\n ') + one_line = (hr_record(T0 + timedelta(minutes=1), 61) + hr_record(T0 + timedelta(minutes=2), 62)).replace('\n', '') + with open(self.export, 'w', encoding='utf-8') as f: + f.write(f'\n\n{wrapped}{one_line}\n\n') + segments, hr = ds.parse_export(self.export) + self.assertEqual([s for _, _, s in segments], ['light']) + self.assertEqual([v for _, v in hr], [61.0, 62.0]) + + def test_nonfinite_watch_hr_is_dropped(self): + with open(self.export, 'w', encoding='utf-8') as f: + f.write('\n' + ''.join(hr_record(T0 + timedelta(minutes=m), v) + for m, v in enumerate(('nan', 'inf', '61'))) + '\n') + _, hr = ds.parse_export(self.export) + self.assertEqual([v for _, v in hr], [61.0]) + + +class JoinTest(Fixture): + def test_rows_per_minute_with_labels_movement_quality_and_watch_hr(self): + write_export(self.export) + make_db(self.db_path).close() + out, _ = self.run_main(self.export, self.db_path, '--side', 'left') + rows = list(csv.DictReader(io.StringIO(out))) + self.assertEqual(list(rows[0].keys()), list(ds.CSV_FIELDS)) + # Window = 02:00-02:30 padded 10 min = 01:50-02:40 → minutes -5..34 all present. + self.assertEqual(len(rows), 40) + self.assertEqual({r['night'] for r in rows}, {'2026-10-08'}) + by_min = {(int(r['ts']) - int(T0.timestamp())) // 60: r for r in rows} + self.assertEqual(by_min[-1]['watch_stage'], '') + self.assertEqual(by_min[0]['watch_stage'], 'light') + self.assertEqual(by_min[9]['watch_stage'], 'light') + self.assertEqual(by_min[10]['watch_stage'], 'deep') + self.assertEqual(by_min[25]['watch_stage'], 'rem') + self.assertEqual(by_min[30]['watch_stage'], '') + # Vitals at :05; previous minute's movement row at :42 (23 s away) beats this minute's (37 s). + self.assertEqual(by_min[3]['movement'], '20') + self.assertEqual(by_min[3]['hr_quality'], '0.5') + # Watch HR at :20 of each minute 0..29 is 15 s from the vitals sample. + self.assertEqual(by_min[4]['watch_hr'], by_min[4]['hr']) + self.assertEqual(by_min[-2]['watch_hr'], '') + self.assertEqual(by_min[32]['watch_hr'], '') + + def test_nearest_respects_tolerance_and_prefers_earlier_on_tie(self): + times, vals = [100, 200], ['a', 'b'] + self.assertEqual(ds.nearest(times, vals, 150), 'a') + self.assertEqual(ds.nearest(times, vals, 190), 'b') + self.assertIsNone(ds.nearest(times, vals, 300)) + self.assertIsNone(ds.nearest([], [], 5)) + + def test_only_requested_side_is_read(self): + write_export(self.export) + make_db(self.db_path, side='right').close() + out, err = self.run_main(self.export, self.db_path, '--side', 'left') + self.assertEqual(out.strip(), ','.join(ds.CSV_FIELDS)) + self.assertIn('no left vitals', err) + + def test_from_db_matches_export_rows(self): + # reference_stages has no tz; night labels use local time, so pin it to the export's zone. + old_tz = os.environ.get('TZ') + os.environ['TZ'] = 'America/Los_Angeles' + time.tzset() + self.addCleanup(self._restore_tz, old_tz) + write_export(self.export, with_hr=False) + conn = make_db(self.db_path) + add_reference_stages(conn) + conn.close() + from_export, _ = self.run_main(self.export, self.db_path) + from_db, _ = self.run_main('--from-db', self.db_path) + self.assertEqual(from_db, from_export) + self.assertEqual(len(from_db.strip().splitlines()), 41) + + @staticmethod + def _restore_tz(old_tz): + if old_tz is None: + os.environ.pop('TZ', None) + else: + os.environ['TZ'] = old_tz + time.tzset() + + def test_out_matching_an_input_is_refused(self): + write_export(self.export) + make_db(self.db_path).close() + size = os.path.getsize(self.db_path) + for target in (self.db_path, self.export): + with self.assertRaises(SystemExit): + self.run_main(self.tmp.name, self.db_path, '--out', target) + self.assertEqual(os.path.getsize(self.db_path), size) + + def test_join_matches_rows_just_outside_the_window(self): + conn = sqlite3.connect(self.db_path) + conn.executescript(''' + CREATE TABLE vitals (id INTEGER PRIMARY KEY, side TEXT, timestamp INTEGER, + heart_rate REAL, hrv REAL, breathing_rate REAL); + CREATE TABLE movement (id INTEGER PRIMARY KEY, side TEXT, timestamp INTEGER, total_movement INTEGER); + ''') + base = int(T0.timestamp()) + conn.execute("INSERT INTO vitals (side, timestamp, heart_rate, hrv, breathing_rate) VALUES ('left',?,60,50,14)", + (base,)) + conn.execute("INSERT INTO movement (side, timestamp, total_movement) VALUES ('left',?,77)", (base - 10,)) + segs = [(T0, T0 + timedelta(minutes=30), 'light')] + [(_night, rows, movement, _mad, _matched)] = ds.build(conn, segs, [(base - 10, 60.0)], 'left', pad_min=0) + conn.close() + self.assertEqual((rows[0]['movement'], rows[0]['watch_hr']), (77, 60.0)) + # The classifier's movement stays inside the window. + self.assertEqual(movement, []) + + def test_db_path_with_uri_characters(self): + write_export(self.export) + db_dir = os.path.join(self.tmp.name, 'pod#copy?1%') + os.mkdir(db_dir) + db_path = os.path.join(db_dir, 'biometrics.db') + make_db(db_path).close() + out, _ = self.run_main(self.export, db_path) + self.assertEqual(len(out.strip().splitlines()), 41) + + def test_missing_export_is_a_usage_error(self): + make_db(self.db_path).close() + err = io.StringIO() + with contextlib.redirect_stderr(err), self.assertRaises(SystemExit) as cm: + ds.main([os.path.join(self.tmp.name, 'nope.xml'), self.db_path]) + self.assertEqual(cm.exception.code, 2) + self.assertIn('nope.xml not found', err.getvalue()) + + def test_from_db_without_table_exits_with_message(self): + make_db(self.db_path).close() + with self.assertRaises(SystemExit) as cm: + self.run_main('--from-db', self.db_path) + self.assertIn('reference_stages', str(cm.exception.code)) + + +class LabelAndWindowTest(unittest.TestCase): + def test_sample_before_mid_minute_transition_keeps_earlier_stage(self): + t = T0 + timedelta(minutes=1, seconds=30) + segs = [(T0, t, 'light'), (t, T0 + timedelta(minutes=5), 'deep')] + stage_at = ds.stage_lookup(segs) + base = int(T0.timestamp()) + self.assertEqual(stage_at(base + 70), 'light') + self.assertEqual(stage_at(base + 90), 'deep') + self.assertIsNone(stage_at(base + 300)) + self.assertIsNone(stage_at(base - 1)) + + def test_overlap_resolution_ignores_input_order(self): + segs = [(T0, T0 + timedelta(minutes=30), 'light'), + (T0 + timedelta(minutes=10), T0 + timedelta(minutes=12), 'rem')] + base = int(T0.timestamp()) + for order in (segs, segs[::-1]): + stage_at = ds.stage_lookup(order) + self.assertEqual([stage_at(base + m * 60) for m in (5, 11, 20)], ['light', 'rem', 'light']) + + def test_sleep_spanning_noon_belongs_to_one_night(self): + noon = datetime(2026, 10, 9, 12, 0, tzinfo=TZ) + segs = [(noon - timedelta(minutes=10), noon + timedelta(minutes=10), 'light'), + (noon + timedelta(minutes=10), noon + timedelta(minutes=30), 'deep')] + windows = ds.group_nights(segs, 10) + self.assertEqual(list(windows), ['2026-10-08']) + start, end = windows['2026-10-08'] + self.assertEqual((start, end), (int(noon.timestamp()) - 20 * 60, int(noon.timestamp()) + 40 * 60)) + + def test_dst_change_keeps_one_night_and_contiguous_labels(self): + # Segment times carry the offset in force when they were written, so a + # night across a DST change mixes two offsets. + pdt, pst = timezone(timedelta(hours=-7)), timezone(timedelta(hours=-8)) + fall = [(datetime(2026, 11, 1, 0, 30, tzinfo=pdt), datetime(2026, 11, 1, 1, 50, tzinfo=pdt), 'light'), + (datetime(2026, 11, 1, 1, 50, tzinfo=pdt), datetime(2026, 11, 1, 1, 20, tzinfo=pst), 'deep'), + (datetime(2026, 11, 1, 1, 20, tzinfo=pst), datetime(2026, 11, 1, 6, 0, tzinfo=pst), 'rem')] + spring = [(datetime(2027, 3, 14, 0, 30, tzinfo=pst), datetime(2027, 3, 14, 1, 50, tzinfo=pst), 'light'), + (datetime(2027, 3, 14, 3, 50, tzinfo=pdt), datetime(2027, 3, 14, 6, 0, tzinfo=pdt), 'deep')] + for segs, night in ((fall, '2026-10-31'), (spring, '2027-03-13')): + windows = ds.group_nights(segs, 0) + self.assertEqual(list(windows), [night]) + self.assertEqual(windows[night], (int(segs[0][0].timestamp()), int(segs[-1][1].timestamp()))) + # The repeated 01:00-02:00 hour labels by absolute time: 01:50 PDT is + # 01:20 + 30 min PST, so deep covers exactly 30 minutes. + stage_at = ds.stage_lookup(fall) + deep_start = int(fall[1][0].timestamp()) + self.assertEqual(int(fall[1][1].timestamp()) - deep_start, 30 * 60) + self.assertEqual([stage_at(deep_start - 1), stage_at(deep_start), stage_at(deep_start + 30 * 60)], + ['light', 'deep', 'rem']) + + +class SideMismatchGuardTest(Fixture): + def test_night_within_threshold_is_kept(self): + write_export(self.export, hr_offset=3.0) + make_db(self.db_path).close() + out, err = self.run_main(self.export, self.db_path) + self.assertEqual(len(out.strip().splitlines()), 41) + self.assertNotIn('MAD', err) + + def test_night_over_threshold_warns_and_is_skipped(self): + write_export(self.export, hr_offset=12.0) + make_db(self.db_path).close() + out, err = self.run_main(self.export, self.db_path) + self.assertEqual(out.strip(), ','.join(ds.CSV_FIELDS)) + self.assertIn('2026-10-08', err) + self.assertRegex(err, r'HR MAD 12\.\d bpm > 8') + + def test_threshold_is_configurable(self): + write_export(self.export, hr_offset=12.0) + make_db(self.db_path).close() + out, _ = self.run_main(self.export, self.db_path, '--max-hr-mad', '15') + self.assertEqual(len(out.strip().splitlines()), 41) + + def test_too_few_hr_matches_warns_but_keeps(self): + write_export(self.export, with_hr=False) + make_db(self.db_path).close() + out, err = self.run_main(self.export, self.db_path) + self.assertEqual(len(out.strip().splitlines()), 41) + self.assertIn('guard not applied', err) + + def test_hr_mad(self): + rows = [{'hr': 60.0, 'watch_hr': 70.0}, {'hr': 60.0, 'watch_hr': 58.0}, + {'hr': None, 'watch_hr': 90.0}, {'hr': 60.0, 'watch_hr': None}] + self.assertEqual(ds.hr_mad(rows), (6.0, 2)) + self.assertEqual(ds.hr_mad(rows + [{'hr': 60.0, 'watch_hr': float('nan')}]), (6.0, 2)) + + +def vit(minute, hr, hrv=None, br=None): + return (1_790_000_000 + minute * 60, hr, hrv, br) + + +def mov(minute, total): + return (1_790_000_000 + minute * 60, total) + + +class ClassifierPortTest(unittest.TestCase): + """Mirrors src/lib/tests/sleep-stages.test.ts for classifySleepStages.""" + + def stages(self, vitals, movement, cq=0.0): + return [s for _, s in ds.classify_sleep_stages(vitals, movement, cq)] + + def test_empty(self): + self.assertEqual(ds.classify_sleep_stages([], []), []) + + def test_high_movement_is_wake(self): + self.assertEqual(self.stages([vit(0, 70, 40, 15)], [mov(0, 500)]), ['wake']) + + def test_movement_only_mode_never_emits_deep_or_rem(self): + v = [vit(0, 80, 35, 15), vit(5, 65, 40, 15), vit(10, 63, 50, 14), vit(15, 75, 38, 15), vit(20, 72, 36, 15)] + self.assertEqual(self.stages(v, [], 0.0), ['light'] * 5) + + def test_low_hr_ratio_is_deep(self): + v = [vit(0, 80, 35, 15), vit(5, 65, 40, 15), vit(10, 63, 50, 14), vit(15, 75, 38, 15), vit(20, 72, 36, 15)] + self.assertEqual(self.stages(v, [], 1.0)[2], 'deep') + + def test_elevated_hr_low_hrv_low_movement_is_rem(self): + v = [vit(0, 55, 20, 14), vit(5, 62, 20, 14), vit(10, 65, 20, 14), vit(15, 68, 20, 14), vit(20, 78, 18, 16)] + self.assertEqual(self.stages(v, [mov(20, 10)], 1.0)[4], 'rem') + + def test_missing_movement_does_not_produce_rem(self): + v = [vit(m, 70, 20, 14) for m in (0, 5, 10, 15, 20)] + self.assertNotIn('rem', self.stages(v, [], 1.0)) + + def test_aba_smoothing(self): + v = [vit(0, 60, 40, 14), vit(5, 60, 40, 14), vit(10, 60, 40, 14)] + self.assertEqual(self.stages(v, [mov(0, 10), mov(5, 300), mov(10, 10)]), ['light'] * 3) + + def test_transition_wake_to_deep_passes_through_light(self): + # Epoch 0 wake by movement; epoch 1 deep by HR ratio; HR window kept varied so + # the outlier filter keeps it. + v = [vit(0, 70, 40, 14), vit(5, 58, 40, 14), vit(10, 66, 40, 14), vit(15, 72, 40, 14)] + raw = [ds.classify_epoch(h, x, m, 66.5, 1.0) for (_, h, x, _), m in zip(v, [300, None, None, None])] + self.assertEqual(raw[:2], ['wake', 'deep']) + self.assertEqual(self.stages(v, [mov(0, 300)], 1.0)[1], 'light') + + def test_outlier_filter_hard_limits_and_median(self): + out = ds.filter_outliers([vit(0, 40, 0.5, 30), vit(1, 60, 50, 14), vit(2, 61, 50, 14), + vit(3, 100, 50, 14), vit(4, 60, 50, 14), vit(5, 61, 50, 14)]) + self.assertEqual(out[0][1:], (None, None, None)) + self.assertIsNone(out[3][1]) # 100 vs window median 61, std ~15.6 → > 2σ + self.assertEqual(out[1][1], 60) + + def test_single_sample_window_keeps_hr(self): + v = [vit(0, None), vit(5, None), vit(10, 68, 40, 14), vit(15, None), vit(20, None)] + self.assertEqual(ds.filter_outliers(v)[2][1], 68) + + def test_movement_uses_five_minute_buckets_last_row_wins(self): + # Vitals at minute 1 rounds to bucket 0; movement rows at minutes 0 and 2 share it. + self.assertEqual(self.stages([vit(1, 60)], [mov(0, 10), mov(2, 300)]), ['wake']) + self.assertEqual(self.stages([vit(1, 60)], [mov(0, 300), mov(2, 10)]), ['light']) + + +class FormatTest(unittest.TestCase): + def test_fmt(self): + self.assertEqual([ds._fmt(v) for v in (None, 0.0, 0.5, 61.23456, 7)], ['', '0', '0.5', '61.235', 7]) + self.assertEqual([ds._fmt(v) for v in (1e-7, -0.0003)], ['1e-07', '-0.0003']) + + +class SummaryTest(Fixture): + def test_summary_reports_agreement_per_night(self): + write_export(self.export) + make_db(self.db_path).close() + out, _ = self.run_main(self.export, self.db_path, '--summary') + line = next(ln for ln in out.splitlines() if ln.startswith('2026-10-08')) + _night, side, rows, labeled, _mad, deployed, _ios, light = line.split() + self.assertEqual((side, rows, labeled), ('left', '40', '30')) + # 10 light / 10 deep / 10 rem Watch minutes. Movement-only mode only ever says + # light or wake, so both it and always-light score the 10 light minutes. + self.assertEqual(light, '33%') + self.assertEqual(deployed, '33%') + + def test_agreement_ignores_unlabeled_rows(self): + rows = [{'ts': 1, 'watch_stage': 'light'}, {'ts': 2, 'watch_stage': None}, {'ts': 3, 'watch_stage': 'deep'}] + self.assertEqual(ds.agreement(rows, {1: 'light', 2: 'deep', 3: 'light'}), 50.0) + self.assertIsNone(ds.agreement([{'ts': 1, 'watch_stage': None}], {})) + + +if __name__ == '__main__': + unittest.main()