diff --git a/docs/CAUSAL_STAGER.md b/docs/CAUSAL_STAGER.md new file mode 100644 index 00000000..d2a457a0 --- /dev/null +++ b/docs/CAUSAL_STAGER.md @@ -0,0 +1,235 @@ +# Causal sleep stager + +`CausalStager.observe` in `lib/src/onehz/sleep/causal_stager.dart`, exported +from `package:openstrap_analytics/onehz.dart`. + +It answers one question: **right now, using only what the band has reported so +far, is the sleeper awake, in NREM, or in REM — and if it cannot tell, why not?** + +## What it is + +A live, past-only version of `cardioStager`. It uses the same per-epoch +features, the same REM axis weights, the same REM cutoff and the same wake, +atonia and HR-floor rules. Where `cardioStager` looks at both sides of an epoch, +this looks only backwards. + +It is built for decisions that cannot wait for the night to end, such as waking +someone during estimated REM. + +## What it is not + +- **Not polysomnography.** It is a wrist autonomic estimate (tier `ESTIMATE`). + It never labels N1/N2/N3 and never diagnoses anything. +- **Not more accurate than `cardioStager`.** It has fewer smoothing passes. The + retrospective stager itself scores kappa 0.13 against PSG on the held-out + DREAMT split. Agreement with the offline hypnogram below is a consistency + check, not an accuracy figure. +- **Not a sleep-onset detector.** Start it at in-bed time (`prior == null`). If + you feed it the waking afternoon, its baselines come from waking physiology. +- **Not a probability.** `confidence` measures how much evidence there was (HR + and RR coverage, capped at 0.6). It does not say how likely the stage is to be + right. + +## API + +```dart +import 'package:openstrap_analytics/onehz.dart'; + +final obs = CausalStager.observe(window, priorState); // priorState == null at night start +obs.stage; // CausalStage.wake | nrem | rem | absent +obs.confidence; // 0 when absent, else 0.15 .. 0.6 +obs.evidenceAgeMs; // now minus the end of the newest second with valid HR + accel +obs.abstentionReason; // CausalAbstention?, non-null exactly when stage == absent +obs.runSec; // how long this stage has held (0 when absent) +obs.epochStartMs; // the 30 s epoch the stage describes +obs.note; // 'warmup:have=H,need=N' while warming up +obs.trace; // plain-data decision trace (features, thresholds, counters) +obs.nextState; // pass this to the next call +``` + +`CausalSampleWindow({required double nowMs, List hr, List accel, RrSeries? rr})` +uses the package's existing sample types. `nowMs` is the decision time, supplied +by the caller (the package has no clock). `hr == 0` means off-skin. + +### Rules the contract enforces + +- **No look-ahead.** Samples stamped at or after `nowMs` are ignored and counted + in `trace['ignoredFuture']`. An epoch is staged only once it has closed. +- **Deterministic.** The same sequence of windows gives byte-identical + observations and states. Chunking does not matter (30 s windows and one big + catch-up window give the same answer) as long as a single catch-up window + spans no more than `historyEpochs`. +- **Replacement-tolerant.** A window is authoritative for the time span it + covers, per channel. Re-sent, overlapping and corrected windows converge on the + state a clean feed would have produced. Samples more than 600 s behind the + newest stored sample are ignored (`trace['ignoredStale']`); their features are + frozen. Send whole seconds. +- **Never carries a stage across an abstention.** An epoch that cannot be staged + is `absent`; the next answer comes from the next good epoch. + +### Latency + +A stage describes the newest *closed* 30 s epoch, and the RR features use +trailing 3 to 5 minute windows whose centres sit 1.5 to 2.5 minutes behind the +epoch. That is the cost of not looking ahead. Measured against the offline +hypnogram, agreement peaks at a lag of only 30 s and falls off slowly after that, +so one night cannot pin the lag down more precisely than "about a minute or two". + +## State + +`CausalStagerState` is plain data: `toJson()` and `CausalStagerState.fromJson()` +(returns `null` for anything it did not write, so the caller restarts and pays a +warm-up instead of getting a guessed state). It holds: + +| field | contents | +|---|---| +| `config` | `warmupEpochs` (40 = 20 min), `maxEvidenceAgeSec` (120), `historyEpochs` (360 = 3 h), `remScoreCut` (the shipped 0.5) | +| `lastNowMs` | latest decision time; an earlier `nowMs` is a clock regression | +| `lastUsableSec` | newest second with valid on-skin HR and valid accel | +| `hrTail`, `accelTail`, `rrTail` | the last 15 minutes of raw samples (`[sec,bpm]`, `[sec,x,y,z]`, `[ts_ms,rr_ms]`) | +| `rows` | one feature row per closed 30 s epoch (up to `historyEpochs`): motion, HR, HR SD, SDNN, R(k), LF/HF, coverage counts, and the raw stage decided when the epoch closed | + +It is about 160 KB of JSON for a long night, and it crosses an isolate boundary +as is. A call costs about 5 ms on the VM; a 3 hour catch-up in one window takes +about 1 s. Run it in an isolate, not on the UI thread. + +## Abstention reasons + +`absent` always carries one of these. They are checked roughly in this order. + +| reason | meaning | +|---|---| +| `clockRegressed` | `nowMs` is earlier than a time already seen, or not finite. State is returned untouched. | +| `noEvidence` | no sample of any kind has ever arrived | +| `staleEvidence` | nothing at all arrived within `maxEvidenceAgeSec` of `nowMs` (disconnect, out of range) | +| `offWrist` | the newest closed epoch is mostly `hr == 0` | +| `missingHr` | no valid HR in the newest closed epoch (absent, NaN, negative) | +| `missingAccel` | no valid accel in it. HR is checked first, so an epoch with neither says `missingHr`. | +| `lowCoverage` | HR or accel covers under half the epoch (`minHrCoverage`), or recent history is mostly unusable | +| `warmup` | fewer than `warmupEpochs` usable epochs so far; `note` says `warmup:have=H,need=N` | + +The 20-minute warm-up is an engineering floor (enough epochs for the robust-z +baselines and the local HR gates to exist), not a physiological claim. Epochs +that were off-skin or too thin never count toward it. For a Natural Wake window +of `N` minutes, start collecting at least `N` plus the warm-up before the window +opens, plus margin for the lag above. + +## How it differs from `cardioStager` + +| | `cardioStager` | `CausalStager` | +|---|---|---| +| RR windows (SDNN, R(k), LF/HF) | centred, ±150 s and ±90 s | trailing, the same total lengths (300 s, 180 s) | +| 1 g reference for motion | centred 330 s median | **trailing 120 s median** (see below) | +| local HR gates | ±90 min | past 90 min | +| baselines (motion MAD, robust z) | whole night | expanding, up to 3 h | +| Webster rescore, bout consolidation | yes | **no**, they need the future side of a bout | +| mode filter | centred 3-epoch | trailing 2-of-3 vote | +| deep overlay, personal profile | yes | no | + +Same on purpose: motion/HR/SDNN/R(k)/LF-HF definitions, the physiologic RR gate, +`kRemWeight*` (the measured DREAMT effect sizes), `kDefaultRemScoreCut`, the wake +gate (`hr` above local median plus max(6, SD), or big movement with an HR lift +or a big-move predecessor), the atonia and local-p25 HR-floor REM preconditions, +`minHrCoverage`, and the 0.6 confidence ceiling. The shared pieces were lifted +out of `cardio_stager.dart` (`weightedAxisScore`, `cleanRrBeatsBetween`, +`remFeaturesFromBeats`, `kRemWeight*`) so there is one copy, and that refactor is +bit-identical: `cardioStager` output on the real night (stages, deep flags, +confidence, at two cutoffs and with no RR) is unchanged. + +**The 120 s motion reference is the one number chosen on data.** A trailing +window the same length as the centred one must wait for more than half of itself +to be the new posture, so 150 s after every static posture change reads as +movement and the wake gate fires. I tried 300, 180, 120 and 90 s on the one +in-tree night: wake over-call against the offline stager fell 104, 83, 62, 52 min +and kappa rose 0.305, 0.335, 0.369, 0.372. 120 s is the middle of the plateau. +It was selected on the same night it is validated on, so the validation below is +a little optimistic by exactly that much. + +## Validation + +`dart run tool/causal_stager_validate.dart` replays the in-tree real night +(`test/onehz/fixtures/real_night_2026_07_*.csv`, 8 h 54 min) through both stagers. + +**What exists to validate against.** There is no PSG-labelled set in this repo +(DREAMT, used by `tool/stager_harness.dart`, lives outside it). The one real night +has only Apple Watch whole-night minutes, no epoch labels. So everything below is +agreement with the offline stager, on one night from one person. + +Stage minutes: offline wake 26 / NREM 358 / REM 150; causal (staged epochs) +wake 62.5 / NREM 374 / REM 78; Apple Watch summary wake 3 / REM 162 / light 330 +/ deep 38. The causal stager staged 1029 of 1068 epochs; the 39 it declined were +all `warmup`. Replay was deterministic. + +Per-epoch agreement with the offline hypnogram (3 classes): + +| causal(k) against offline(k-d) | agreement | kappa | +|---|---|---| +| d = 0 | 70.3 % | 0.369 | +| d = 1 (best) | 71.3 % | 0.391 | +| d = 4 (2 min) | 69.2 % | 0.343 | +| d = 8 (4 min) | 67.6 % | 0.307 | + +Against the offline stage at the same epoch: wake recall 54 %, precision 22 %; +NREM recall 87 %, precision 79 %; REM recall 36 %, precision 69 %. The wake +precision is low because the causal stager does not bridge short arousals the way +Webster rescoring does after the fact. A short wake run is not yet known to be +short. Read `runSec` before acting on a wake. + +Warm-up length barely moves agreement (kappa 0.369 at 10 and 20 min, 0.365 at +30, 0.350 at 60) and costs staged coverage (98 %, 96 %, 95 %, 89 %). + +### False and late triggers for "wake during REM" + +A trigger is `stage == rem && runSec >= X`. The reference is the 16 REM bouts the +offline stager emits (several are under 5 minutes). Chance level for a trigger +landing on offline REM is 29 %, and within 5 minutes of it 47 %, so the "no false +trigger" test is only as strict as that. + +| X | trigger epochs | on offline REM | within 5 min of it | far from REM | bouts caught (of 16) | median lateness | tail epochs after a bout | +|---|---|---|---|---|---|---|---| +| 60 s | 116 | 70.7 % | 92.2 % | 9 | 14 | 1.0 min | 25 | +| 120 s | 69 | 71.0 % | 100 % | 0 | 11 | 1.5 min | 20 | +| 180 s | 46 | 67.4 % | 100 % | 0 | 7 | 2.5 min | 15 | +| 300 s | 26 | 57.7 % | 100 % | 0 | 3 | 0.5 min | 11 | + +What this says: + +- **False triggers are rare but not absent at short persistence.** At X = 60 s, + 8 % of trigger epochs sit more than 5 minutes from any offline REM. At 120 s and + above there were none. +- **Late triggers are real.** Trailing windows keep the tail of a REM bout in + view, so between a fifth (X = 60 s) and two fifths (X = 300 s) of trigger epochs + fall in the five minutes after a bout ended. For X up to 180 s the first trigger + inside a bout arrives a median 1 to 2.5 minutes after the bout starts. +- **It misses REM, and X = 300 s misses most of it.** REM recall per epoch is + 36 %. The five-minute rule catches 3 of 16 bouts. Two minutes catches 11 of 16. + The cause is mostly a flickery causal REM label (no bout consolidation) plus + the lag. If Natural Wake needs the five-minute rule it will abstain in most + windows on a night like this one. +- **Window simulation** (fire once, at the first trigger in `[T-N, T)`, over every + must-be-up time with a fully observable window): at X = 120 s the trigger fires + in 38 % of 15-minute windows, 56 % of 30-minute, 80 % of 60-minute and every + 120-minute window. All fires landed within 5 minutes of offline REM; 77 to 87 % landed exactly + on it. At X = 300 s: 14 %, 27 %, 43 % and 77 %. The remaining windows contained + offline REM and did not fire (missed), or contained none. + +These are one night. The offline stager is itself an estimate. Do not turn them +into a promise to users. + +## Open caveats + +- One recorded night, no epoch labels, no PSG. The agreement figures are a + consistency check against another estimate. +- The 120 s motion reference was picked on the validation night. +- Wake is over-called relative to the offline stager (62 vs 26 min on the real + night) because Webster bridging is retrospective. +- REM recall is low and arrives late; the persistence threshold is a direct trade + between false triggers and missed ones. +- Baselines assume the stream starts at in-bed time. There is no sleep-onset + detector inside. +- A single catch-up window longer than `historyEpochs` (3 h) gives run lengths + that differ slightly from a live feed of the same data. The final stage and + state agree. +- The decision rules are re-implemented (the offline ones are inline in + `classifyCardioEpochs`). They are covered by the real-night agreement test, not + by a line-for-line equivalence test. diff --git a/lib/aggregate.dart b/lib/aggregate.dart index 8fe963d1..65a803d6 100644 --- a/lib/aggregate.dart +++ b/lib/aggregate.dart @@ -8,7 +8,7 @@ export 'src/aggregate/primitives.dart'; export 'src/onehz/clinical/hrv_time_aggregator.dart' hide hrvTimeOneSlice; - +export 'src/onehz/sleep/hypnogram_aggregator.dart'; export 'src/onehz/clinical/prsa_aggregator.dart'; export 'src/onehz/foundations/rr_sandwich.dart'; diff --git a/lib/src/onehz/sleep/advanced_stager.dart b/lib/src/onehz/sleep/advanced_stager.dart index 7d39f87b..75edc58f 100644 --- a/lib/src/onehz/sleep/advanced_stager.dart +++ b/lib/src/onehz/sleep/advanced_stager.dart @@ -36,12 +36,17 @@ // Pure Dart, dart:math only. import 'dart:math' as math; +import '../../aggregate/aggregator.dart'; import '../types.dart'; import '../util.dart'; import '../clinical/hrv_time.dart'; import 'cardio_stager.dart' show cardioStager, minHrCoverage; +import 'cardio_row_cache.dart'; import 'accounting.dart' show SleepStage; import 'band_offset.dart'; +import 'hypnogram.dart'; +export 'hypnogram.dart'; +import 'hypnogram_aggregator.dart'; /// Which per-session staging engine [AdvancedSleepStager] runs. See the file /// header for the 2026-07 comparison behind this choice. Do not switch the @@ -99,86 +104,6 @@ class RespTs { const RespTs(this.ts, this.raw); } -/// A scored stage span. [stage] ∈ {'wake','light','deep','rem'}. -class StageSegment { - final int start; - final int end; - final String stage; - const StageSegment(this.start, this.end, this.stage); - Map toJson() => {'start': start, 'end': end, 'stage': stage}; -} - -/// One detected sleep session + its hypnogram. -class SleepSession { - final int start; - final int end; - final double efficiency; // [0,1] - final List stages; - final int? restingHr; // lowest 5-min rolling-mean HR (bpm) - final double? avgHrv; // mean RMSSD over 5-min windows (ms) - const SleepSession({ - required this.start, - required this.end, - required this.efficiency, - required this.stages, - required this.restingHr, - required this.avgHrv, - }); -} - -/// AASM-style hypnogram metrics over a session. -class HypnogramMetrics { - final int tibS; - final int tstS; - final int sptS; - /// Sleep-onset latency (s). NULL when NOTHING staged as sleep — reporting - /// the whole time in bed as "time to fall asleep" alongside TST 0 is a - /// fabricated measurement, not an abstention. - final int? solS; - final double remLatencyS; // NaN if no REM - final int wasoS; - final double efficiency; // [0,1] - final int disturbances; - final double deepMin; - final double remMin; - final double lightMin; - final double deepPct; - final double remPct; - final double lightPct; - const HypnogramMetrics({ - required this.tibS, - required this.tstS, - required this.sptS, - required this.solS, - required this.remLatencyS, - required this.wasoS, - required this.efficiency, - required this.disturbances, - required this.deepMin, - required this.remMin, - required this.lightMin, - required this.deepPct, - required this.remPct, - required this.lightPct, - }); - Map toJson() => { - 'tib_s': tibS, - 'tst_s': tstS, - 'spt_s': sptS, - 'sol_s': solS, - 'rem_latency_s': remLatencyS.isNaN ? null : remLatencyS, - 'waso_s': wasoS, - 'efficiency': efficiency, - 'disturbances': disturbances, - 'deep_min': deepMin, - 'rem_min': remMin, - 'light_min': lightMin, - 'deep_pct': deepPct, - 'rem_pct': remPct, - 'light_pct': lightPct, - }; -} - /// Advanced sleep stager — constants + the full V1/V2 algorithm. class AdvancedSleepStager { // ── Stage-0 constants ─────────────────────────────────────────────────────── @@ -315,6 +240,7 @@ class AdvancedSleepStager { int tzOffsetSec = 0, int Function(int tsSec)? tzOffsetResolver, StagingMethod method = StagingMethod.cardio, + CardioRowCache? rowCache, List> wristOff = const [], // each [start,end] List> bandSleepState = const [], // each [ts,state] }) { @@ -387,7 +313,8 @@ class AdvancedSleepStager { } final stages = switch (method) { - StagingMethod.cardio => _stageSessionCardio(p.start, p.end, grav, hrS, rrS), + StagingMethod.cardio => + _stageSessionCardio(p.start, p.end, grav, hrS, rrS, rowCache), StagingMethod.v2 => _stageSessionV2(p.start, p.end, grav, hrS, rrS), StagingMethod.v1 => _stageSession(p.start, p.end, grav, hrS, rrS, respS), }; @@ -444,9 +371,11 @@ class AdvancedSleepStager { List rr = const [], List resp = const [], StagingMethod method = StagingMethod.cardio, + CardioRowCache? rowCache, }) { final stages = switch (method) { - StagingMethod.cardio => _stageSessionCardio(startSec, endSec, gravity, hr, rr), + StagingMethod.cardio => + _stageSessionCardio(startSec, endSec, gravity, hr, rr, rowCache), StagingMethod.v2 => _stageSessionV2(startSec, endSec, gravity, hr, rr), StagingMethod.v1 => _stageSession(startSec, endSec, gravity, hr, rr, resp), }; @@ -1455,7 +1384,8 @@ class AdvancedSleepStager { /// length, and not as 'wake' either: an abstained run can still carry HR, so /// `segmentSleep` would see it as observed and publish it as measured WASO. static List _stageSessionCardio(int start, int end, - List grav, List hr, List rr) { + List grav, List hr, List rr, + [CardioRowCache? rowCache]) { final span = end - start; if (span <= 0) return const []; final epSec = epochS.round(); @@ -1558,6 +1488,7 @@ class AdvancedSleepStager { rrMs: rrMs, rrTsMs: rrTsMs, runStarts: runStarts, + rowCache: rowCache, ); final stages = result.base.stages; // Empty = cardioStager abstained: everything stays 'unobserved'. @@ -2011,59 +1942,7 @@ class AdvancedSleepStager { static HypnogramMetrics hypnogramMetrics(SleepSession session) { final segs = [...session.stages]..sort((a, b) => a.start.compareTo(b.start)); - final tib = math.max(0, session.end - session.start); - final sleepSegs = segs - .where((s) => s.stage == 'light' || s.stage == 'deep' || s.stage == 'rem') - .toList(); - var tst = 0, deepS = 0, remS = 0, lightS = 0; - for (final s in sleepSegs) { - final d = s.end - s.start; - tst += d; - if (s.stage == 'deep') deepS += d; - if (s.stage == 'rem') remS += d; - if (s.stage == 'light') lightS += d; - } - int onset, sptEnd; - int? sol; - if (sleepSegs.isNotEmpty) { - onset = sleepSegs.first.start; - sptEnd = sleepSegs.last.end; - sol = math.max(0, onset - session.start); - } else { - onset = session.end; - sptEnd = session.end; - sol = null; // never observed sleep — no latency to report - } - final remSegs = sleepSegs.where((s) => s.stage == 'rem').toList(); - final remLatency = - remSegs.isNotEmpty ? (remSegs.first.start - onset).toDouble() : double.nan; - var waso = 0, disturbances = 0; - for (final s in segs.where((s) => s.stage == 'wake')) { - final w0 = math.max(s.start, onset); - final w1 = math.min(s.end, sptEnd); - if (w1 > w0) { - waso += w1 - w0; - disturbances++; - } - } - final se = tib > 0 ? tst / tib : 0.0; - double pct(int x) => tst > 0 ? x / tst * 100 : 0; - return HypnogramMetrics( - tibS: tib, - tstS: tst, - sptS: math.max(0, sptEnd - onset), - solS: sol, - remLatencyS: remLatency, - wasoS: waso, - efficiency: math.min(1.0, se), - disturbances: disturbances, - deepMin: deepS / 60, - remMin: remS / 60, - lightMin: lightS / 60, - deepPct: pct(deepS), - remPct: pct(remS), - lightPct: pct(lightS), - ); + return HypnogramAggregator(session.start, session.end).runList(segs); } // ── shared math (population std, percentile) ────────────────────────────── diff --git a/lib/src/onehz/sleep/cardio_row_cache.dart b/lib/src/onehz/sleep/cardio_row_cache.dart new file mode 100644 index 00000000..8d96427a --- /dev/null +++ b/lib/src/onehz/sleep/cardio_row_cache.dart @@ -0,0 +1,172 @@ +// Persistent, exact feature rows for the cardio sleep stager. + +import '../../aggregate/json_support.dart'; + +// Reformulates cardioStager() feature extraction and classification: van Hees +// 2013 motion and Long 2016 / Fonseca 2015/2018 cardiorespiratory staging. +// Internal fitted weights and rules remain ESTIMATE with low confidence. +// Session-onset anchoring and post-gap re-anchoring preserve the pre-bump grid; +// shared arithmetic preserves labels except for separately documented +// RR/spectral changes. Cached rows require matching dependencies; missing phases +// are recomputed. +/// A cache identity for one cardio feature epoch. +/// +/// [fingerprint] is an unsigned 64-bit FNV-1a digest assembled by the stager +/// from every input that can affect the row. It deliberately is not a Dart +/// [Object.hashCode]: cache files have to work across processes and releases. +class CardioRowKey { + final int startSec; + final int epochSec; + final int fingerprint; + + const CardioRowKey(this.startSec, this.epochSec, this.fingerprint); + + @override + bool operator ==(Object other) => + other is CardioRowKey && + startSec == other.startSec && + epochSec == other.epochSec && + fingerprint == other.fingerprint; + + @override + int get hashCode => Object.hash(startSec, epochSec, fingerprint); +} + +/// The seven raw feature values consumed by one cardio epoch classifier row. +class CardioEpochRow { + final double motion; + final double hr; + final double hrSd; + final double rmssd; + final double lfhf; + final double rk; + final double sdnn; + + const CardioEpochRow( + this.motion, this.hr, this.hrSd, this.rmssd, this.lfhf, this.rk, this.sdnn); +} + +/// Reusable, mergeable cache of exact cardio feature rows. +class CardioRowCache { + // 2: window RMSSD and R(k) skip seam pairs (upstream merge); older rows + // hold values computed across seams. + // 3: fingerprints are hashed byte by byte; older keys came from a word-wise + // mix in which two sign-bit changes cancelled. + static const int _version = 3; + final Map _rows = {}; + + CardioRowCache(); + + /// Number of successful lookups since this cache was created or decoded. + int hits = 0; + + /// Number of unsuccessful lookups since this cache was created or decoded. + int misses = 0; + + int get length => _rows.length; + + CardioEpochRow? lookup(CardioRowKey key) { + final row = _rows[key]; + if (row == null) { + misses++; + } else { + hits++; + } + return row; + } + + void insert(CardioRowKey key, CardioEpochRow row) { + _rows[key] = row; + } + + void retainWhere(bool Function(CardioRowKey key) keep) { + _rows.removeWhere((key, _) => !keep(key)); + } + + /// Union [other] into this cache. Rows already held here win identically-keyed + /// ties; an identical key can only describe an identical input row. + void merge(CardioRowCache other) { + for (final entry in other._rows.entries) { + _rows.putIfAbsent(entry.key, () => entry.value); + } + } + + Map toJson() { + final entries = _rows.entries.toList() + ..sort((a, b) { + final k = a.key, q = b.key; + var c = k.startSec.compareTo(q.startSec); + if (c != 0) return c; + c = k.epochSec.compareTo(q.epochSec); + if (c != 0) return c; + return k.fingerprint.compareTo(q.fingerprint); + }); + return { + 'version': _version, + 'rows': [ + for (final entry in entries) + { + 'startSec': entry.key.startSec, + 'epochSec': entry.key.epochSec, + // Decimal strings keep a 64-bit word lossless in JSON + // implementations whose numeric type is IEEE-754. + 'fingerprint': entry.key.fingerprint.toSigned(64).toString(), + 'motion': encodeDouble(entry.value.motion), + 'hr': encodeDouble(entry.value.hr), + 'hrSd': encodeDouble(entry.value.hrSd), + 'rmssd': encodeDouble(entry.value.rmssd), + 'lfhf': encodeDouble(entry.value.lfhf), + 'rk': encodeDouble(entry.value.rk), + 'sdnn': encodeDouble(entry.value.sdnn), + } + ], + }; + } + + factory CardioRowCache.fromJson(Map json) { + try { + if (json['version'] != _version) { + throw const FormatException('Unsupported CardioRowCache version'); + } + final raw = json['rows']; + if (raw is! List) throw const FormatException('Invalid CardioRowCache rows'); + final cache = CardioRowCache(); + for (final item in raw) { + if (item is! Map) throw const FormatException('Invalid CardioRowCache row'); + final row = item.cast(); + int readInt(String name) { + final value = row[name]; + if (value is! int) throw FormatException('Invalid $name'); + return value; + } + final fingerprint = row['fingerprint']; + if (fingerprint is! String) { + throw const FormatException('Invalid fingerprint'); + } + final fp = int.tryParse(fingerprint); + if (fp == null) { + throw const FormatException('Invalid fingerprint'); + } + final key = CardioRowKey(readInt('startSec'), readInt('epochSec'), fp); + if (key.epochSec <= 0) throw const FormatException('Invalid epochSec'); + if (cache._rows.containsKey(key)) { + throw const FormatException('Duplicate CardioRowCache row'); + } + cache._rows[key] = CardioEpochRow( + decodeDouble(row['motion']), + decodeDouble(row['hr']), + decodeDouble(row['hrSd']), + decodeDouble(row['rmssd']), + decodeDouble(row['lfhf']), + decodeDouble(row['rk']), + decodeDouble(row['sdnn']), + ); + } + return cache; + } on FormatException { + rethrow; + } catch (_) { + throw const FormatException('Invalid CardioRowCache'); + } + } +} diff --git a/lib/src/onehz/sleep/cardio_stager.dart b/lib/src/onehz/sleep/cardio_stager.dart index e13bf6c1..f5349b62 100644 --- a/lib/src/onehz/sleep/cardio_stager.dart +++ b/lib/src/onehz/sleep/cardio_stager.dart @@ -78,7 +78,9 @@ // tier ESTIMATE; confidence reflects RR coverage (RMSSD drives REM/deep). import 'dart:math' as math; +import 'dart:typed_data'; import '../types.dart'; +import 'cardio_row_cache.dart'; import '../util.dart'; import 'accounting.dart' show SleepStage; import 'stager.dart' show StagerResult, consolidateSleepStages; @@ -89,7 +91,26 @@ class CardioStagerResult { final StagerResult base; final List deepFlag; final double confidence; - const CardioStagerResult(this.base, this.deepFlag, this.confidence); + + /// Per-epoch REM / deep scores as the classify loop computed them, BEFORE + /// the mode filter, Webster rescore and consolidation. Null where the epoch + /// was decided wake first (no score taken) or no axis was measurable. Same + /// length as `base.stages`, or empty when the stager abstained or the result + /// was built without them. Read through `epochEvidence` (stage_ranges.dart). + final List remScores; + final List deepScores; + + /// Per-epoch: the epoch carried at least one valid HR sample. + final List hrCovered; + + const CardioStagerResult( + this.base, + this.deepFlag, + this.confidence, { + this.remScores = const [], + this.deepScores = const [], + this.hrCovered = const [], + }); } const int _epochSec = 30; @@ -117,6 +138,13 @@ const double _wRemSdnn = 0.32; const double _wRemHrSd = 0.27; const double _wRemLfhf = 0.12; +/// The REM axis weights, exposed (read-only) for [CausalStager], which scores +/// REM from the SAME measured effect sizes instead of re-declaring them. +const double kRemWeightRk = _wRemRk; +const double kRemWeightSdnn = _wRemSdnn; +const double kRemWeightHrSd = _wRemHrSd; +const double kRemWeightLfhf = _wRemLfhf; + const double _wDeepRk = 0.53; const double _wDeepHrSd = 0.43; const double _wDeepSdnn = 0.41; @@ -388,10 +416,65 @@ CardioStagerResult cardioStager( SleepUserProfile? userProfile, double remScoreCut = _remScoreCut, double deepScoreCut = _deepScoreCut, + CardioRowCache? rowCache, +}) { + final features = cardioEpochFeatures( + hr1hz, + accel, + rrMs: rrMs, + rrTsMs: rrTsMs, + runStarts: runStarts, + epochSec: epochSec, + rowCache: rowCache, + ); + final n = math.min(hr1hz.length, accel.length); + final cadence = runStarts.isEmpty ? null : sampleCadenceSeconds( + [for (var i = 0; i < n; i++) accel[i].tsMs / 1000.0]); + final perEpoch = math.max(1, (epochSec / (cadence ?? 1)).round()); + return classifyCardioEpochs( + features, + epochSec: epochSec, + segmentStarts: [for (final i in runStarts) + if (i > 0 && i ~/ perEpoch < features.length) i ~/ perEpoch], + userProfile: userProfile ?? cardioUserProfile, + remScoreCut: remScoreCut, + deepScoreCut: deepScoreCut, + ); +} + +/// Extracts the raw per-epoch inputs consumed by [classifyCardioEpochs]. +/// +/// The returned rows cover `[startEpoch, endEpoch)`, clamped to the session's +/// complete epoch grid. A row for epoch `e` depends on: its own `perEpoch` +/// samples of [hr1hz] and [accel]; accel magnitudes from `300 s / 2` on either +/// side of that epoch's start/end for its local 1 g median (clipped at the +/// session bounds); [rrMs]/[rrTsMs] in the inclusive ±150 s window centred on +/// the epoch for RMSSD/SDNN; and those inputs in the inclusive ±90 s window +/// for LF/HF and R(k). It also depends on the full input's measured accel +/// cadence, its clipped length, [epochSec], and the absolute accel timestamps +/// that define the RR windows. No row depends on neighbouring feature rows or +/// any user profile/classification setting. +/// +/// [rrTsMs] must be non-decreasing when it is paired with [rrMs]. Invalid grid +/// inputs return zero rows, as does a session with fewer than three epochs. +// Reformulates cardioStager() feature extraction and classification: van Hees +// 2013 motion and Long 2016 / Fonseca 2015/2018 cardiorespiratory staging. +// Internal fitted weights and rules remain ESTIMATE with low confidence. +// Session-onset anchoring and post-gap re-anchoring preserve the pre-bump grid; +// shared arithmetic preserves labels except for separately documented +// RR/spectral changes. Cached rows require matching dependencies; missing phases +// are recomputed. +CardioEpochFeatures cardioEpochFeatures( + List hr1hz, + List accel, { + List rrMs = const [], + List rrTsMs = const [], + List runStarts = const [], + int epochSec = _epochSec, + int startEpoch = 0, + int? endEpoch, + CardioRowCache? rowCache, }) { - // Explicit arg wins (unit tests); else the ambient profile the edge set for - // this staging pass; else null ⇒ pure per-night-local (cold-start behavior). - final profile = userProfile ?? cardioUserProfile; // `_cleanBeatsInWindow` binary-searches this, so it has to be non-decreasing. // The producer guarantees it — beats inherit their record's second and the // records are sorted, and the one path that folds in loose live beats @@ -421,7 +504,7 @@ CardioStagerResult cardioStager( // the Webster/Cole-Kripke continuity rules below are specified at this epoch // length. Stretching `epochSec` to fit the device would make a number appear // where the evidence for it does not; abstain instead. - if (cadenceSec == null || cadenceSec > epochSec) return _abstain(epochSec); + if (cadenceSec == null || cadenceSec > epochSec) return _emptyCardioFeatures; final perEpoch = math.max(1, (epochSec / cadenceSec).round()); // `perEpoch` samples at this cadence must actually SPAN `epochSec` — a // cadence that does not divide it evenly (4 s against a 30 s epoch rounds @@ -433,16 +516,40 @@ CardioStagerResult cardioStager( // Abstain rather than publish a grid whose real spacing does not match its // own label. if ((perEpoch * cadenceSec - epochSec).abs() > 1e-6) - return _abstain(epochSec); + return _emptyCardioFeatures; final nEpoch = n ~/ perEpoch; - if (nEpoch < 3) return _abstain(epochSec); + if (nEpoch < 3) return _emptyCardioFeatures; + assert(runStarts.every((i) => i % perEpoch == 0), 'runStarts must fall on epoch boundaries'); - final segStarts = [ - for (final i in runStarts) - if (i ~/ perEpoch > 0 && i ~/ perEpoch < nEpoch) i ~/ perEpoch - ]; - final segOf = _segmentOfEpoch(segStarts, nEpoch); + final segmentStarts = [for (final i in runStarts) + if (i > 0 && i ~/ perEpoch < nEpoch) i ~/ perEpoch]; + final segOf = _segmentOfEpoch(segmentStarts, nEpoch); + final firstEpoch = startEpoch.clamp(0, nEpoch); + final lastEpoch = (endEpoch ?? nEpoch).clamp(firstEpoch, nEpoch); + final rows = lastEpoch - firstEpoch; + + // The cache is deliberately inserted before the acceleration magnitude and + // median work. A hit avoids the Lomb–Scargle calculation AND all of the + // otherwise per-pass feature extraction. A miss range delegates back to this + // exact routine with the same full arrays, so its local median's clipped + // bounds and every floating-point operation remain byte-for-byte unchanged. + if (rowCache != null) { + return _cachedCardioEpochFeatures( + hr1hz, + accel, + rrMs, + rrTsMs, + epochSec, + n, + cadenceSec, + perEpoch, + firstEpoch, + lastEpoch, + rowCache, + runStarts, + ); + } // ── per-second ENMO (motion) against a LOCALLY-ADAPTIVE 1 g reference ────── // A single whole-night gravity-magnitude reference (the old approach) is @@ -471,78 +578,247 @@ CardioStagerResult cardioStager( const int _gRefWinSec = 300; // 5 min window, centered per epoch // …in SAMPLES at the measured cadence. 300 at 1 Hz, unchanged. final gRefHalf = math.max(1, (_gRefWinSec / cadenceSec).round()) ~/ 2; - final gRefByEpoch = List.filled(nEpoch, 1.0); - for (var e = 0; e < nEpoch; e++) { - final es = e * perEpoch; - // Clipped to this epoch's own run: across a gap the wearer may have - // changed posture, and a reference straddling two postures reads the - // offset between them as motion. - final segLo = segOf.lo[e] * perEpoch; - final segHi = segOf.hi[e] == nEpoch ? n : segOf.hi[e] * perEpoch; - final lo = math.max(segLo, es - gRefHalf); - final hi = math.min(segHi, es + perEpoch + gRefHalf); - gRefByEpoch[e] = median(mag.sublist(lo, hi)) ?? 1.0; - } + final gRefByEpoch = segmentStarts.isEmpty + ? _epochGRefMedians(mag, perEpoch, gRefHalf, firstEpoch, lastEpoch) + : [for (var e = firstEpoch; e < lastEpoch; e++) + median(mag.sublist( + math.max(segOf.lo[e] * perEpoch, e * perEpoch - gRefHalf), + math.min(segOf.hi[e] == nEpoch ? n : segOf.hi[e] * perEpoch, + (e + 1) * perEpoch + gRefHalf))) ?? 1.0]; // ── per-epoch features ───────────────────────────────────────────────────── - final motion = List.filled(nEpoch, 0); - final hr = List.filled(nEpoch, double.nan); - final hrSd = List.filled(nEpoch, 0); - final rmssd = List.filled(nEpoch, double.nan); - final sdnn = List.filled(nEpoch, double.nan); + final motion = List.filled(rows, 0); + final hr = List.filled(rows, double.nan); + final hrSd = List.filled(rows, 0); + final rmssd = List.filled(rows, double.nan); // REM autonomic features (P1): LF/HF from the RR spectrum + R(k) = rolling // mean |ΔIHR|. REM's signature at 1 Hz beat-timing is a SYMPATHETIC shift — // LF/HF rises, RR variability drops (low RMSSD), and instantaneous HR gets // "jittery" (elevated |ΔIHR|). The old REM rule leaned only on the RMSSD drop // and under-called REM; these two independent axes recover it (OR-combined // below), gated by atonia + an HR floor so a plain arousal is not miscalled. - final lfhf = List.filled(nEpoch, double.nan); - final rk = List.filled(nEpoch, double.nan); + final sdnn = List.filled(rows, double.nan); + final lfhf = List.filled(rows, double.nan); + final rk = List.filled(rows, double.nan); - for (var e = 0; e < nEpoch; e++) { + for (var e = firstEpoch; e < lastEpoch; e++) { + final row = e - firstEpoch; final s = e * perEpoch, t = math.min(s + perEpoch, n); // motion = mean ENMO over the epoch, against THIS epoch's local reference. - final gRefE = gRefByEpoch[e]; + final gRefE = gRefByEpoch[row]; var ms = 0.0; for (var i = s; i < t; i++) { final d = mag[i] - gRefE; ms += d > 0 ? d : 0.0; } - motion[e] = ms / (t - s); + motion[row] = ms / (t - s); // hr mean/sd over valid (>0) seconds final hv = [ for (var i = s; i < t; i++) if (hr1hz[i] > 0) hr1hz[i] ]; - if (hv.isNotEmpty) hr[e] = mean(hv)!; - if (hv.length >= 2) hrSd[e] = stddev(hv) ?? 0; + if (hv.isNotEmpty) hr[row] = mean(hv)!; + if (hv.length >= 2) hrSd[row] = stddev(hv) ?? 0; // rmssd over RR beats within a ±2.5-min window centred on the epoch - rmssd[e] = _windowRmssd(rrMs, rrTsMs, accel, s, t, epochSec); - sdnn[e] = _windowSdnn(rrMs, rrTsMs, accel, s, t, epochSec); + rmssd[row] = _windowRmssd(rrMs, rrTsMs, accel, s, t, epochSec); + sdnn[row] = _windowSdnn(rrMs, rrTsMs, accel, s, t, epochSec); // LF/HF + R(k) over a ±90-s RR window centred on the epoch. final rem = _windowRemFeatures(rrMs, rrTsMs, accel, s, t, epochSec); - if (rem.lfhf != null) lfhf[e] = rem.lfhf!; - if (rem.rk != null) rk[e] = rem.rk!; + if (rem.lfhf != null) lfhf[row] = rem.lfhf!; + if (rem.rk != null) rk[row] = rem.rk!; } - return classifyCardioEpochs( - CardioEpochFeatures( - motion: motion, - hr: hr, - hrSd: hrSd, - rmssd: rmssd, - lfhf: lfhf, - rk: rk, - sdnn: sdnn, - ), - epochSec: epochSec, - segmentStarts: segStarts, - userProfile: profile, - remScoreCut: remScoreCut, - deepScoreCut: deepScoreCut, + return CardioEpochFeatures( + motion: motion, + hr: hr, + hrSd: hrSd, + rmssd: rmssd, + lfhf: lfhf, + rk: rk, + sdnn: sdnn, ); } +/// Cache-aware extraction. The key's context is intentionally the contiguous +/// usable run supplied to this stager, not the caller's larger record: samples +/// outside that run cannot influence [cardioEpochFeatures]. +CardioEpochFeatures _cachedCardioEpochFeatures( + List hr1hz, + List accel, + List rrMs, + List rrTsMs, + int epochSec, + int n, + double cadenceSec, + int perEpoch, + int firstEpoch, + int lastEpoch, + CardioRowCache cache, + List runStarts, +) { + final count = lastEpoch - firstEpoch; + final keys = List.filled( + count, const CardioRowKey(0, 0, 0)); + final values = List.filled(count, null); + final missing = List.filled(count, false); + final gRefHalf = math.max(1, (300 / cadenceSec).round()) ~/ 2; + for (var e = firstEpoch; e < lastEpoch; e++) { + final row = e - firstEpoch; + final key = _cardioRowKey( + hr1hz, accel, rrMs, rrTsMs, epochSec, n, cadenceSec, perEpoch, e, gRefHalf, runStarts); + keys[row] = key; + final value = cache.lookup(key); + values[row] = value; + missing[row] = value == null; + } + + // One recursive feature extraction per contiguous cold span. It keeps the + // old vectorized median path intact and avoids turning a changed handful of + // rows into a handful of Lomb–Scargle setup passes. + var row = 0; + while (row < count) { + if (!missing[row]) { + row++; + continue; + } + final first = row; + while (row < count && missing[row]) { + row++; + } + final features = cardioEpochFeatures( + hr1hz, + accel, + rrMs: rrMs, + rrTsMs: rrTsMs, + epochSec: epochSec, + runStarts: runStarts, + startEpoch: firstEpoch + first, + endEpoch: firstEpoch + row, + ); + for (var i = first; i < row; i++) { + final f = i - first; + final value = CardioEpochRow( + features.motion[f], + features.hr[f], + features.hrSd[f], + features.rmssd[f], + features.lfhf[f], + features.rk[f], + features.sdnn[f], + ); + values[i] = value; + cache.insert(keys[i], value); + } + } + return CardioEpochFeatures( + motion: [for (final value in values) value!.motion], + hr: [for (final value in values) value!.hr], + hrSd: [for (final value in values) value!.hrSd], + rmssd: [for (final value in values) value!.rmssd], + lfhf: [for (final value in values) value!.lfhf], + rk: [for (final value in values) value!.rk], + sdnn: [for (final value in values) value!.sdnn], + ); +} + +const int _fnvOffset64 = 0xcbf29ce484222325; +const int _fnvPrime64 = 0x100000001b3; +const int _u64Mask = (1 << 64) - 1; + +/// Exact-bit fingerprint of the data one row can observe. The clipped run +/// length never enters a row as a global scalar: it only changes an epoch whose +/// 300-s gravity reference window touches a run edge. The explicit clip +/// distances below capture precisely that dependency while rows away from both +/// edges survive a moved session boundary. +CardioRowKey _cardioRowKey( + List hr, + List accel, + List rr, + List rrTs, + int epochSec, + int n, + double cadenceSec, + int perEpoch, + int epoch, + int gRefHalf, + List runStarts, +) { + var hash = _fnvOffset64; + // Byte-wise FNV-1a. Mixing a whole 64-bit word at a time lets two sign-bit + // flips cancel: flipping bit 63 survives every later multiply by the odd + // prime as the same flip, so two such changes undo each other. + void word(int value) { + var v = value & _u64Mask; + for (var i = 0; i < 8; i++) { + hash ^= v & 0xff; + hash = (hash * _fnvPrime64) & _u64Mask; + v >>>= 8; + } + } + void number(double value) { + final bytes = ByteData(8)..setFloat64(0, value, Endian.little); + word(bytes.getUint64(0, Endian.little)); + } + + final s = epoch * perEpoch; + final t = math.min(s + perEpoch, n); + var runLo = 0, runHi = n; + for (final boundary in runStarts) { + if (boundary <= s) runLo = math.max(runLo, boundary); + if (boundary > s) runHi = math.min(runHi, boundary); + } + final lo = math.max(runLo, s - gRefHalf); + final hi = math.min(runHi, t + gRefHalf); + word(0x43415244494f32); // domain separator: CARDIO2 (byte-wise hash) + word(epochSec); + word(perEpoch); + number(cadenceSec); + // Only an actually clipped edge is length-sensitive. A stable marker for an + // unclipped side means shifting a far-away session edge preserves the row. + word(lo == runLo ? gRefHalf - (s - runLo) : -1); + word(hi == runHi ? t + gRefHalf - runHi : -1); + word(hi - lo); + for (var i = lo; i < hi; i++) { + final a = accel[i]; + number(a.tsMs); + number(a.x); + number(a.y); + number(a.z); + } + word(t - s); + for (var i = s; i < t; i++) { + number(hr[i]); + } + // RR has no influence at all for an unpaired input, matching the feature + // extractor's early exit. Otherwise include every raw beat in the widest + // (±150 s), inclusive window; the ±90 s REM window is a subset. + if (rr.length != rrTs.length) { + word(0); + } else { + word(1); + final mid = (s + t) ~/ 2; + final centre = accel[mid].tsMs; + final low = centre - 150000.0, high = centre + 150000.0; + var a = 0, b = rrTs.length; + while (a < b) { + final m = (a + b) >> 1; + if (rrTs[m] < low) { + a = m + 1; + } else { + b = m; + } + } + var count = 0; + for (var i = a; i < rrTs.length && rrTs[i] <= high; i++) { + number(rrTs[i]); + number(rr[i]); + count++; + } + word(count); + } + return CardioRowKey((accel[s].tsMs / 1000.0).round(), epochSec, hash); +} + /// For each epoch, the [lo, hi) epoch range of the run it belongs to. ({List lo, List hi}) _segmentOfEpoch( List segmentStarts, int nEpoch) { @@ -619,6 +895,86 @@ class CardioEpochFeatures { ].reduce(math.min); } +const _emptyCardioFeatures = CardioEpochFeatures( + motion: [], + hr: [], + hrSd: [], + rmssd: [], + lfhf: [], + rk: [], + sdnn: [], +); + +/// The 1 g reference windows advance by an epoch, rather than one sample, and +/// have an even length at the normal 1 Hz / 30 s grid. That is not the centred +/// odd-width shape accepted by [slidingMedians], but it uses the same one +/// sorted-window binary-insert/remove algorithm and percentile arithmetic. +List _epochGRefMedians( + List mag, + int perEpoch, + int halfWin, + int firstEpoch, + int lastEpoch, +) { + final rows = lastEpoch - firstEpoch; + if (rows == 0) return []; + + final n = mag.length; + var lo = math.max(0, firstEpoch * perEpoch - halfWin); + var hi = math.min(n, firstEpoch * perEpoch + perEpoch + halfWin); + final sorted = []; + for (var i = lo; i < hi; i++) { + _insertEpochSorted(sorted, mag[i]); + } + + final out = List.filled(rows, 1.0); + for (var row = 0; row < rows; row++) { + out[row] = percentileSorted(sorted, 50) ?? 1.0; + if (row + 1 == rows) break; + + final epoch = firstEpoch + row + 1; + final nextLo = math.max(0, epoch * perEpoch - halfWin); + final nextHi = math.min(n, epoch * perEpoch + perEpoch + halfWin); + while (lo < nextLo) { + _removeEpochSorted(sorted, mag[lo]); + lo++; + } + while (hi < nextHi) { + _insertEpochSorted(sorted, mag[hi]); + hi++; + } + } + return out; +} + +void _insertEpochSorted(List sorted, double value) { + var lo = 0; + var hi = sorted.length; + while (lo < hi) { + final mid = lo + (hi - lo) ~/ 2; + if (sorted[mid].compareTo(value) < 0) { + lo = mid + 1; + } else { + hi = mid; + } + } + sorted.insert(lo, value); +} + +void _removeEpochSorted(List sorted, double value) { + var lo = 0; + var hi = sorted.length; + while (lo < hi) { + final mid = lo + (hi - lo) ~/ 2; + if (sorted[mid].compareTo(value) < 0) { + lo = mid + 1; + } else { + hi = mid; + } + } + sorted.removeAt(lo); +} + /// The staging DECISION layer: per-epoch features → stages + deep flags. /// /// Extracted from [cardioStager] verbatim (no behavioural change) so the rules @@ -844,6 +1200,10 @@ CardioStagerResult classifyCardioEpochs( // ── classify ─────────────────────────────────────────────────────────────── final stages = List.filled(nEpoch, SleepStage.wake); final deepFlag = List.filled(nEpoch, false); + // Evidence only: recorded for stageMinuteRanges, never read back here, so + // the labels below are unchanged by keeping them. + final remScores = List.filled(nEpoch, null); + final deepScores = List.filled(nEpoch, null); for (var e = 0; e < nEpoch; e++) { final hrMed = hrMedLocal[e]; final hrArousal = hrArousalLocal[e]; @@ -917,18 +1277,7 @@ CardioStagerResult classifyCardioEpochs( final hrSdZ = (sleepHrSd.length >= 4 && hrSd[e] > 0) ? hrSdScale?.z(hrSd[e]) : null; - // Weighted mean over the axes that are actually MEASURABLE this epoch, so - // a night with no usable RR falls back to the hrSd axis alone instead of - // silently scoring every epoch as if the missing axes voted "no". - double? score(List<(double?, double)> terms) { - var num = 0.0, den = 0.0; - for (final (z, w) in terms) { - if (z == null || z.isNaN) continue; - num += z * w; - den += w.abs(); - } - return den == 0 ? null : num / den; - } + final score = weightedAxisScore; final remScore = score([ (rkZ, _wRemRk), @@ -942,6 +1291,8 @@ CardioStagerResult classifyCardioEpochs( (sdnnZ, -_wDeepSdnn), (lfhfZ, -_wDeepLfhf), ]); + remScores[e] = remScore; + deepScores[e] = deepScore; // Atonia and the HR floor are RETAINED as gates, not folded into the score: // they are not weak evidence for REM, they are preconditions. A large @@ -1049,9 +1400,26 @@ CardioStagerResult classifyCardioEpochs( ), deepFlag, conf, + remScores: remScores, + deepScores: deepScores, + hrCovered: [for (var e = 0; e < nEpoch; e++) !hr[e].isNaN], ); } +/// Weighted mean of robust-z axis terms over the axes that are actually +/// MEASURABLE this epoch, so a night with no usable RR falls back to the hrSd +/// axis alone instead of silently scoring every epoch as if the missing axes +/// voted "no". Null when no axis is measurable. Shared with [CausalStager]. +double? weightedAxisScore(List<(double?, double)> terms) { + var num = 0.0, den = 0.0; + for (final (z, w) in terms) { + if (z == null || z.isNaN) continue; + num += z * w; + den += w.abs(); + } + return den == 0 ? null : num / den; +} + /// Honest "cannot stage this window" result: NO epochs, no deep flags, zero /// confidence. Callers must treat an empty [StagerResult.stages] as UNSTAGED /// (see `advanced_stager._stageSessionCardio`), never as sleep. @@ -1135,7 +1503,26 @@ double _windowSdnn(List rrMs, List rrTsMs, final mid = (s + t) ~/ 2; if (mid >= accel.length) return empty; final centreMs = accel[mid].tsMs; - final lo = centreMs - halfWinMs, hi = centreMs + halfWinMs; + return cleanRrBeatsBetween( + rrMs, rrTsMs, centreMs - halfWinMs, centreMs + halfWinMs); +} + +/// The physiologic-gated RR beats whose time falls in `[loMs, hiMs]` +/// (INCLUSIVE at both ends), with each beat's time rebased to [loMs] in seconds. +/// +/// The body of [_cleanBeatsInWindow], lifted out so [CausalStager] gathers its +/// trailing windows through the SAME gate (300-2000 ms, successive jump +/// <= 200 ms) instead of a second copy. [rrTsMs] must be non-decreasing and the +/// same length as [rrMs]; an empty or mismatched pair yields no beats. +({List beats, List tsSec, List seam}) cleanRrBeatsBetween( + List rrMs, + List rrTsMs, + double loMs, + double hiMs, +) { + const empty = (beats: [], tsSec: [], seam: []); + if (rrMs.isEmpty || rrTsMs.length != rrMs.length) return empty; + final lo = loMs, hi = hiMs; // [rrTsMs] is ascending, so lower-bound into it and stop at the far edge // instead of walking every beat of the night once per epoch per feature. // @@ -1344,8 +1731,15 @@ void _websterRescore(List sm, int epochSec) { // one; see [_cleanBeatsInWindow]. final win = _cleanBeatsInWindow(rrMs, rrTsMs, accel, s, t, halfWinMs: 90 * 1000); - final beats = win.beats; // clean RR (ms) - final beatTsSec = win.tsSec; // matching beat times (s), rebased to window + return remFeaturesFromBeats(win.beats, win.tsSec, win.seam); +} + +/// LF/HF and R(k) from already-gated beats ([beats] in ms, [beatTsSec] their +/// times in seconds, any common origin, [seam] as [cleanRrBeatsBetween] marks +/// it). The numeric half of [_windowRemFeatures], shared with [CausalStager]; +/// nulls when fewer than 16 beats (the spectral stability gate). +({double? lfhf, double? rk}) remFeaturesFromBeats( + List beats, List beatTsSec, List seam) { if (beats.length < 16) return (lfhf: null, rk: null); // spectral stability gate // R(k): mean absolute successive difference of instantaneous HR (bpm). @@ -1354,7 +1748,7 @@ void _websterRescore(List sm, int epochSec) { double? prevIhr; for (var k = 0; k < beats.length; k++) { final ihr = 60000.0 / beats[k]; - if (prevIhr != null && !win.seam[k]) { + if (prevIhr != null && !seam[k]) { rkSum += (ihr - prevIhr).abs(); rkCnt++; } diff --git a/lib/src/onehz/sleep/causal_stager.dart b/lib/src/onehz/sleep/causal_stager.dart new file mode 100644 index 00000000..2ec85138 --- /dev/null +++ b/lib/src/onehz/sleep/causal_stager.dart @@ -0,0 +1,1142 @@ +// SLEEP — CAUSAL (online) 3-class stager: wake | nrem | rem | absent. +// +// WHY THIS EXISTS. [cardioStager] is a RETROSPECTIVE night scorer: every +// feature window is centred on its epoch (so it reads 90-150 s of the future), +// the motion / HR / z-score baselines are whole-night, and three post-passes +// (mode filter, Webster rescore, bout consolidation) all look BOTH ways. None +// of that exists at 06:40 when something has to decide, right now, whether the +// sleeper is in REM. Re-running the retrospective stager every tick is not a +// causal stager either: its newest epochs would be scored on truncated windows +// and a baseline that is still moving. This file is the online counterpart, +// built from the SAME features, weights and cutoffs wherever those can be +// computed from the past alone: +// +// same motion (ENMO vs a local 1 g reference), HR mean / SD, SDNN, +// R(k), LF/HF; the physiologic RR gate; the REM axis weights +// (kRemWeight*, the measured DREAMT effect sizes); the shipped +// REM cutoff (kDefaultRemScoreCut); the wake / atonia / HR-floor +// rules; minHrCoverage; the 0.6 confidence ceiling. +// made causal every centred window becomes a TRAILING window ending at the +// epoch's end (300 s SDNN; 180 s R(k) and LF/HF — their full +// lengths; 120 s for the 1 g reference, see [_gRefWinSec]; 90 min +// for the local HR gates); +// whole-night baselines become expanding baselines over what has +// been seen so far (capped at [CausalStagerConfig.historyEpochs]). +// dropped Webster rescore and bout consolidation: both need the FUTURE +// side of a bout. Nothing replaces them; a causal wake is +// therefore not bridged the way the retrospective one is, and +// [CausalStageObservation.runSec] says how long a stage has +// persisted so the caller can demand persistence itself. The +// 3-epoch mode filter is replaced by a trailing 2-of-3 vote. +// not carried the personal profile blend ([SleepUserProfile]) and the deep +// overlay. This is wake / nrem / rem, per-night-local only. +// +// WHAT IT IS NOT. A wrist autonomic ESTIMATE (tier ESTIMATE), never +// polysomnography, never an EEG-defined stage, never a diagnosis. The offline +// stager scores kappa 0.13 against PSG on the held-out DREAMT split; this is +// a causal cousin of that stager with fewer smoothing passes, and agreement +// with the offline hypnogram is a consistency check, not an accuracy figure. +// `confidence` is EVIDENCE QUALITY (HR coverage, RR coverage), capped at +// [kMaxSleepConfidence] — it is not a probability that the stage is right. +// +// THE CONTRACT +// +// CausalStager.observe(window, priorState) -> CausalStageObservation +// +// * CAUSAL. Only samples stamped strictly before window.nowMs are read (a +// sample stamped t describes [t, t+1 s)); the rest are counted and ignored. An epoch is staged only after it has fully closed, +// and its features look backwards only. +// * REPLAYABLE. Pure and deterministic: no clock, no randomness, no I/O, no +// hash-order dependence. The caller supplies `nowMs`. The same sequence of +// windows gives byte-identical observations and states. +// * REPLACEMENT-TOLERANT. A window is authoritative for the time span it +// covers, per channel: stored samples inside that span are replaced by the +// window's. Re-sending a window, overlapping windows, and corrected +// re-sends converge on the state a clean feed would have produced. Samples +// older than 600 s behind the newest stored sample are IGNORED (counted in +// trace['ignoredStale']): their features are frozen. +// * SERIALISABLE. [CausalStagerState] is plain data (toJson/fromJson) and +// bounded: ~15 min of raw samples plus [CausalStagerConfig.historyEpochs] +// feature rows. It crosses an isolate boundary as-is. +// * HONEST. It abstains, with a [CausalAbstention] reason, instead of +// guessing: no / stale / regressing clock, off-skin, missing HR or accel, +// thin coverage, or warm-up evidence too thin. An absent stage always has +// confidence 0 and runSec 0, and a previous stage is never carried across +// an abstention. +// +// INPUT CONTRACT. 1 Hz HR and accel (one sample per second bucket; whole-second +// windows) plus RR beats whose timestamps are the record second (`rr_ts_ms`). +// Start from `prior == null` at in-bed time: the API stages whatever it is +// given and has no sleep-onset detector, so feeding it the waking afternoon +// would build its baselines from waking physiology. +// +// LATENCY, stated plainly: a stage describes the newest CLOSED 30 s epoch, and +// its RR features are trailing 3-5 min windows whose centres sit 1.5-2.5 min +// behind it — the price of not looking ahead. See docs/CAUSAL_STAGER.md for the +// measured agreement and the false- / late-trigger behaviour. + +import 'dart:math' as math; +import '../types.dart'; +import '../util.dart'; +import 'cardio_stager.dart' + show + cleanRrBeatsBetween, + kDefaultRemScoreCut, + kRemWeightHrSd, + kRemWeightLfhf, + kRemWeightRk, + kRemWeightSdnn, + minHrCoverage, + remFeaturesFromBeats, + weightedAxisScore; +import 'segment.dart' show kMaxSleepConfidence; + +// Published staging basis inherited from cardioStager(): Long 2016; Fonseca +// 2015/2018. This unpublished past-only adaptation is an internal rule, +// ESTIMATE with low +// confidence; trailing windows and baselines intentionally differ from +// retrospective staging. +/// Epoch length (s). Fixed: every cutoff inherited from [cardioStager] is +/// specified at 30 s. +const int kCausalEpochSec = 30; + +// Trailing window lengths (s). The RR windows keep the FULL length of the +// centred windows in cardio_stager.dart (RMSSD/SDNN ±150 s, R(k)/LF-HF ±90 s). +// +// The 1 g reference is the one place the full length does NOT carry over. The +// retrospective reference is a 330 s window CENTRED on the epoch, so after a +// static posture change it adapts as soon as the epoch reaches the change (it +// reads the future half). A trailing window of the same length has to wait +// until MORE THAN HALF of it is the new posture, i.e. 150 s of every posture +// change reads as "motion" — and a few consecutive big-move epochs are what the +// wake gate calls awake. That is the same posture artifact cardio_stager.dart's +// header describes, re-created by causality. A shorter trailing reference +// adapts in half its length while still being longer than the movement bouts +// it must not absorb. 120 s was picked from {300, 180, 120, 90} s by replaying +// the one in-tree night (test/onehz/fixtures; `tool/causal_stager_validate.dart`): +// wake over-call vs the offline stager fell 104 -> 83 -> 62 -> 52 min and kappa +// rose .305 -> .335 -> .369 -> .372. It is the one number in this file chosen +// on data, on a single night: treat the validation as optimistic by that much. +const int _gRefWinSec = 120; +const int _sdnnWinSec = 300; +const int _remWinSec = 180; + +/// Local HR gate window, epochs (90 min — one ultradian cycle; the retrospective +/// stager uses ±180 epochs, the causal one only the past half). +const int _hrWinEpochs = 180; + +/// Raw samples retained in the state (s). Must cover the longest feature +/// lookback ([_sdnnWinSec]) plus [_replaceHorizonSec]. +const int _rawRetainSec = 900; + +/// Samples this far behind the newest stored sample or older are ignored: a +/// row older than this has lost part of its lookback and is frozen. +const int _replaceHorizonSec = _rawRetainSec - _sdnnWinSec; + +/// Persisted-state schema version. Bump on any change to the state JSON. +const int kCausalStateVersion = 1; + +/// What the causal stager can say. [absent] always comes with an +/// [CausalAbstention] and zero confidence. +enum CausalStage { wake, nrem, rem, absent } + +/// Why the stager declined to name a stage. Checked roughly in declaration +/// order; [lowCoverage] is also raised for thin recent history once the +/// warm-up count is met. +enum CausalAbstention { + /// `nowMs` is earlier than a time already observed, or not finite. State is + /// returned untouched: acting on it would mean reading the future. + clockRegressed, + + /// No sample of any kind has ever been received. + noEvidence, + + /// Nothing at all arrived within [CausalStagerConfig.maxEvidenceAgeSec] of + /// `nowMs` (disconnect, strap out of range, delivery stalled). + staleEvidence, + + /// The newest closed epoch is mostly `hr == 0` (off-skin). Never read as + /// bradycardia. + offWrist, + + /// No valid HR in the newest closed epoch (absent, NaN or negative). + missingHr, + + /// No valid accel in the newest closed epoch. HR is checked first, so an + /// epoch with neither reports [missingHr]. + missingAccel, + + /// HR or accel covers under [minHrCoverage] of the epoch's seconds, or the + /// recent history is mostly unusable. + lowCoverage, + + /// Fewer than [CausalStagerConfig.warmupEpochs] usable epochs seen so far; + /// the baselines would be built on nothing. `note` reads + /// `warmup:have=H,need=N`. + warmup, +} + +/// Tunables. Carried inside [CausalStagerState] so a persisted state keeps the +/// settings it was built with. +class CausalStagerConfig { + /// Usable epochs required before any stage is named. 40 = 20 min. + /// An engineering floor (enough epochs for the robust-z baselines and the + /// local HR gates to exist), not a physiological claim. + final int warmupEpochs; + + /// Nothing received for longer than this ⇒ [CausalAbstention.staleEvidence]. + final double maxEvidenceAgeSec; + + /// Feature rows kept, which is also the span of the expanding baselines + /// (360 epochs = 3 h). + final int historyEpochs; + + /// REM decision cutoff on the weighted robust-z REM score. Defaults to the + /// shipped [kDefaultRemScoreCut]; exposed for calibration tooling only. + final double remScoreCut; + + const CausalStagerConfig({ + this.warmupEpochs = 40, + this.maxEvidenceAgeSec = 120, + this.historyEpochs = 360, + this.remScoreCut = kDefaultRemScoreCut, + }) : assert(warmupEpochs > 0), + assert(historyEpochs >= warmupEpochs), + assert(maxEvidenceAgeSec > 0); + + static const CausalStagerConfig defaults = CausalStagerConfig(); + + Map toJson() => { + 'warmup_epochs': warmupEpochs, + 'max_evidence_age_sec': maxEvidenceAgeSec, + 'history_epochs': historyEpochs, + 'rem_score_cut': remScoreCut, + }; + + static CausalStagerConfig? fromJson(Object? j) { + if (j is! Map) return null; + final w = j['warmup_epochs'], a = j['max_evidence_age_sec']; + final h = j['history_epochs'], c = j['rem_score_cut']; + if (w is! num || a is! num || h is! num || c is! num) return null; + if (w <= 0 || h < w || a <= 0 || !c.isFinite) return null; + return CausalStagerConfig( + warmupEpochs: w.toInt(), + maxEvidenceAgeSec: a.toDouble(), + historyEpochs: h.toInt(), + remScoreCut: c.toDouble(), + ); + } +} + +/// One call's worth of new (or re-sent) samples plus the decision time. +/// +/// Whole-second windows. Samples may overlap earlier windows (a re-send): see +/// the replacement rule in the file header. Timestamps are absolute epoch ms, +/// the same clock as [nowMs]. +class CausalSampleWindow { + /// Decision time (absolute epoch ms), supplied by the caller — this package + /// has no clock. Samples stamped at or after it are ignored (a sample + /// stamped t describes the second [t, t+1 s), which is not over yet). + final double nowMs; + final List hr; // hr == 0 is off-skin + final List accel; + final RrSeries rr; // tsMs = record second in ms; rrMs = interval + + CausalSampleWindow({ + required this.nowMs, + this.hr = const [], + this.accel = const [], + RrSeries? rr, + }) : rr = rr ?? RrSeries(const [], const []); +} + +/// One closed 30 s epoch's features. Plain data. +/// +/// `null` means "not measurable", never zero. [label] is the raw (unsmoothed) +/// stage decided when the epoch closed, from baselines built on epochs up to +/// and including this one — null when the epoch was unusable or the stager was +/// still warming up. +class CausalEpochRow { + /// Absolute index: floor(startMs / 30000). + final int epoch; + final int hrN; // seconds with valid on-skin HR (> 0) + final int hrOffN; // seconds with hr == 0 + final int accelN; // seconds with valid accel + final double? motion; // mean ENMO vs trailing 1 g reference (g) + final double? hr; // mean valid HR (bpm) + final double? hrSd; // SD of valid HR (bpm); null under 2 samples + final double? sdnn; // ms, trailing 300 s; null under 5 clean beats + final double? rk; // mean |ΔIHR| (bpm), trailing 180 s; null under 16 beats + final double? lfhf; // trailing 180 s; null under 16 beats + final CausalStage? label; + + const CausalEpochRow({ + required this.epoch, + required this.hrN, + required this.hrOffN, + required this.accelN, + this.motion, + this.hr, + this.hrSd, + this.sdnn, + this.rk, + this.lfhf, + this.label, + }); + + /// Seconds of valid HR / accel needed for the epoch to count. + static final int minSeconds = (minHrCoverage * kCausalEpochSec).ceil(); + + bool get offWrist => hrOffN > 0 && hrOffN >= hrN; + bool get usable => + !offWrist && hrN >= minSeconds && accelN >= minSeconds && hr != null; + + CausalEpochRow withLabel(CausalStage? l) => CausalEpochRow( + epoch: epoch, + hrN: hrN, + hrOffN: hrOffN, + accelN: accelN, + motion: motion, + hr: hr, + hrSd: hrSd, + sdnn: sdnn, + rk: rk, + lfhf: lfhf, + label: l, + ); + + List toJson() => + [epoch, hrN, hrOffN, accelN, motion, hr, hrSd, sdnn, rk, lfhf, label?.index]; + + static CausalEpochRow? fromJson(Object? j) { + if (j is! List || j.length != 11) return null; + for (var i = 0; i < 4; i++) { + if (j[i] is! num) return null; + } + double? d(Object? v) => v == null ? null : (v is num ? v.toDouble() : double.nan); + final vals = [for (var i = 4; i < 10; i++) d(j[i])]; + if (vals.any((v) => v != null && !v.isFinite)) return null; + final li = j[10]; + if (li != null && (li is! int || li < 0 || li > 2)) return null; + final row = CausalEpochRow( + epoch: (j[0] as num).toInt(), + hrN: (j[1] as num).toInt(), + hrOffN: (j[2] as num).toInt(), + accelN: (j[3] as num).toInt(), + motion: vals[0], + hr: vals[1], + hrSd: vals[2], + sdnn: vals[3], + rk: vals[4], + lfhf: vals[5], + label: li == null ? null : CausalStage.values[li as int], + ); + // Cross-field invariants a row built by the stager always satisfies. A row + // that breaks one would pass `usable` and then fail inside a decision + // (motion is read for every usable row), so it is refused here and the + // caller restarts, as the persistence contract says. + if (row.hrN < 0 || + row.hrOffN < 0 || + row.accelN < 0 || + row.hrN > kCausalEpochSec || + row.hrOffN > kCausalEpochSec || + row.accelN > kCausalEpochSec) { + return null; + } + if (row.usable && row.motion == null) return null; + // A label is only ever decided for a usable row. + if (row.label != null && !row.usable) return null; + return row; + } + + /// Whether every measured feature is the same (null counts as the same). + bool sameFeatures(CausalEpochRow o) => + hrN == o.hrN && + hrOffN == o.hrOffN && + accelN == o.accelN && + motion == o.motion && + hr == o.hr && + hrSd == o.hrSd && + sdnn == o.sdnn && + rk == o.rk && + lfhf == o.lfhf; +} + +/// The carried state. Plain data: JSON-serialisable and isolate-sendable. +/// +/// Treat every list as immutable. Raw tails are columns: +/// hr [second, bpm] accel [second, x, y, z] (valid only) +/// rr [ts_ms, rr_ms] +class CausalStagerState { + final CausalStagerConfig config; + + /// Latest `nowMs` observed; a smaller `nowMs` is a clock regression. + final double? lastNowMs; + + /// Latest second carrying valid on-skin HR AND valid accel, if ever. + final int? lastUsableSec; + final List> hrTail; + final List> accelTail; + final List> rrTail; + final List rows; // ascending by epoch, closed epochs only + + const CausalStagerState({ + required this.config, + this.lastNowMs, + this.lastUsableSec, + this.hrTail = const [], + this.accelTail = const [], + this.rrTail = const [], + this.rows = const [], + }); + + /// A fresh state — what `observe(window, null)` starts from. + factory CausalStagerState.initial( + [CausalStagerConfig config = CausalStagerConfig.defaults]) => + CausalStagerState(config: config); + + /// Newest stored second across all channels, if any. + int? get newestSec { + int? m; + void up(num v) { + final s = v.floor(); + if (m == null || s > m!) m = s; + } + + if (hrTail.isNotEmpty) up(hrTail.last[0]); + if (accelTail.isNotEmpty) up(accelTail.last[0]); + if (rrTail.isNotEmpty) up(rrTail.last[0] / 1000.0); + return m; + } + + Map toJson() => { + 'v': kCausalStateVersion, + 'cfg': config.toJson(), + 'last_now_ms': lastNowMs, + 'last_usable_sec': lastUsableSec, + 'hr': hrTail, + 'accel': accelTail, + 'rr': rrTail, + 'rows': [for (final r in rows) r.toJson()], + }; + + /// Null for anything that is not a current-version state this class wrote — + /// the caller starts fresh (which only costs a warm-up), it never gets a + /// guessed-at state. + static CausalStagerState? fromJson(Object? j) { + if (j is! Map || j['v'] != kCausalStateVersion) return null; + final cfg = CausalStagerConfig.fromJson(j['cfg']); + if (cfg == null) return null; + final now = j['last_now_ms'], usable = j['last_usable_sec']; + if (now != null && (now is! num || !now.isFinite)) return null; + if (usable != null && usable is! num) return null; + List>? cols(Object? v, int width) { + if (v is! List) return null; + final out = >[]; + for (final e in v) { + if (e is! List || e.length != width) return null; + final r = []; + for (final x in e) { + if (x is! num || !x.isFinite) return null; + r.add(x.toDouble()); + } + out.add(r); + } + return out; + } + + final hr = cols(j['hr'], 2), ac = cols(j['accel'], 4), rr = cols(j['rr'], 2); + final rowsJ = j['rows']; + if (hr == null || ac == null || rr == null || rowsJ is! List) return null; + final rows = []; + for (final r in rowsJ) { + final row = CausalEpochRow.fromJson(r); + if (row == null) return null; + rows.add(row); + } + return CausalStagerState( + config: cfg, + lastNowMs: (now as num?)?.toDouble(), + lastUsableSec: (usable as num?)?.toInt(), + hrTail: hr, + accelTail: ac, + rrTail: rr, + rows: rows, + ); + } +} + +/// One answer. [nextState] must be passed to the next call. +class CausalStageObservation { + /// The stage of the newest CLOSED 30 s epoch, or [CausalStage.absent]. + final CausalStage stage; + + /// Evidence quality in [0.15, kMaxSleepConfidence] when staged, exactly 0 + /// when absent. Not a probability of being right. + final double confidence; + + /// ms between `nowMs` and the end of the newest second that carried valid + /// on-skin HR and valid accel; null when there never was one. Present even + /// when absent — it is the staleness the caller should log. + final double? evidenceAgeMs; + + final CausalStagerState nextState; + + /// Non-null exactly when [stage] is [CausalStage.absent]. + final CausalAbstention? abstentionReason; + + /// Start (absolute ms) of the epoch [stage] describes; null if none. + final double? epochStartMs; + + /// How long [stage] has held, in seconds (whole epochs, trailing 2-of-3 + /// vote). 0 when absent. A caller wanting a "stable REM candidate" should + /// threshold this; `remEpisodeMinMin` (5 min) is the retrospective stager's + /// shortest credible REM bout. + final double runSec; + + /// `warmup:have=H,need=N` while warming up; null otherwise. + final String? note; + + /// Plain-data decision trace (features, thresholds, counters) for logging. + final Map trace; + + const CausalStageObservation({ + required this.stage, + required this.confidence, + required this.evidenceAgeMs, + required this.nextState, + required this.abstentionReason, + required this.epochStartMs, + required this.runSec, + required this.note, + required this.trace, + }); + + /// The observation without the state (persist [nextState] separately). + Map toJson() => { + 'stage': stage.name, + 'confidence': round6(confidence) ?? 0.0, + 'evidence_age_ms': evidenceAgeMs, + 'abstention_reason': abstentionReason?.name, + 'epoch_start_ms': epochStartMs, + 'run_sec': runSec, + if (note != null) 'note': note, + 'trace': trace, + }; +} + +/// The causal stager. See the file header for the contract. +abstract final class CausalStager { + /// Fold [window] into [priorState] (null = start of night) and name the stage + /// of the newest closed epoch, or abstain. + static CausalStageObservation observe( + CausalSampleWindow window, CausalStagerState? priorState) { + final st = priorState ?? CausalStagerState.initial(); + final cfg = st.config; + final now = window.nowMs; + + // 0. Clock. Going backwards would mean answering about a time the state has + // already moved past; an unusable clock cannot age anything. + if (!now.isFinite || (st.lastNowMs != null && now < st.lastNowMs!)) { + return _absent(st, CausalAbstention.clockRegressed); + } + + // 1. Merge (causal filter, span replacement) ----------------------------- + final m = _merge(st, window); + + // 2. Feature rows --------------------------------------------------------- + final newest = m.newestSec; + final kLatest = newest == null ? -1 : ((newest + 1) ~/ kCausalEpochSec) - 1; + final rows = _rebuildRows(st, cfg, m, kLatest); + + // 3. Bookkeeping that goes into the next state ---------------------------- + final usableSec = m.latestUsableSec ?? st.lastUsableSec; + final keepFromSec = (newest ?? 0) - _rawRetainSec; + final next = CausalStagerState( + config: cfg, + lastNowMs: now, + lastUsableSec: usableSec, + hrTail: [ + for (final e in m.hr) if (e[0] >= keepFromSec) e + ], + accelTail: [ + for (final e in m.accel) if (e[0] >= keepFromSec) e + ], + rrTail: [ + for (final e in m.rr) if (e[0] >= keepFromSec * 1000.0) e + ], + rows: rows, + ); + + final ageMs = usableSec == null + ? null + : math.max(0.0, now - (usableSec + 1) * 1000.0); + final trace = { + 'ignoredFuture': m.ignoredFuture, + 'ignoredStale': m.ignoredStale, + 'rows': rows.length, + }; + + CausalStageObservation no(CausalAbstention why, + {String? note, CausalEpochRow? row}) => + _mk(next, why, ageMs, trace, note: note, row: row); + + // 4. Reasons to abstain, most fundamental first --------------------------- + if (newest == null) return no(CausalAbstention.noEvidence); + if (now - (newest + 1) * 1000.0 > cfg.maxEvidenceAgeSec * 1000.0) { + return no(CausalAbstention.staleEvidence); + } + final byEpoch = {for (final r in rows) r.epoch: r}; + final usableCount = rows.where((r) => r.usable).length; + if (kLatest < 0) { + return no(CausalAbstention.warmup, + note: _warmNote(0, cfg.warmupEpochs)); + } + final row = byEpoch[kLatest]; + trace['epoch'] = kLatest; + if (row == null) return no(CausalAbstention.missingHr); + trace['hrN'] = row.hrN; + trace['hrOffN'] = row.hrOffN; + trace['accelN'] = row.accelN; + if (row.offWrist) return no(CausalAbstention.offWrist, row: row); + if (row.hrN == 0) return no(CausalAbstention.missingHr, row: row); + if (row.accelN == 0) return no(CausalAbstention.missingAccel, row: row); + if (!row.usable) return no(CausalAbstention.lowCoverage, row: row); + trace['usableRows'] = usableCount; + if (usableCount < cfg.warmupEpochs) { + return no(CausalAbstention.warmup, + note: _warmNote(usableCount, cfg.warmupEpochs), row: row); + } + final recent = _recentCoverage(byEpoch, rows.first.epoch, kLatest, cfg); + final span = recent.span; + final recentUsable = recent.usable, recentRr = recent.rr; + if (!recent.ok) { + return no(CausalAbstention.lowCoverage, row: row); + } + + // 5. Stage: trailing 2-of-3 vote over the labels decided at close time ---- + CausalStage? voted(int e) { + final here = byEpoch[e]?.label; + if (here == null) return null; + final counts = {here: 1}; + for (final d in const [1, 2]) { + final l = byEpoch[e - d]?.label; + if (l != null) counts[l] = (counts[l] ?? 0) + 1; + } + var best = here; + for (final en in counts.entries) { + if (en.value > counts[best]!) best = en.key; + } + return best; // a tie keeps the newest label + } + + final stage = voted(kLatest); + if (stage == null) { + // Warm and usable, but the label was never decided (e.g. the epoch + // closed before warm-up completed and nothing has rebuilt it). + return no(CausalAbstention.warmup, + note: _warmNote(usableCount, cfg.warmupEpochs), row: row); + } + var run = 1; + while (voted(kLatest - run) == stage) { + run++; + } + + final hrCov = recentUsable / span; + final rrCov = recentRr / span; + final conf = + ((0.35 + 0.25 * rrCov) * hrCov).clamp(0.15, kMaxSleepConfidence); + trace['raw'] = row.label?.name; + trace['motion'] = row.motion; + trace['hr'] = row.hr; + trace['hrSd'] = row.hrSd; + trace['sdnn'] = row.sdnn; + trace['rk'] = row.rk; + trace['lfhf'] = row.lfhf; + trace['hrCoverage'] = hrCov; + trace['rrCoverage'] = rrCov; + final why = _explain(rows, kLatest, cfg); + if (why != null) trace.addAll(why); + + return CausalStageObservation( + stage: stage, + confidence: conf.toDouble(), + evidenceAgeMs: ageMs, + nextState: next, + abstentionReason: null, + epochStartMs: kLatest * kCausalEpochSec * 1000.0, + runSec: run * kCausalEpochSec.toDouble(), + note: null, + trace: trace, + ); + } + + static String _warmNote(int have, int need) => 'warmup:have=$have,need=$need'; + + static CausalStageObservation _mk(CausalStagerState next, + CausalAbstention why, double? ageMs, Map trace, + {String? note, CausalEpochRow? row}) { + return CausalStageObservation( + stage: CausalStage.absent, + confidence: 0, + evidenceAgeMs: ageMs, + nextState: next, + abstentionReason: why, + epochStartMs: + row == null ? null : row.epoch * kCausalEpochSec * 1000.0, + runSec: 0, + note: note, + trace: trace, + ); + } + + static CausalStageObservation _absent( + CausalStagerState st, CausalAbstention why) { + return CausalStageObservation( + stage: CausalStage.absent, + confidence: 0, + evidenceAgeMs: null, + nextState: st, + abstentionReason: why, + epochStartMs: null, + runSec: 0, + note: null, + trace: { + 'ignoredFuture': 0, + 'ignoredStale': 0, + 'rows': st.rows.length, + }, + ); + } +} + +// ── Merge ──────────────────────────────────────────────────────────────────── + +class _Merged { + final List> hr; // [sec, bpm], ascending by sec + final List> accel; // [sec, x, y, z] + final List> rr; // [ts, rr], ascending by ts + final int ignoredFuture; + final int ignoredStale; + final int? dirtyFromSec; // earliest second touched by the window + final int? newestSec; + final int? latestUsableSec; + const _Merged(this.hr, this.accel, this.rr, this.ignoredFuture, + this.ignoredStale, this.dirtyFromSec, this.newestSec, + this.latestUsableSec); +} + +int _secOf(double tsMs) => (tsMs / 1000.0).floor(); + +_Merged _merge(CausalStagerState st, CausalSampleWindow w) { + final now = w.nowMs; + final prevNewest = st.newestSec; + // Anything this far behind (or more) the newest stored sample has frozen + // features: ignore it rather than half-apply it. + final staleBeforeMs = prevNewest == null + ? double.negativeInfinity + : (prevNewest - _replaceHorizonSec) * 1000.0; + var future = 0, stale = 0; + + // Time-acceptance, shared by all channels. + bool accept(double ts) { + if (!ts.isFinite) return false; + if (ts >= now) { + future++; + return false; + } + if (ts < staleBeforeMs) { + stale++; + return false; + } + return true; + } + + int? dirty; + void touch(int sec) { + if (dirty == null || sec < dirty!) dirty = sec; + } + + // HR: window span replaces stored span; last duplicate second wins. + final hrIn = {}; + int? hrLo, hrHi; + for (final s in w.hr) { + if (!accept(s.tsMs)) continue; + final sec = _secOf(s.tsMs); + hrLo = hrLo == null ? sec : math.min(hrLo, sec); + hrHi = hrHi == null ? sec : math.max(hrHi, sec); + if (s.hr.isFinite && s.hr >= 0) { + hrIn[sec] = s.hr; + } else { + hrIn.remove(sec); // an unusable sample is a missing one + } + } + final hr = _spliceSec(st.hrTail, hrLo, hrHi, { + for (final e in hrIn.entries) e.key: [e.key.toDouble(), e.value] + }); + if (hrLo != null) touch(hrLo); + + // Accel. + final acIn = >{}; + int? acLo, acHi; + for (final a in w.accel) { + if (!accept(a.tsMs)) continue; + final sec = _secOf(a.tsMs); + acLo = acLo == null ? sec : math.min(acLo, sec); + acHi = acHi == null ? sec : math.max(acHi, sec); + if (a.valid && a.x.isFinite && a.y.isFinite && a.z.isFinite) { + acIn[sec] = [sec.toDouble(), a.x, a.y, a.z]; + } else { + acIn.remove(sec); + } + } + final accel = _spliceSec(st.accelTail, acLo, acHi, acIn); + if (acLo != null) touch(acLo); + + // RR: ties on a second are normal, so no per-key upsert — the window's span + // replaces the stored span wholesale. + final n = math.min(w.rr.tsMs.length, w.rr.rrMs.length); + final rrIn = >[]; + double? rrLo, rrHi; + for (var i = 0; i < n; i++) { + final ts = w.rr.tsMs[i]; + if (!accept(ts)) continue; + rrLo = rrLo == null ? ts : math.min(rrLo, ts); + rrHi = rrHi == null ? ts : math.max(rrHi, ts); + final v = w.rr.rrMs[i]; + if (v.isFinite && v > 0) rrIn.add([ts, v]); + } + // Stable by timestamp; ties keep window order. + final order = List.generate(rrIn.length, (i) => i) + ..sort((a, b) { + final c = rrIn[a][0].compareTo(rrIn[b][0]); + return c != 0 ? c : a.compareTo(b); + }); + final rr = >[]; + if (rrLo == null) { + rr.addAll(st.rrTail); + } else { + for (final e in st.rrTail) { + if (e[0] < rrLo) rr.add(e); + } + for (final i in order) { + rr.add(rrIn[i]); + } + for (final e in st.rrTail) { + if (e[0] > rrHi!) rr.add(e); + } + touch(_secOf(rrLo)); + } + + int? newest; + void up(int s) { + if (newest == null || s > newest!) newest = s; + } + + if (hr.isNotEmpty) up(hr.last[0].toInt()); + if (accel.isNotEmpty) up(accel.last[0].toInt()); + if (rr.isNotEmpty) up(_secOf(rr.last[0])); + + // Newest second with valid on-skin HR AND valid accel. + int? usable; + { + final hrBy = {for (final e in hr) e[0].toInt(): e[1]}; + for (var i = accel.length - 1; i >= 0; i--) { + final sec = accel[i][0].toInt(); + final bpm = hrBy[sec]; + if (bpm != null && bpm > 0) { + usable = sec; + break; + } + } + } + return _Merged(hr, accel, rr, future, stale, dirty, newest, usable); +} + +/// Replace the stored entries with `lo <= sec <= hi` by [incoming]; keep the +/// rest; result ascending by second. +List> _spliceSec( + List> stored, int? lo, int? hi, Map> incoming) { + if (lo == null) return stored; + final out = >[ + for (final e in stored) if (e[0] < lo) e + ]; + final keys = incoming.keys.toList()..sort(); + for (final k in keys) { + out.add(incoming[k]!); + } + for (final e in stored) { + if (e[0] > hi!) out.add(e); + } + return out; +} + +// ── Feature rows ───────────────────────────────────────────────────────────── + +List _rebuildRows( + CausalStagerState st, CausalStagerConfig cfg, _Merged m, int kLatest) { + final oldestKept = kLatest - cfg.historyEpochs + 1; + final byEpoch = { + for (final r in st.rows) + if (r.epoch >= oldestKept && r.epoch <= kLatest) r.epoch: r + }; + final dirty = m.dirtyFromSec; + if (dirty != null && kLatest >= 0) { + final kLo = math.max((dirty / kCausalEpochSec).floor(), math.max(0, oldestKept)); + if (kLo <= kLatest) { + final hrBy = {for (final e in m.hr) e[0].toInt(): e[1]}; + final acBy = {for (final e in m.accel) e[0].toInt(): e}; + final rrTs = [for (final e in m.rr) e[0]]; + final rrV = [for (final e in m.rr) e[1]]; + final gRefMag = { + for (final e in acBy.entries) + e.key: math.sqrt(e.value[1] * e.value[1] + + e.value[2] * e.value[2] + + e.value[3] * e.value[3]) + }; + for (var k = kLo; k <= kLatest; k++) { + final row = _buildRow(k, hrBy, gRefMag, rrTs, rrV); + if (row == null) { + byEpoch.remove(k); + } else { + byEpoch[k] = row; + } + } + // Decide labels in epoch order, each on baselines built from rows up to + // and including its own epoch — never later ones. + final before = {for (final r in st.rows) r.epoch: r}; + for (var k = kLo; k <= kLatest; k++) { + final row = byEpoch[k]; + if (row == null) continue; + // Once history has rolled over, the rows an earlier epoch's baseline + // used are gone, so a re-decision here would run on a shorter baseline + // and could clear (or change) a label that nothing about the epoch has + // changed. An unchanged row keeps the label it was decided with. + final truncated = oldestKept > 0 && k - cfg.historyEpochs + 1 < oldestKept; + final prior = before[k]; + if (truncated && prior != null && prior.sameFeatures(row)) { + byEpoch[k] = row.withLabel(prior.label); + continue; + } + final d = _decide(byEpoch, k, cfg); + byEpoch[k] = row.withLabel(d?.stage); + } + } + } + final keys = byEpoch.keys.toList()..sort(); + return [for (final k in keys) byEpoch[k]!]; +} + +CausalEpochRow? _buildRow(int k, Map hrBy, + Map mag, List rrTs, List rrV) { + final s0 = k * kCausalEpochSec; + final hrVals = []; + var off = 0; + var anyHr = false; + for (var s = s0; s < s0 + kCausalEpochSec; s++) { + final b = hrBy[s]; + if (b == null) continue; + anyHr = true; + if (b > 0) { + hrVals.add(b); + } else { + off++; + } + } + final epochMags = [ + for (var s = s0; s < s0 + kCausalEpochSec; s++) + if (mag[s] != null) mag[s]! + ]; + if (!anyHr && epochMags.isEmpty) return null; + + // Motion: ENMO against the trailing 300 s median of |a| (the causal form of + // the retrospective stager's centred local 1 g reference). + double? motion; + if (epochMags.isNotEmpty) { + final ref = [ + for (var s = s0 + kCausalEpochSec - _gRefWinSec; + s < s0 + kCausalEpochSec; + s++) + if (mag[s] != null) mag[s]! + ]; + final g = median(ref) ?? 1.0; + var sum = 0.0; + for (final v in epochMags) { + final d = v - g; + sum += d > 0 ? d : 0.0; + } + motion = sum / epochMags.length; + } + + // RR: trailing windows ending at the epoch's last millisecond (a beat stamped + // at the next second boundary belongs to the next epoch). + final endMs = (s0 + kCausalEpochSec) * 1000.0 - 1e-3; + double? sdnn, rk, lfhf; + if (rrTs.isNotEmpty) { + final w300 = cleanRrBeatsBetween(rrV, rrTs, endMs - _sdnnWinSec * 1000.0, endMs); + if (w300.beats.length >= 5) sdnn = stddev(w300.beats); + final w180 = cleanRrBeatsBetween(rrV, rrTs, endMs - _remWinSec * 1000.0, endMs); + final rem = remFeaturesFromBeats(w180.beats, w180.tsSec, w180.seam); + rk = rem.rk; + lfhf = rem.lfhf; + } + + return CausalEpochRow( + epoch: k, + hrN: hrVals.length, + hrOffN: off, + accelN: epochMags.length, + motion: motion, + hr: hrVals.isEmpty ? null : mean(hrVals), + hrSd: hrVals.length >= 2 ? stddev(hrVals) : null, + sdnn: sdnn, + rk: rk, + lfhf: lfhf, + ); +} + +// ── Decision (the retrospective stager's rules, on expanding baselines) ────── + +class _Decision { + final CausalStage stage; + final double? remScore; + final double hrArousal, hrP25, hrMed; + final bool bigMove; + const _Decision(this.stage, this.remScore, this.hrArousal, this.hrP25, + this.hrMed, this.bigMove); +} + +/// The recent-coverage rule for epoch [k]: of the last `warmupEpochs` epochs +/// (fewer at the start of the held history, from [firstEpoch]) enough must be +/// usable. One rule for [observe] and for [_decide], so an epoch the observer +/// would abstain on never gets a stored label that a later run could count. +({int usable, int rr, int span, bool ok}) _recentCoverage( + Map byEpoch, int firstEpoch, int k, + CausalStagerConfig cfg) { + final from = math.max(firstEpoch, k - cfg.warmupEpochs + 1); + final span = k - from + 1; + var usable = 0, rr = 0; + for (var e = from; e <= k; e++) { + final r = byEpoch[e]; + if (r != null && r.usable) { + usable++; + if (r.sdnn != null) rr++; + } + } + return ( + usable: usable, + rr: rr, + span: span, + ok: usable / span >= minHrCoverage + ); +} + +/// Decide epoch [k] using only usable rows with epoch in (k - historyEpochs, k]. +/// Null when [k] is unusable, fewer than `warmupEpochs` usable rows exist, or +/// the recent-coverage rule of [observe] fails. +_Decision? _decide( + Map byEpoch, int k, CausalStagerConfig cfg) { + final row = byEpoch[k]; + if (row == null || !row.usable) return null; + if (!_recentCoverage(byEpoch, byEpoch.keys.reduce(math.min), k, cfg).ok) { + return null; + } + final base = [ + for (var e = k - cfg.historyEpochs + 1; e <= k; e++) + if (byEpoch[e] != null && byEpoch[e]!.usable) byEpoch[e]! + ]; + if (base.length < cfg.warmupEpochs) return null; + + // Motion thresholds: whole-window scalars (as offline: only the absolute + // magnitude references needed to be local). MAD 0 ⇒ the motion axis abstains. + final motSample = [for (final r in base) r.motion!]; + final motMed = median(motSample) ?? 0; + final motMadRaw = mad(motSample) ?? 0; + final motUsable = motMadRaw > 0 && motMadRaw.isFinite; + final motMad = motUsable ? motMadRaw : 0.0; + final stillCut = motMed + 1.5 * motMad; + final bigMoveCut = motMed + 5.0 * motMad; + bool still(CausalEpochRow r) => !motUsable || r.motion! <= stillCut; + bool bigMove(CausalEpochRow r) => motUsable && r.motion! > bigMoveCut; + + final sleepHr = [ + for (final r in base) + if (still(r)) r.hr! + ]; + final hrAll = [for (final r in base) r.hr!]; + final hrMedGlobal = median(sleepHr) ?? mean(hrAll)!; + + // Local trailing HR gates (retrospective: ±180 epochs; causal: past 180). + final win = [ + for (final r in base) + if (r.epoch >= k - _hrWinEpochs && still(r)) r.hr! + ]; + final sd = stddev(win); // before the sort: float sums are order-dependent + win.sort(); + final m = percentileSorted(win, 50); + double hrMed = hrMedGlobal, hrArousal = hrMedGlobal + 6.0, hrP25 = hrMedGlobal; + if (m != null) { + hrMed = m; + hrArousal = m + math.max(6.0, (sd ?? 6)); + hrP25 = percentileSorted(win, 25) ?? m; + } + + final hr = row.hr!; + final big = bigMove(row); + final prev = byEpoch[k - 1]; + final bigPrev = prev != null && prev.usable && bigMove(prev); + final hrUp = hr >= hrArousal; + final bigMoveWake = big && (hr >= hrMed || bigPrev); + if (hrUp || bigMoveWake) { + return _Decision(CausalStage.wake, null, hrArousal, hrP25, hrMed, big); + } + + // REM: weighted robust-z score over the axes measurable this epoch. + final sleepRk = [for (final r in base) if (still(r) && r.rk != null) r.rk!]; + final sleepSdnn = [for (final r in base) if (still(r) && r.sdnn != null) r.sdnn!]; + final sleepLfhf = [for (final r in base) if (still(r) && r.lfhf != null) r.lfhf!]; + final sleepHrSd = [ + for (final r in base) + if (still(r) && r.hrSd != null && r.hrSd! > 0) r.hrSd! + ]; + final rkZ = (sleepRk.length >= 4 && row.rk != null) + ? RobustScale.of(sleepRk)?.z(row.rk!) + : null; + final sdnnZ = (sleepSdnn.length >= 4 && row.sdnn != null) + ? RobustScale.of(sleepSdnn)?.z(row.sdnn!) + : null; + final lfhfZ = (sleepLfhf.length >= 4 && row.lfhf != null) + ? RobustScale.of(sleepLfhf)?.z(row.lfhf!) + : null; + final hrSdZ = (sleepHrSd.length >= 4 && row.hrSd != null && row.hrSd! > 0) + ? RobustScale.of(sleepHrSd)?.z(row.hrSd!) + : null; + final remScore = weightedAxisScore([ + (rkZ, kRemWeightRk), + (sdnnZ, kRemWeightSdnn), + (hrSdZ, kRemWeightHrSd), + (lfhfZ, kRemWeightLfhf), + ]); + final atonia = !big; + final hrTowardWake = hr >= hrP25; + final isRem = + remScore != null && remScore > cfg.remScoreCut && atonia && hrTowardWake; + return _Decision(isRem ? CausalStage.rem : CausalStage.nrem, remScore, + hrArousal, hrP25, hrMed, big); +} + +/// Trace fields for the newest epoch's decision, recomputed (not stored). +Map? _explain( + List rows, int k, CausalStagerConfig cfg) { + final byEpoch = {for (final r in rows) r.epoch: r}; + final d = _decide(byEpoch, k, cfg); + if (d == null) return null; + return { + 'remScore': d.remScore, + 'remCut': cfg.remScoreCut, + 'hrArousal': d.hrArousal, + 'hrP25': d.hrP25, + 'hrMedLocal': d.hrMed, + 'bigMove': d.bigMove, + }; +} diff --git a/lib/src/onehz/sleep/epoch_rows.dart b/lib/src/onehz/sleep/epoch_rows.dart new file mode 100644 index 00000000..4b44a731 --- /dev/null +++ b/lib/src/onehz/sleep/epoch_rows.dart @@ -0,0 +1,854 @@ +// Anchored 30 s epoch feature rows, accumulated. +// test/onehz/epoch_rows_test.dart and epoch_rows_anchor_test.dart pin the +// contract. +// +// * Epochs are [kEpochMs] steps from an anchor. The first gap-free run (the +// run structure of valid accelerometer coverage, with the stager's 60 s +// carry-forward as the gap threshold) is anchored at `anchorMs` when given +// (the staging onset: grid option 2), else at its first sample. A run that +// follows a gap longer than the carry re-anchors at its own first sample, +// as the pre-bump stager did. A run's anchor is final as soon as it is +// known. Rows start at the anchor, or at the epoch holding the first +// sample when the anchor lies before it; earlier samples only serve as +// feature context. +// * A row's features are `cardioEpochFeatures`' for that epoch: motion, hr, +// hrSd from the epoch's own seconds (motion against the local 1 g median +// of the +-150 s of accel magnitudes around it), rmssd / sdnn from the +// +-150 s RR window centred on the epoch, lfhf / rk from the +-90 s one. +// Lookback therefore reaches 165 s (accel, HR) and 150 s (RR) past an +// epoch's edges. +// * A row is FINAL once every sample it can read has arrived (newest sample +// is at least ~3 min past the epoch); the last few rows are PROVISIONAL and +// are recomputed with the data clipped at the newest sample. +// * Chunk invariance is exact: any chunking, with a save/restore through JSON +// text between chunks, gives the same rows as one fold. + +import 'dart:math' as math; +import 'dart:typed_data'; + +import '../../aggregate/json_support.dart'; +import 'advanced_stager.dart' show AdvancedSleepStager, StageSegment; +import 'accounting.dart' show SleepStage; +import 'cardio_stager.dart'; +import 'cardio_row_cache.dart'; +import '../types.dart'; +import '../foundations/rr_correction_kernel.dart' show SortedMultiset; +import '../util.dart'; + +// Reformulates cardioStager() feature extraction and classification: van Hees +// 2013 motion and Long 2016 / Fonseca 2015/2018 cardiorespiratory staging. +// Internal fitted weights and rules remain ESTIMATE with low confidence. +// Session-onset anchoring and post-gap re-anchoring preserve the pre-bump grid; +// shared arithmetic preserves labels except for separately documented +// RR/spectral changes. Cached rows require matching dependencies; missing phases +// are recomputed. +/// Epoch length, ms. +const int kEpochMs = 30000; + +/// Seconds a valid accelerometer sample covers: the stager's bounded +/// carry-forward plus the sample's own second. A valid sample within this many +/// seconds of the previous one continues the run; a later one starts a new run. +const int _kCoverSec = AdvancedSleepStager.maxAccelCarryForwardSec + 1; + +/// One gap-free run of valid accelerometer coverage and its epoch grid. +class _Run { + _Run(this.startSec, this.endSec, this.anchorMs, this.cursorMs); + int startSec, endSec; // coverage [startSec, endSec) in seconds + final int anchorMs; // first sample; epochs are anchorMs + k * kEpochMs + int cursorMs; // next epoch start to generate +} + +/// Raw features of one epoch; NaN means not measurable. +class EpochFeatureRow { + final int startMs; + final double motion, hr, hrSd, rmssd, lfhf, rk, sdnn; + final int _hrSeconds; + const EpochFeatureRow(this.startMs, this.motion, this.hr, this.hrSd, + this.rmssd, this.lfhf, this.rk, this.sdnn, + {int hrSeconds = 0}) + : _hrSeconds = hrSeconds; + + List _encode() => [ + startMs, + for (final v in [motion, hr, hrSd, rmssd, lfhf, rk, sdnn]) + encodeDouble(v), + _hrSeconds + ]; + + static EpochFeatureRow _decode(Object? value) { + final l = decodeList(value); + if (l.length != 9 || + l[0] is! int || + l[8] is! int || + (l[8] as int) < 0) { + throw const FormatException('Invalid epoch row'); + } + return EpochFeatureRow( + l[0] as int, + decodeDouble(l[1]), + decodeDouble(l[2]), + decodeDouble(l[3]), + decodeDouble(l[4]), + decodeDouble(l[5]), + decodeDouble(l[6]), + decodeDouble(l[7]), + hrSeconds: l[8] as int); + } +} + +/// Final feature rows and the raw time lookback needed by pending rows. +/// Each fold advances all input channels through the same watermark. A channel +/// omitted from a fold has no samples there, rather than arriving late later. +/// +/// [continuingAnchorMs]: the first valid accelerometer sample folded joins a +/// gap-free run whose first sample (at that time) is not being re-folded, so the +/// grid steps from there. Use it to start folding mid-run without computing the +/// run's earlier rows. +/// +/// [anchorMs]: the first run's grid passes through this absolute time (rows are +/// `anchorMs + k * kEpochMs`). Without either option the first sample anchors. +class EpochRows { + EpochRows({CardioRowCache? rowCache, int? continuingAnchorMs, int? anchorMs}) + : _cache = rowCache, + _seedMs = continuingAnchorMs, + _anchorReq = continuingAnchorMs == null ? anchorMs : null; + final CardioRowCache? _cache; + int? _seedMs; + int? _anchorReq; // requested first-run anchor, until the first run opens + final _hr = []; + final _accel = []; + final _rr = [], _rt = []; + final _final = []; + final _pending = {}; + // Accel coverage with the same bounded 60-second carry as the stager, and + // each run's grid. These cost one entry per actual outage, not one per second. + final _runs = <_Run>[]; + int _genRun = 0; // runs before this one have generated all their epochs + int? _newest; + + int? get newestMs => _newest; + + /// Anchor of the first run's grid, mod [kEpochMs]; null while unknown. + int? get phaseMs { + final a = _runs.isNotEmpty ? _runs.first.anchorMs : (_seedMs ?? _anchorReq); + return a == null ? null : a % kEpochMs; + } + + /// First sample (ms) of every gap-free run seen so far: the anchors of the + /// epoch grids. Final as soon as the run's first sample has been folded. + List get runAnchorsMs => [for (final r in _runs) r.anchorMs]; + + /// Start of the epoch (of the run whose coverage includes [tsMs]) that holds + /// [tsMs], or null when no run covers it. + int? epochStartMsFor(int tsMs) { + for (var i = _runs.length - 1; i >= 0; i--) { + final r = _runs[i]; + if (tsMs >= r.anchorMs) { + return tsMs < r.endSec * 1000 ? _epochIn(r, tsMs) : null; + } + } + return null; + } + + // The run whose grid holds the epoch starting at [startMs]. + _Run? _runAt(int startMs) { + for (var i = _runs.length - 1; i >= 0; i--) { + if (startMs >= _runs[i].anchorMs) return _runs[i]; + } + return null; + } + + static int _epochIn(_Run r, int tsMs) => + r.anchorMs + (tsMs - r.anchorMs) ~/ kEpochMs * kEpochMs; + + // Extends the last run or opens a new one for a valid accel sample at tsMs. + void _cover(double tsMs) { + final ms = tsMs.floor(); + final second = (tsMs / 1000).floor(); + if (_runs.isNotEmpty && second <= _runs.last.endSec) { + final run = _runs.last; + run.endSec = math.max(run.endSec, second + _kCoverSec); + return; + } + final seed = _runs.isEmpty ? _seedMs : null; + if (seed != null && seed > ms) { + throw ArgumentError('continuing anchor is after the first sample'); + } + final want = _runs.isEmpty ? _anchorReq : null; + if (seed == null && want != null) { + // Rows start at the anchor, or at the epoch holding the first sample + // when the anchor lies before it. + _runs.add(_Run(second, second + _kCoverSec, want, + want >= ms ? want : want + (ms - want) ~/ kEpochMs * kEpochMs)); + } else { + final anchor = seed ?? ms; + _runs.add(_Run((anchor / 1000).floor(), second + _kCoverSec, anchor, + anchor + (ms - anchor) ~/ kEpochMs * kEpochMs)); + } + _seedMs = null; + _anchorReq = null; + } + + // Opens the epochs that exist so far: starts on a run's grid that lie inside + // its coverage and are not after the newest sample. + void _generate() { + for (var i = _genRun; i < _runs.length; i++) { + final r = _runs[i]; + final limit = math.min(r.endSec * 1000, _newest! + 1); + for (; r.cursorMs < limit; r.cursorMs += kEpochMs) { + _pending.add(r.cursorMs); + } + // only the newest run can still grow + if (i + 1 < _runs.length) _genRun = i + 1; + } + } + + // Rows of the trailing carry (epochs that start after a run's last real + // sample) exist for staging only: the pre-bump stager classified them, with + // the carried vector as their accelerometer. The row API hides them. + List _visible(List rows) { + final out = []; + var i = 0; + for (final row in rows) { + while (i + 1 < _runs.length && _runs[i + 1].anchorMs <= row.startMs) { + i++; + } + if (row.startMs < (_runs[i].endSec - _kCoverSec + 1) * 1000) { + out.add(row); + } + } + return out; + } + + List get finalRows => List.unmodifiable(_visible(_final)); + + void fold( + {List hr = const [], + List accel = const [], + List rrMs = const [], + List rrTsMs = const [], + int? throughMs}) { + if (rrMs.length != rrTsMs.length) { + throw ArgumentError('RR columns differ in length'); + } + void check(Iterable times, double? last) { + for (final t in times) { + if (!t.isFinite || (last != null && t < last)) { + throw ArgumentError('samples must have finite, ascending times'); + } + if (_newest != null && t.floor() < _newest! - 240000 + kEpochMs) { + throw ArgumentError('sample precedes the settled watermark'); + } + last = t; + } + } + + check(hr.map((v) => v.tsMs), _hr.isEmpty ? null : _hr.last.tsMs); + check(accel.map((v) => v.tsMs), _accel.isEmpty ? null : _accel.last.tsMs); + check(rrTsMs, _rt.isEmpty ? null : _rt.last); + void observe(double t) { + final ms = t.floor(); + _newest = _newest == null ? ms : math.max(_newest!, ms); + } + + for (final h in hr) { + _hr.add(h); + observe(h.tsMs); + } + for (final a in accel) { + _accel.add(a); + observe(a.tsMs); + if (a.valid) _cover(a.tsMs); + } + _rr.addAll(rrMs); + _rt.addAll(rrTsMs); + for (final t in rrTsMs) { + observe(t); + } + if (throughMs != null) { + _newest = _newest == null ? throughMs : math.max(_newest!, throughMs); + } + if (_newest == null) return; + _generate(); + final starts = _pending.where((s) => s <= _newest! - 240000).toList() + ..sort(); + _final.addAll(_rows(starts)); + _pending.removeAll(starts); + // The oldest still-possible row starts after the final watermark; retain + // its widest left window, even when there are no pending rows in a gap. + final oldest = _newest! - 240000 + 1 - 165000; + _hr.removeWhere((h) => h.tsMs < oldest); + // 61 s more: carry from a dropped sample can still reach a live window. + _accel.removeWhere((a) => a.tsMs < oldest - 61000); + final drop = _lower(_rt, oldest.toDouble()); + if (drop > 0) { + _rr.removeRange(0, drop); + _rt.removeRange(0, drop); + } + } + + List _provisional() { + final starts = _pending.toList()..sort(); + return _rows(starts); + } + + List provisionalRows() => _visible(_provisional()); + + List allRows() => _visible([..._final, ..._provisional()]); + + // The accelerometer as the stager sees it: valid samples plus the bounded + // carry-forward (the last vector repeated each second, at most 60 s, until + // the next real sample), so features match a dense per-second array. + List _effAccel() { + final out = []; + AccelSample? prev; + void carry(int toSecIncl) { + final p = prev!; + final from = (p.tsMs / 1000).floor(); + final to = math.min(toSecIncl, from + AdvancedSleepStager.maxAccelCarryForwardSec); + for (var sec = from + 1; sec <= to; sec++) { + out.add(AccelSample(sec * 1000.0, p.x, p.y, p.z)); + } + } + + for (final a in _accel) { + if (!a.valid) continue; + if (prev != null) carry((a.tsMs / 1000).floor() - 1); + // Two valid samples at one time: the later one stands. + if (out.isNotEmpty && out.last.tsMs == a.tsMs) out.removeLast(); + out.add(a); + prev = a; + } + if (prev != null) carry(_newest! ~/ 1000); + return out; + } + + static int _lower(List times, double value) { + var a = 0, b = times.length; + while (a < b) { + final m = (a + b) >> 1; + if (times[m] < value) { + a = m + 1; + } else { + b = m; + } + } + return a; + } + + static int _sampleLower( + List samples, double value, double Function(T) time) { + var a = 0, b = samples.length; + while (a < b) { + final m = (a + b) >> 1; + if (time(samples[m]) < value) { + a = m + 1; + } else { + b = m; + } + } + return a; + } + + /// Where a run's data stops for the gravity reference. Batch staging cuts + /// every run that another one follows to whole epochs before extracting + /// features, and only the last run keeps its sub-epoch tail; the same bound + /// here keeps a reference window from reading samples the batch never sees. + double _refEndMs(_Run run) { + if (identical(run, _runs.last)) return run.endSec * 1000.0; + const epSec = kEpochMs ~/ 1000; + final whole = (run.endSec - run.startSec) ~/ epSec; + return (run.startSec + whole * epSec) * 1000.0; + } + + List _rows(List starts) { + if (starts.isEmpty) return []; + final eff = _effAccel(); + if (_cache != null) return [for (final s in starts) _row(s, eff)]; + // Slide the same sorted 330-second gravity window used by the batch + // extractor, computing each sample's magnitude once per fold. + final mag = [ + for (final a in eff) math.sqrt(a.x * a.x + a.y * a.y + a.z * a.z) + ]; + final sorted = SortedMultiset(); + var lo = 0, hi = 0; + final out = []; + for (final start in starts) { + // The gravity reference reads only the epoch's own run, as the pre-bump + // stager classified each run on its own. + final run = _runAt(start); + final nextLo = _sampleLower( + eff, + math.max(start - 150000.0, + run == null ? -double.infinity : run.startSec * 1000.0), + (a) => a.tsMs); + final nextHi = _sampleLower( + eff, + math.min(start + 180000.0, + run == null ? double.infinity : _refEndMs(run)), + (a) => a.tsMs); + while (lo < nextLo && lo < hi) { + sorted.remove(mag[lo]); + lo++; + } + lo = nextLo; + hi = math.max(hi, nextLo); + while (hi < nextHi) { + sorted.add(mag[hi]); + hi++; + } + out.add(_row(start, eff, reference: percentileSorted(sorted.a, 50) ?? 1.0)); + } + return out; + } + + EpochFeatureRow _row(int start, List eff, {double? reference}) { + final end = start + kEpochMs; + final run = _runAt(start); + final ab = _sampleLower( + eff, + math.max(start - (reference == null ? 150000.0 : 0.0), + run == null ? -double.infinity : run.startSec * 1000.0), + (a) => a.tsMs); + final ae = _sampleLower( + eff, + math.min(end + (reference == null ? 150000.0 : 0.0), + run == null ? double.infinity : _refEndMs(run)), + (a) => a.tsMs); + final hb = _sampleLower(_hr, start.toDouble(), (h) => h.tsMs); + final he = _sampleLower(_hr, end.toDouble(), (h) => h.tsMs); + final av = [ + for (var i = ab; i < ae; i++) + if (eff[i].valid) eff[i] + ]; + // One value per timestamp, the last one: a later record replaces an + // earlier one at the same time (a later 0 off-skin erases it). + final hv = [ + for (var i = hb; i < he; i++) + if (_hr[i].valid && (i + 1 == he || _hr[i + 1].tsMs != _hr[i].tsMs)) + _hr[i].hr + ]; + final low = start - 135000.0, high = start + 165000.0; + final rb = _lower(_rt, low), re = _lower(_rt, high + 0.001); + CardioRowKey? key; + if (_cache != null) { + var hash = 0xcbf29ce484222325; + final bits = ByteData(8); + // Byte-wise FNV-1a: mixing a whole word lets two sign-bit flips cancel. + void number(double v) { + bits.setFloat64(0, v, Endian.little); + for (var i = 0; i < 8; i++) { + hash = (hash ^ bits.getUint8(i)) * 0x100000001b3; + } + } + + number(0x45504f434834.toDouble()); // EPOCH4: byte-wise hash. + for (final a in av) { + number(a.tsMs); + number(a.x); + number(a.y); + number(a.z); + } + number(-1); + for (var i = hb; i < he; i++) { + number(_hr[i].tsMs); + number(_hr[i].hr); + } + number(-2); + for (var i = rb; i < re; i++) { + number(_rt[i]); + number(_rr[i]); + } + key = CardioRowKey(start ~/ 1000, 30, hash); + final hit = _cache.lookup(key); + if (hit != null) + return EpochFeatureRow(start, hit.motion, hit.hr, hit.hrSd, hit.rmssd, + hit.lfhf, hit.rk, hit.sdnn, + hrSeconds: hv.length); + } + final mag = [ + for (final a in av) math.sqrt(a.x * a.x + a.y * a.y + a.z * a.z) + ]; + final ref = reference ?? median(mag) ?? 1.0; + var motion = 0.0, count = 0; + for (var i = 0; i < av.length; i++) { + if (av[i].tsMs >= start && av[i].tsMs < end) { + final d = mag[i] - ref; + motion += d > 0 ? d : 0.0; + count++; + } + } + final beats = cleanRrBeatsBetween(_rr, _rt, low, high); + var ss = 0.0, pairs = 0; + for (var i = 1; i < beats.beats.length; i++) { + if (beats.seam[i]) continue; + final d = beats.beats[i] - beats.beats[i - 1]; + ss += d * d; + pairs++; + } + final remBeats = + cleanRrBeatsBetween(_rr, _rt, start - 75000.0, start + 105000.0); + final rem = + remFeaturesFromBeats(remBeats.beats, remBeats.tsSec, remBeats.seam); + final row = EpochFeatureRow( + start, + count == 0 ? double.nan : motion / count, + mean(hv) ?? double.nan, + hv.length >= 2 ? stddev(hv) ?? 0 : 0, + beats.beats.length >= 5 && pairs >= 4 + ? math.sqrt(ss / pairs) + : double.nan, + rem.lfhf ?? double.nan, + rem.rk ?? double.nan, + beats.beats.length >= 5 + ? stddev(beats.beats) ?? double.nan + : double.nan, + hrSeconds: hv.length); + if (key != null) + _cache!.insert( + key, + CardioEpochRow(row.motion, row.hr, row.hrSd, row.rmssd, row.lfhf, + row.rk, row.sdnn)); + return row; + } + + /// Classifies cached features with session-wide baselines. + /// Uses upstream's pooled-run baseline and per-run coverage rules. Continuity + /// passes receive segment boundaries so they cannot bridge an unobserved gap. Features retain + /// their observed context when a session's start or end moves. + List stageWindow(int startSec, int endSec) { + if (endSec <= startSec) return const []; + final rows = [..._final, ..._provisional()]; + final pieces = []; + var cursor = startSec; + final byStart = {for (final r in rows) r.startMs: r}; + final runs = <({int lo, int hi, List chosen})>[]; + final pooled = []; + final segmentStarts = []; + for (final run in _runs) { + final lo = math.max(math.max(startSec, run.startSec), run.anchorMs ~/ 1000), + hi = math.min(endSec, run.endSec); + if (hi <= lo) continue; + final first = lo * 1000 <= run.anchorMs + ? run.anchorMs + : _epochIn(run, lo * 1000); + final chosen = [ + for (var s = first; s + kEpochMs <= hi * 1000; s += kEpochMs) + byStart[s] ?? + EpochFeatureRow(s, 0, double.nan, 0, double.nan, double.nan, + double.nan, double.nan) + ]; + // Upstream gates HR per run before pooling. Missing-HR runs must not + // inherit the observed coverage of the other runs in the session. + if (chosen.length < 3 || + chosen.where((r) => !r.hr.isNaN).length < minHrCoverage * chosen.length) { + continue; + } + segmentStarts.add(pooled.length); + pooled.addAll(chosen); + runs.add((lo: lo, hi: hi, chosen: chosen)); + } + final result = classifyCardioEpochs(CardioEpochFeatures( + motion: [for (final r in pooled) r.motion], + hr: [for (final r in pooled) r.hr], + hrSd: [for (final r in pooled) r.hrSd], + rmssd: [for (final r in pooled) r.rmssd], + lfhf: [for (final r in pooled) r.lfhf], + rk: [for (final r in pooled) r.rk], + sdnn: [for (final r in pooled) r.sdnn]), + segmentStarts: segmentStarts); + final labels = result.base.stages; + var offset = 0; + for (final run in runs) { + final lo = run.lo, hi = run.hi, chosen = run.chosen; + if (lo > cursor) pieces.add(StageSegment(cursor, lo, 'unobserved')); + if (labels.isEmpty) { + pieces.add(StageSegment(lo, hi, 'unobserved')); + } else { + var at = lo; + for (var i = 0; i < chosen.length; i++) { + final g = offset + i; + final label = switch (labels[g]) { + SleepStage.wake => 'wake', + SleepStage.rem => 'rem', + SleepStage.nrem => result.deepFlag[g] ? 'deep' : 'light', + }; + final until = i + 1 == chosen.length + ? hi + : math.min(hi, chosen[i + 1].startMs ~/ 1000); + if (until > at) pieces.add(StageSegment(at, until, label)); + at = until; + } + } + offset += chosen.length; + cursor = hi; + } + if (cursor < endSec) pieces.add(StageSegment(cursor, endSec, 'unobserved')); + final out = []; + for (final p in pieces) { + if (out.isNotEmpty && + out.last.end == p.start && + out.last.stage == p.stage) { + final prev = out.removeLast(); + out.add(StageSegment(prev.start, p.end, p.stage)); + } else { + out.add(p); + } + } + return out; + } + + Map toJson() => { + 'type': 'EpochRows', + 'version': 4, + 'phase': phaseMs, + 'newest': _newest, + 'seed': _seedMs, + 'anchor': _anchorReq, + 'genRun': _genRun, + 'rows': [for (final r in _final) r._encode()], + 'pending': _pending.toList()..sort(), + 'hr': [ + for (final h in _hr) [h.tsMs, encodeDouble(h.hr)] + ], + 'accel': [ + for (final a in _accel) + [ + a.tsMs, + encodeDouble(a.x), + encodeDouble(a.y), + encodeDouble(a.z), + a.valid + ] + ], + 'rr': [for (final v in _rr) encodeDouble(v)], + 'rt': _rt, + 'runs': [ + for (final r in _runs) [r.startSec, r.endSec, r.anchorMs, r.cursorMs] + ], + }; + + factory EpochRows.fromJson(Map json) { + try { + if (json['type'] != 'EpochRows' || json['version'] != 4) { + throw const FormatException('Unsupported EpochRows checkpoint'); + } + final er = EpochRows( + continuingAnchorMs: field(json, 'seed') as int?, + anchorMs: field(json, 'anchor') as int?); + er._genRun = field(json, 'genRun') as int; + er._newest = field(json, 'newest') as int?; + er._final.addAll([ + for (final v in decodeList(field(json, 'rows'))) + EpochFeatureRow._decode(v) + ]); + final pending = decodeList(field(json, 'pending')).cast(); + er._pending.addAll(pending); + if (er._pending.length != pending.length) { + throw const FormatException('Duplicate pending epoch'); + } + for (final v in decodeList(field(json, 'hr'))) { + final l = decodeList(v); + if (l.length != 2) throw const FormatException('Invalid HR row'); + er._hr.add(HrSample(decodeDouble(l[0]), decodeDouble(l[1]))); + } + for (final v in decodeList(field(json, 'accel'))) { + final l = decodeList(v); + if (l.length != 5 || l[4] is! bool) + throw const FormatException('Invalid accel row'); + er._accel.add(AccelSample(decodeDouble(l[0]), decodeDouble(l[1]), + decodeDouble(l[2]), decodeDouble(l[3]), + valid: l[4] as bool)); + } + er._rr.addAll(decodeDoubles(field(json, 'rr'))); + er._rt.addAll(decodeDoubles(field(json, 'rt'))); + for (final v in decodeList(field(json, 'runs'))) { + final l = decodeList(v); + if (l.length != 4 || + l.any((x) => x is! int) || + (l[0] as int) >= (l[1] as int) || + (l[3] as int) < (l[2] as int) || + ((l[3] as int) - (l[2] as int)) % kEpochMs != 0 || + (er._runs.isNotEmpty && (l[0] as int) <= er._runs.last.endSec)) { + throw const FormatException('Invalid coverage run'); + } + er._runs.add(_Run(l[0] as int, l[1] as int, l[2] as int, l[3] as int)); + } + // Every epoch start belongs to the grid of the run that covers it. + bool onGrid(int s) { + for (var i = er._runs.length - 1; i >= 0; i--) { + final r = er._runs[i]; + if (s >= r.anchorMs) { + return s < r.endSec * 1000 && (s - r.anchorMs) % kEpochMs == 0; + } + } + return false; + } + bool ascending(Iterable values) { + double? prev; + for (final v in values) { + if (!v.isFinite || (prev != null && v < prev)) return false; + prev = v; + } + return true; + } + + if (er._rr.length != er._rt.length || + !ascending(er._hr.map((h) => h.tsMs)) || + !ascending(er._accel.map((a) => a.tsMs)) || + !ascending(er._rt) || + !ascending(er._final.map((r) => r.startMs.toDouble())) || + er._final.map((r) => r.startMs).toSet().length != er._final.length || + er._final.any((r) => !onGrid(r.startMs)) || + er._pending.any((s) => + !onGrid(s) || + (er._final.isNotEmpty && s <= er._final.last.startMs)) || + er._genRun > er._runs.length || + (er._runs.isEmpty && er._genRun != 0) || + (er._runs.isNotEmpty && er._genRun >= er._runs.length) || + (er._seedMs != null && er._runs.isNotEmpty) || + (er._anchorReq != null && er._runs.isNotEmpty) || + field(json, 'phase') != er.phaseMs || + (er._newest != null && + (er._final.any((r) => r.startMs > er._newest! - 240000) || + er._pending.any((s) => + s <= er._newest! - 240000 || s > er._newest!) || + er._pending.length > 20 || + er._hr.any((h) => h.tsMs >= er._newest! + 1) || + er._accel.any((a) => a.tsMs >= er._newest! + 1) || + er._rt.any((t) => t >= er._newest! + 1) || + er._runs.any((r) => r.endSec > er._newest! ~/ 1000 + _kCoverSec))) || + (er._newest == null && + (er._final.isNotEmpty || + er._pending.isNotEmpty || + er._hr.isNotEmpty || + er._accel.isNotEmpty || + er._rt.isNotEmpty))) { + throw const FormatException('Inconsistent EpochRows checkpoint'); + } + return er; + } on FormatException { + rethrow; + } catch (e) { + throw FormatException('Malformed EpochRows checkpoint: $e'); + } + } +} + +class _PhaseEntry { + _PhaseEntry(this.rows, this.anchorMs, this.hr, this.accel, this.rr); + final EpochRows rows; + final int anchorMs; // earliest anchor these rows were built for + int hr, accel, rr; // cumulative samples already folded, per channel +} + +/// Epoch rows keyed by grid phase (`anchorMs % kEpochMs`). The caller hands the +/// cumulative input (append-only between calls). A held phase folds only the +/// samples it has not consumed; an unheld phase, or a request for an earlier +/// anchor, is built from scratch (an exact recompute). At most [maxPhases] +/// phases are kept, least recently used dropped. +class EpochRowsMemo { + EpochRowsMemo({this.maxPhases = 2, CardioRowCache? rowCache}) + : _cache = rowCache; + final int maxPhases; + final CardioRowCache? _cache; + final _held = {}; // least recently used first + + /// Phases built from scratch, rebuilds included. + int phaseBuilds = 0; + + /// Samples handed to [EpochRows.fold] (hr + accel + rr). + int samplesFolded = 0; + + List get phases => List.unmodifiable(_held.keys); + + EpochRows rowsFor(int anchorMs, + {List hr = const [], + List accel = const [], + List rrMs = const [], + List rrTsMs = const []}) { + final phase = anchorMs % kEpochMs; + var e = _held.remove(phase); + if (e != null && + (anchorMs < e.anchorMs || + hr.length < e.hr || + accel.length < e.accel || + rrMs.length < e.rr)) { + e = null; + } + if (e == null) { + phaseBuilds++; + e = _PhaseEntry(EpochRows(anchorMs: anchorMs, rowCache: _cache), anchorMs, + 0, 0, 0); + } + if (hr.length > e.hr || accel.length > e.accel || rrMs.length > e.rr) { + e.rows.fold( + hr: hr.sublist(e.hr), + accel: accel.sublist(e.accel), + rrMs: rrMs.sublist(e.rr), + rrTsMs: rrTsMs.sublist(e.rr)); + samplesFolded += + hr.length - e.hr + accel.length - e.accel + rrMs.length - e.rr; + e.hr = hr.length; + e.accel = accel.length; + e.rr = rrMs.length; + } + _held[phase] = e; + while (_held.length > maxPhases) { + _held.remove(_held.keys.first); + } + return e.rows; + } + + Map toJson() => { + 'type': 'EpochRowsMemo', + 'version': 1, + 'maxPhases': maxPhases, + 'phases': [ + for (final en in _held.entries) + { + 'phase': en.key, + 'anchor': en.value.anchorMs, + 'consumed': [en.value.hr, en.value.accel, en.value.rr], + 'rows': en.value.rows.toJson(), + } + ], + }; + + factory EpochRowsMemo.fromJson(Map json) { + try { + if (json['type'] != 'EpochRowsMemo' || json['version'] != 1) { + throw const FormatException('Unsupported EpochRowsMemo checkpoint'); + } + final max = field(json, 'maxPhases'); + if (max is! int || max < 1) throw const FormatException('Invalid maxPhases'); + final m = EpochRowsMemo(maxPhases: max); + for (final v in decodeList(field(json, 'phases'))) { + final p = (v as Map).cast(); + final phase = field(p, 'phase'), anchor = field(p, 'anchor'); + final c = decodeList(field(p, 'consumed')); + if (phase is! int || + anchor is! int || + anchor % kEpochMs != phase || + c.length != 3 || + c.any((x) => x is! int || x < 0) || + m._held.containsKey(phase)) { + throw const FormatException('Invalid memo phase'); + } + final rows = EpochRows.fromJson( + (field(p, 'rows') as Map).cast()); + if (rows.phaseMs != phase) { + throw const FormatException('Memo phase contradicts its rows'); + } + m._held[phase] = + _PhaseEntry(rows, anchor, c[0] as int, c[1] as int, c[2] as int); + } + if (m._held.length > max) throw const FormatException('Too many phases'); + return m; + } on FormatException { + rethrow; + } catch (e) { + throw FormatException('Malformed EpochRowsMemo checkpoint: $e'); + } + } +} diff --git a/lib/src/onehz/sleep/hypnogram.dart b/lib/src/onehz/sleep/hypnogram.dart new file mode 100644 index 00000000..7889bdc6 --- /dev/null +++ b/lib/src/onehz/sleep/hypnogram.dart @@ -0,0 +1,79 @@ +/// A scored stage span. [stage] is normally 'wake', 'light', 'deep', or 'rem'. +class StageSegment { + final int start; + final int end; + final String stage; + const StageSegment(this.start, this.end, this.stage); + Map toJson() => {'start': start, 'end': end, 'stage': stage}; +} + +/// One detected sleep session and its hypnogram. +class SleepSession { + final int start; + final int end; + final double efficiency; // [0,1] + final List stages; + final int? restingHr; // lowest 5-min rolling-mean HR (bpm) + final double? avgHrv; // mean RMSSD over 5-min windows (ms) + const SleepSession({ + required this.start, + required this.end, + required this.efficiency, + required this.stages, + required this.restingHr, + required this.avgHrv, + }); +} + +/// AASM-style hypnogram metrics over a session. +class HypnogramMetrics { + final int tibS; + final int tstS; + final int sptS; + /// Sleep-onset latency (s). NULL when NOTHING staged as sleep — reporting + /// the whole time in bed as "time to fall asleep" alongside TST 0 is a + /// fabricated measurement, not an abstention. + final int? solS; + final double remLatencyS; // NaN if no REM + final int wasoS; + final double efficiency; // [0,1] + final int disturbances; + final double deepMin; + final double remMin; + final double lightMin; + final double deepPct; + final double remPct; + final double lightPct; + const HypnogramMetrics({ + required this.tibS, + required this.tstS, + required this.sptS, + required this.solS, + required this.remLatencyS, + required this.wasoS, + required this.efficiency, + required this.disturbances, + required this.deepMin, + required this.remMin, + required this.lightMin, + required this.deepPct, + required this.remPct, + required this.lightPct, + }); + Map toJson() => { + 'tib_s': tibS, + 'tst_s': tstS, + 'spt_s': sptS, + 'sol_s': solS, + 'rem_latency_s': remLatencyS.isNaN ? null : remLatencyS, + 'waso_s': wasoS, + 'efficiency': efficiency, + 'disturbances': disturbances, + 'deep_min': deepMin, + 'rem_min': remMin, + 'light_min': lightMin, + 'deep_pct': deepPct, + 'rem_pct': remPct, + 'light_pct': lightPct, + }; +} diff --git a/lib/src/onehz/sleep/hypnogram_aggregator.dart b/lib/src/onehz/sleep/hypnogram_aggregator.dart new file mode 100644 index 00000000..5710cd52 --- /dev/null +++ b/lib/src/onehz/sleep/hypnogram_aggregator.dart @@ -0,0 +1,190 @@ +import 'dart:math' as math; + +import '../../aggregate/aggregator.dart'; +import '../../aggregate/json_support.dart'; +import 'hypnogram.dart'; + +// Definition basis: Berry, R. B., et al. The AASM Manual for the Scoring of +// Sleep and Associated Events: Rules, Terminology and Technical Specifications. +// American Academy of Sleep Medicine. TST sums light/deep/REM durations; SOL +// runs from session start to first sleep; efficiency is TST / session duration. +// Application conventions: session bounds stand in for lights-off/on, stages +// are inferred rather than PSG-scored, and WASO counts explicit wake only +// between first sleep and the end of final sleep, excluding terminal wake. +// Gaps are not counted as wake. Sleep segments are summed without clipping +// to session bounds; efficiency is capped at 1. No sleep gives null SOL. +// Reformulation of AdvancedSleepStager.hypnogramMetrics(): sums integer stage +// durations and retains onset, final wake, first REM and wake-transition +// boundaries, reproducing its existing hypnogram accounting. +/// Mergeable AASM-style hypnogram metrics over start-sorted segments. +class HypnogramAggregator + extends Aggregator<_HypnogramState, ListBatch, HypnogramMetrics> { + HypnogramAggregator(this.sessionStart, this.sessionEnd); + + final int sessionStart; + final int sessionEnd; + + @override + _HypnogramState empty() => _HypnogramState(); + + @override + _HypnogramState addSlice( + _HypnogramState state, ListBatch batch, int start, int end) { + for (var i = start; i < end; i++) { + final segment = batch.rows[i]; + switch (segment.stage) { + case 'light': + case 'deep': + case 'rem': + final duration = segment.end - segment.start; + state.tst += duration; + if (segment.stage == 'light') state.light += duration; + if (segment.stage == 'deep') state.deep += duration; + if (segment.stage == 'rem') { + state.rem += duration; + if (!state.hasRem) { + state + ..hasRem = true + ..firstRemStart = segment.start; + } + } + if (!state.hasSleep) { + state + ..hasSleep = true + ..onset = segment.start; + } + state.sptEnd = segment.end; + case 'wake': + state.wakes.add((segment.start, segment.end)); + } + } + return state; + } + + @override + _HypnogramState combine(_HypnogramState left, _HypnogramState right) { + left + ..tst += right.tst + ..deep += right.deep + ..rem += right.rem + ..light += right.light + ..wakes.addAll(right.wakes); + if (!left.hasSleep && right.hasSleep) { + left + ..hasSleep = true + ..onset = right.onset; + } + if (right.hasSleep) left.sptEnd = right.sptEnd; + if (!left.hasRem && right.hasRem) { + left + ..hasRem = true + ..firstRemStart = right.firstRemStart; + } + return left; + } + + @override + bool get persistable => true; + + @override + Object? encodeState(_HypnogramState state) => { + 'tst': state.tst, + 'deep': state.deep, + 'rem': state.rem, + 'light': state.light, + 'hasSleep': state.hasSleep, + 'onset': state.onset, + 'sptEnd': state.sptEnd, + 'hasRem': state.hasRem, + 'firstRemStart': state.firstRemStart, + 'wakes': [ + for (final wake in state.wakes) [wake.$1, wake.$2] + ], + }; + + @override + _HypnogramState decodeState(Object? json) { + final map = decodeMap(json); + final state = _HypnogramState() + ..tst = _int(field(map, 'tst')) + ..deep = _int(field(map, 'deep')) + ..rem = _int(field(map, 'rem')) + ..light = _int(field(map, 'light')) + ..hasSleep = _bool(field(map, 'hasSleep')) + ..onset = _int(field(map, 'onset')) + ..sptEnd = _int(field(map, 'sptEnd')) + ..hasRem = _bool(field(map, 'hasRem')) + ..firstRemStart = _int(field(map, 'firstRemStart')); + for (final item in decodeList(field(map, 'wakes'))) { + final pair = decodeList(item); + if (pair.length != 2) throw const FormatException('Expected a wake pair'); + state.wakes.add((_int(pair[0]), _int(pair[1]))); + } + if (state.deep + state.rem + state.light != state.tst) { + throw const FormatException('Stage totals disagree with tst'); + } + if (state.hasRem && !state.hasSleep) { + throw const FormatException('REM without sleep'); + } + return state; + } + + @override + HypnogramMetrics finish(_HypnogramState state) { + final tib = math.max(0, sessionEnd - sessionStart); + final onset = state.hasSleep ? state.onset : sessionEnd; + final sptEnd = state.hasSleep ? state.sptEnd : sessionEnd; + var waso = 0; + var disturbances = 0; + for (final wake in state.wakes) { + final w0 = math.max(wake.$1, onset); + final w1 = math.min(wake.$2, sptEnd); + if (w1 > w0) { + waso += w1 - w0; + disturbances++; + } + } + final efficiency = tib > 0 ? state.tst / tib : 0.0; + double pct(int value) => state.tst > 0 ? value / state.tst * 100 : 0; + return HypnogramMetrics( + tibS: tib, + tstS: state.tst, + sptS: math.max(0, sptEnd - onset), + solS: state.hasSleep ? math.max(0, onset - sessionStart) : null, + remLatencyS: + state.hasRem ? (state.firstRemStart - onset).toDouble() : double.nan, + wasoS: waso, + efficiency: math.min(1.0, efficiency), + disturbances: disturbances, + deepMin: state.deep / 60, + remMin: state.rem / 60, + lightMin: state.light / 60, + deepPct: pct(state.deep), + remPct: pct(state.rem), + lightPct: pct(state.light), + ); + } +} + +class _HypnogramState { + var tst = 0; + var deep = 0; + var rem = 0; + var light = 0; + var hasSleep = false; + var onset = 0; + var sptEnd = 0; + var hasRem = false; + var firstRemStart = 0; + final wakes = <(int, int)>[]; +} + +int _int(Object? value) { + if (value is! int) throw const FormatException('Expected an int'); + return value; +} + +bool _bool(Object? value) { + if (value is! bool) throw const FormatException('Expected a bool'); + return value; +} diff --git a/lib/src/onehz/sleep/segment.dart b/lib/src/onehz/sleep/segment.dart index 2a9c7b65..3f2b2ef8 100644 --- a/lib/src/onehz/sleep/segment.dart +++ b/lib/src/onehz/sleep/segment.dart @@ -45,6 +45,7 @@ import 'van_hees.dart'; import 'accounting.dart' show SleepStage; import 'advanced_stager.dart'; import 'band_offset.dart'; +import 'cardio_row_cache.dart'; /// Minimum in-bed duration to qualify as the main sleep (ARCHITECTURE_V2: ~3 h). const int _minQualifyingSleepSec = 3 * 3600; @@ -451,6 +452,7 @@ SleepSegmentation segmentSleep( /// of being published as a night that ended at the cut. See /// [kUnfinishedNightEdgeSec]. bool recordContinues = false, + CardioRowCache? rowCache, }) { final n = math.min(accel.length, hr1hz.length); // A forced window (manual entry / user confirmation, Approach 1) is asserted @@ -518,7 +520,8 @@ SleepSegmentation segmentSleep( if (offsetSec <= onsetSec) return SleepSegmentation.absent; // Stage the user-asserted window directly — no detection, no gates. final session = - AdvancedSleepStager.stageWindow(onsetSec, offsetSec, grav, hr, rr: rr); + AdvancedSleepStager.stageWindow(onsetSec, offsetSec, grav, hr, + rr: rr, rowCache: rowCache); chosen = _SleepGroup( sessions: [session], start: onsetSec, @@ -531,6 +534,7 @@ SleepSegmentation segmentSleep( hr, rr: rr, tzOffsetResolver: tzAt, + rowCache: rowCache, ); if (sessions.isEmpty) return SleepSegmentation.absent; diff --git a/lib/src/onehz/sleep/sleep.dart b/lib/src/onehz/sleep/sleep.dart index 7ceb3766..34170220 100644 --- a/lib/src/onehz/sleep/sleep.dart +++ b/lib/src/onehz/sleep/sleep.dart @@ -27,7 +27,7 @@ export 'segment.dart'; export 'hr_fallback.dart'; export 'advanced_stager.dart'; export 'band_offset.dart'; - +export 'hypnogram.dart'; export 'sri.dart'; export 'accounting.dart'; // stager.dart provides StagerResult + the shared consolidation post-step. Its @@ -41,10 +41,10 @@ export 'cardio_stager.dart' cleanBeatsInWindowForTest, cleanBeatDiffsInWindowForTest, windowRmssdRkForTest; - - - - +export 'stage_ranges.dart'; +export 'cardio_row_cache.dart'; +export 'epoch_rows.dart'; +export 'causal_stager.dart'; export 'circadian_np.dart'; export 'cycles.dart'; export 'night_hrv_shape.dart'; diff --git a/lib/src/onehz/sleep/stage_ranges.dart b/lib/src/onehz/sleep/stage_ranges.dart new file mode 100644 index 00000000..38a5735f --- /dev/null +++ b/lib/src/onehz/sleep/stage_ranges.dart @@ -0,0 +1,180 @@ +// SLEEP — per-stage minute RANGES. +// +// The cardio stager gives each epoch one label. This file turns those labels +// into a [lo, hi] range of minutes per stage, so a caller can show how much of +// each stage figure the evidence actually pins down. +// +// WHAT THE RANGE IS. An honest BOUND, not a statistical confidence interval. +// The stager has no per-epoch posteriors: it compares two weighted robust-z +// scores against fixed cutoffs ([kDefaultRemScoreCut], [kDefaultDeepScoreCut]) +// and reports a night confidence from coverage. So the only per-epoch evidence +// is "which side of the cutoff, and how far". The range counts: +// lo[s] = epochs that can only be s: HR-covered, labelled s, and every score +// that decided the label at least `margin` from its cutoff; +// hi[s] = lo[s] + epochs that could still be s: uncovered or unlabelled +// epochs (any stage), and asleep epochs with a score inside the +// margin (REM score: REM and its NREM substage; deep score: light +// and deep). A score that is missing (null or NaN) is inside the +// margin by definition: it did not decide, so it cannot be clear. +// Owner decision (2026-10-19): a fully covered night has lo == hi only where +// every score is clear of its cutoff. Nothing here moves the stager's label. + +import 'accounting.dart' show SleepStage; +import 'cardio_stager.dart' + show CardioStagerResult, kDefaultDeepScoreCut, kDefaultRemScoreCut; + +// Internal rule, ESTIMATE with low confidence, derived from cardioStager() score +// cutoffs. Long 2016 / Fonseca 2015/2018 supports the underlying staging family, +// not the unpublished margin or bounds. This is not a confidence interval. +/// How far (in stage-score units) a score must sit from its cutoff before the +/// label it decided counts as clear. +/// +/// The scores are weighted means of robust z, and the shipped cutoffs were +/// chosen on an 11-night sweep with a 0.1 grid (see the table above +/// [kDefaultRemScoreCut]). One grid step moves median %REM by about 5 points +/// and %Deep by about 3, and the sweep cannot rank cutoffs finer than that. +/// So a score within 0.1 of its cutoff would flip under a neighbouring cutoff +/// the calibration could not tell apart from the shipped one. That is the only +/// precision we can justify; it is NOT a measured per-epoch error rate. +const double kStageScoreMargin = 0.1; + +/// One epoch's staging evidence, as [stageMinuteRanges] reads it. +class StageEpochEvidence { + /// The final label; null when the epoch has no label at all. + final SleepStage? stage; + + /// Deep flag (meaningful only when [stage] is NREM). + final bool deep; + + /// The epoch carried at least one valid HR sample. + final bool covered; + + /// The REM and deep scores that decided the label; null when not scored. + final double? remScore; + final double? deepScore; + + const StageEpochEvidence({ + this.stage, + this.deep = false, + this.covered = true, + this.remScore, + this.deepScore, + }); +} + +/// A [lo, hi] range of minutes. +class MinuteRange { + final double lo, hi; + const MinuteRange(this.lo, this.hi); + + Map toJson() => {'lo': lo, 'hi': hi}; +} + +/// Per-stage minute ranges over a staged window of [windowMin] minutes. +/// [nrem] is light ∪ deep. +class StageMinuteRanges { + final MinuteRange wake, light, deep, rem, nrem; + final double windowMin; + + const StageMinuteRanges({ + required this.wake, + required this.light, + required this.deep, + required this.rem, + required this.nrem, + required this.windowMin, + }); + + /// The four display stages as `{lo, hi}`. + Map> toJson() => { + 'wake': wake.toJson(), + 'light': light.toJson(), + 'deep': deep.toJson(), + 'rem': rem.toJson(), + }; +} + +extension CardioStagerEvidence on CardioStagerResult { + /// One entry per final epoch: the final label and deep flag, whether the + /// epoch had valid HR, and the scores the classify loop computed for it. + /// Empty when the stager abstained. + List get epochEvidence { + final stages = base.stages; + return [ + for (var e = 0; e < stages.length; e++) + StageEpochEvidence( + stage: stages[e], + deep: e < deepFlag.length && deepFlag[e], + // A result built without coverage cannot vouch for any epoch. + covered: e < hrCovered.length && hrCovered[e], + remScore: e < remScores.length ? remScores[e] : null, + deepScore: e < deepScores.length ? deepScores[e] : null, + ), + ]; + } +} + +const int _wake = 1, _light = 2, _deep = 4, _rem = 8; +const int _any = _wake | _light | _deep | _rem; +const int _nremSet = _light | _deep; + +/// Per-stage minute ranges for [epochs] ([epochSec] seconds each). Null when +/// there are no epochs. See the file header for the rule. +StageMinuteRanges? stageMinuteRanges( + List? epochs, { + int epochSec = 30, + double remScoreCut = kDefaultRemScoreCut, + double deepScoreCut = kDefaultDeepScoreCut, + double margin = kStageScoreMargin, +}) { + if (epochs == null || epochs.isEmpty) return null; + // Written as an open interval on `cut ± margin` so a score of exactly + // `cut + margin` compares equal to the bound and counts as clear. A missing + // score (null, or NaN) decided nothing: it is undecided, never "clear", so + // the decision it would have made stays open on both sides. + bool near(double? s, double cut) => + s == null || s.isNaN || (s > cut - margin && s < cut + margin); + + final lo = {_wake: 0, _light: 0, _deep: 0, _rem: 0, _nremSet: 0}; + final hi = {_wake: 0, _light: 0, _deep: 0, _rem: 0, _nremSet: 0}; + for (final ep in epochs) { + final int can; // bitset of stages this epoch could be + if (!ep.covered || ep.stage == null) { + can = _any; + } else if (ep.stage == SleepStage.wake) { + can = _wake; + } else { + final deepNear = near(ep.deepScore, deepScoreCut); + final int sub; + if (deepNear) { + sub = _nremSet; + } else if (ep.stage == SleepStage.nrem) { + sub = ep.deep ? _deep : _light; + } else { + // A REM epoch's NREM neighbour is the substage its deep score points to. + final ds = ep.deepScore; + sub = (ds != null && ds > deepScoreCut) ? _deep : _light; + } + final remNear = near(ep.remScore, remScoreCut); + can = remNear + ? (_rem | sub) + : (ep.stage == SleepStage.rem ? _rem : sub); + } + for (final s in const [_wake, _light, _deep, _rem]) { + if (can & s != 0) hi[s] = hi[s]! + 1; + if (can == s) lo[s] = lo[s]! + 1; + } + if (can & _nremSet != 0) hi[_nremSet] = hi[_nremSet]! + 1; + if (can & ~_nremSet == 0) lo[_nremSet] = lo[_nremSet]! + 1; + } + final m = epochSec / 60.0; + MinuteRange r(int s) => MinuteRange(lo[s]! * m, hi[s]! * m); + return StageMinuteRanges( + wake: r(_wake), + light: r(_light), + deep: r(_deep), + rem: r(_rem), + nrem: r(_nremSet), + windowMin: epochs.length * m, + ); +} diff --git a/test/aggregate/conversion_check_test.dart b/test/aggregate/conversion_check_test.dart new file mode 100644 index 00000000..4e7f3c52 --- /dev/null +++ b/test/aggregate/conversion_check_test.dart @@ -0,0 +1,260 @@ +// checkConversion: correct aggregators pass; each deliberately broken one is +// caught on the path that names its defect (and not on the paths it gets right). + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +typedef _B = ListBatch; + +int _sliceSum(_B b, int start, int end) { + var t = 0; + for (var i = start; i < end; i++) { + t += b.rows[i]; + } + return t; +} + +int _refSum(_B b) { + var t = 0; + for (final x in b.rows) { + t += x; + } + return t; +} + +/// A correct sum over int rows. +class _Sum extends Aggregator { + const _Sum(); + @override + int empty() => 0; + @override + int addSlice(int s, _B b, int start, int end) => s + _sliceSum(b, start, end); + @override + int combine(int left, int right) => left + right; + @override + int finish(int s) => s; +} + +/// combine drops the right side: wrong only when chunks were combined. +class _DropRight extends _Sum { + const _DropRight(); + @override + int combine(int left, int right) => left; +} + +/// addSlice restarts from the slice, so a second addSlice onto a state that +/// already holds data loses it. Only the 'sliced' path adds twice to one state. +class _IgnoresState extends _Sum { + const _IgnoresState(); + @override + int addSlice(int s, _B b, int start, int end) => _sliceSum(b, start, end); +} + +/// State: [sum, chunks]. combine is right only when the right side is a single +/// chunk (it drops a multi-chunk right side), so left-to-right combining works +/// and right-nested combining does not. +class _LeftOnly extends Aggregator, _B, int> { + const _LeftOnly(); + @override + List empty() => [0, 0]; + @override + List addSlice(List s, _B b, int start, int end) => + [s[0] + _sliceSum(b, start, end), s[1] + (s[1] == 0 ? 1 : 0)]; + @override + List combine(List l, List r) => + r[1] > 1 ? [l[0], l[1] + r[1]] : [l[0] + r[0], l[1] + r[1]]; + @override + int finish(List s) => s[0]; +} + +/// Keeps the FIRST batch it was shown plus index ranges into it, and reads the +/// rows only in finish (clamped, so a foreign batch cannot throw). Over one +/// batch this is a correct sum. Chunks cut out with copyRange are separate +/// batches with their own indexing, so the ranges of later chunks are read from +/// the first chunk's storage: wrong values. Only the 'copies' path differs. +class _Aliasing extends Aggregator<_AliasState, _B, int> { + const _Aliasing(); + @override + _AliasState empty() => _AliasState(null, []); + @override + _AliasState addSlice(_AliasState s, _B b, int start, int end) { + s.ranges.add([start, end]); + return _AliasState(s.source ?? b, s.ranges); + } + + @override + _AliasState combine(_AliasState l, _AliasState r) => + _AliasState(l.source ?? r.source, [...l.ranges, ...r.ranges]); + + @override + int finish(_AliasState s) { + final src = s.source; + if (src == null) return 0; + var t = 0; + for (final r in s.ranges) { + for (var i = r[0]; i < r[1] && i < src.length; i++) { + t += src.rows[i]; + } + } + return t; + } +} + +class _AliasState { + _AliasState(this.source, this.ranges); + final _B? source; + final List> ranges; +} + +/// Sum of (row - origin); origin comes from the case via build. +class _SumFrom extends Aggregator { + const _SumFrom(this.origin); + final int origin; + @override + int empty() => 0; + @override + int addSlice(int s, _B b, int start, int end) => + s + _sliceSum(b, start, end) - origin * (end - start); + @override + int combine(int left, int right) => left + right; + @override + int finish(int s) => s; +} + +int _originOf(_B b) => b.rows.isEmpty ? 0 : b.rows.first; + +Set _paths(List fs) => {for (final f in fs) f.path}; + +void main() { + final cases = randomIntBatches(8, count: 24, maxLen: 30); + + List check(Aggregator agg, + {int Function(_B)? reference, bool laws = true, int seed = 0}) => + checkConversion((_) => agg, cases, reference ?? _refSum, + laws: laws, seed: seed); + + group('correct aggregators pass', () { + test('Sum against a reference loop', () { + expectConversion((_) => const _Sum(), cases, _refSum); + expect(check(const _Sum()), isEmpty); + }); + + test('hypnogram aggregator against hypnogramMetrics', () { + final r = Rng(5); + const stages = ['wake', 'light', 'deep', 'rem']; + final batches = >[]; + for (var c = 0; c < 6; c++) { + final n = c == 0 ? 0 : r.intIn(1, 14); + var t = r.intIn(0, 60); + final segs = []; + for (var i = 0; i < n; i++) { + final d = r.intIn(1, 120); + segs.add(StageSegment(t, t + d, stages[r.intIn(0, 3)])); + t += d + r.intIn(0, 20); + } + batches.add(ListBatch(segs)); + } + expectConversion, HypnogramMetrics>( + (_) => HypnogramAggregator(0, 3000), + batches, + (b) => AdvancedSleepStager.hypnogramMetrics(SleepSession( + start: 0, + end: 3000, + efficiency: 0, + stages: b.rows, + restingHr: null, + avgHrv: null)), + equals: fieldsEqual((m) => [ + m.tibS, m.tstS, m.sptS, m.solS, m.remLatencyS, m.wasoS, + m.efficiency, m.disturbances, m.deepMin, m.remMin, m.lightMin, + m.deepPct, m.remPct, m.lightPct, + ]), + ); + }); + }); + + group('broken aggregators are caught on the right path', () { + test('combine drops right: chunked and law:split, not whole', () { + final p = _paths(check(const _DropRight())); + expect(p, containsAll(['chunked', 'law:split'])); + expect(p, isNot(contains('whole'))); + }); + + test('state aliasing the input: copies fails, whole passes', () { + final p = _paths(check(const _Aliasing())); + expect(p, contains('copies')); + expect(p, isNot(contains('whole'))); + expect(p, isNot(contains('chunked'))); + expect(p, isNot(contains('sliced'))); + }); + + test('addSlice ignoring its state: sliced and law:slicing', () { + final p = _paths(check(const _IgnoresState())); + expect(p, containsAll(['sliced', 'law:slicing'])); + expect(p, isNot(contains('whole'))); + expect(p, isNot(contains('chunked'))); + }); + + test('right-nested combine broken: chunkedRight, not chunked', () { + final p = _paths(check(const _LeftOnly(), laws: false)); + expect(p, contains('chunkedRight')); + expect(p, isNot(contains('whole'))); + expect(p, isNot(contains('chunked'))); + }); + + test('wrong reference everywhere: whole fails on every case', () { + final fs = check(const _Sum(), reference: (b) => _refSum(b) + 1); + final whole = fs.where((f) => f.path == 'whole'); + expect({for (final f in whole) f.caseIndex}, + {for (var i = 0; i < cases.length; i++) i}); + }); + + test('failures carry a detail string', () { + final fs = check(const _DropRight()); + expect(fs, isNotEmpty); + expect(fs.every((f) => f.detail.isNotEmpty), isTrue); + }); + }); + + group('build is context-indexed', () { + test('per-case parameter comes from build(input)', () { + final seen = <_B>[]; + expectConversion((input) { + seen.add(input); + return _SumFrom(_originOf(input)); + }, cases, (b) => _refSum(b) - _originOf(b) * b.length); + expect(seen, isNotEmpty); + expect(seen.toSet(), containsAll(cases)); + }); + + test('a build that ignores the case is caught', () { + final fs = checkConversion((_) => const _SumFrom(0), cases, + (b) => _refSum(b) - _originOf(b) * b.length); + expect(_paths(fs), contains('whole')); + }); + }); + + group('determinism and options', () { + test('same seed gives identical failures', () { + String show(List fs) => + fs.map((f) => '${f.path}|${f.caseIndex}|${f.detail}').join('\n'); + expect(show(check(const _Aliasing(), seed: 3)), + show(check(const _Aliasing(), seed: 3))); + expect(show(check(const _DropRight(), seed: 3)), + show(check(const _DropRight(), seed: 3))); + }); + + test('laws: false skips law paths', () { + expect(_paths(check(const _DropRight())).any((p) => p.startsWith('law:')), + isTrue); + expect( + _paths(check(const _DropRight(), laws: false)) + .any((p) => p.startsWith('law:')), + isFalse); + }); + }); +} diff --git a/test/aggregate/hypnogram_aggregator_test.dart b/test/aggregate/hypnogram_aggregator_test.dart new file mode 100644 index 00000000..ac99cfb9 --- /dev/null +++ b/test/aggregate/hypnogram_aggregator_test.dart @@ -0,0 +1,461 @@ +// HypnogramAggregator parity: folding the sorted segments through the +// aggregator must reproduce AdvancedSleepStager.hypnogramMetrics exactly, for +// any input and any chunking. +// +// _referenceHypnogram is a FROZEN copy of the pre-aggregator implementation. +// Do not edit it when hypnogramMetrics is rewired to use the aggregator: the +// public function is checked against it so the rewire cannot drift. + +import 'dart:math' as math; + +import 'package:openstrap_analytics/aggregate.dart'; +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/checkpoint_support.dart'; +import '../support/aggregator_laws.dart'; +import '../support/property.dart' show Rng; + +// ── frozen reference ──────────────────────────────────────────────────────── + +HypnogramMetrics _referenceHypnogram(SleepSession session) { + final segs = [...session.stages]..sort((a, b) => a.start.compareTo(b.start)); + final tib = math.max(0, session.end - session.start); + final sleepSegs = segs + .where((s) => s.stage == 'light' || s.stage == 'deep' || s.stage == 'rem') + .toList(); + var tst = 0, deepS = 0, remS = 0, lightS = 0; + for (final s in sleepSegs) { + final d = s.end - s.start; + tst += d; + if (s.stage == 'deep') deepS += d; + if (s.stage == 'rem') remS += d; + if (s.stage == 'light') lightS += d; + } + int onset, sptEnd; + int? sol; + if (sleepSegs.isNotEmpty) { + onset = sleepSegs.first.start; + sptEnd = sleepSegs.last.end; + sol = math.max(0, onset - session.start); + } else { + onset = session.end; + sptEnd = session.end; + sol = null; + } + final remSegs = sleepSegs.where((s) => s.stage == 'rem').toList(); + final remLatency = + remSegs.isNotEmpty ? (remSegs.first.start - onset).toDouble() : double.nan; + var waso = 0, disturbances = 0; + for (final s in segs.where((s) => s.stage == 'wake')) { + final w0 = math.max(s.start, onset); + final w1 = math.min(s.end, sptEnd); + if (w1 > w0) { + waso += w1 - w0; + disturbances++; + } + } + final se = tib > 0 ? tst / tib : 0.0; + double pct(int x) => tst > 0 ? x / tst * 100 : 0; + return HypnogramMetrics( + tibS: tib, + tstS: tst, + sptS: math.max(0, sptEnd - onset), + solS: sol, + remLatencyS: remLatency, + wasoS: waso, + efficiency: math.min(1.0, se), + disturbances: disturbances, + deepMin: deepS / 60, + remMin: remS / 60, + lightMin: lightS / 60, + deepPct: pct(deepS), + remPct: pct(remS), + lightPct: pct(lightS), + ); +} + +// ── helpers ───────────────────────────────────────────────────────────────── + +SleepSession _session(int start, int end, List stages) => + SleepSession( + start: start, + end: end, + efficiency: 0, + stages: stages, + restingHr: null, + avgHrv: null); + +/// Stable sort by start. With at most 30 segments, the sort inside the +/// reference/public function (insertion sort below 32 elements) is stable too, +/// so re-sorting this list leaves it unchanged and ties stay in input order. +List _sorted(List xs) { + final idx = [for (var i = 0; i < xs.length; i++) i]; + idx.sort((a, b) { + final c = xs[a].start.compareTo(xs[b].start); + return c != 0 ? c : a.compareTo(b); + }); + return [for (final i in idx) xs[i]]; +} + +String _show(HypnogramMetrics m) => m.toJson().toString() + + ' remLatencyS=${m.remLatencyS}'; + +/// Field by field, exactly: ints ==, doubles bit-identical (NaN == NaN). +void _expectSame(HypnogramMetrics got, HypnogramMetrics want, String tag) { + void i(String n, int? a, int? b) => expect(a, b, reason: '$tag: $n'); + void d(String n, double a, double b) => + expect(a.toString(), b.toString(), reason: '$tag: $n ($a vs $b)'); + i('tibS', got.tibS, want.tibS); + i('tstS', got.tstS, want.tstS); + i('sptS', got.sptS, want.sptS); + i('solS', got.solS, want.solS); + d('remLatencyS', got.remLatencyS, want.remLatencyS); + i('wasoS', got.wasoS, want.wasoS); + d('efficiency', got.efficiency, want.efficiency); + i('disturbances', got.disturbances, want.disturbances); + d('deepMin', got.deepMin, want.deepMin); + d('remMin', got.remMin, want.remMin); + d('lightMin', got.lightMin, want.lightMin); + d('deepPct', got.deepPct, want.deepPct); + d('remPct', got.remPct, want.remPct); + d('lightPct', got.lightPct, want.lightPct); +} + +List _fields(HypnogramMetrics m) => [ + m.tibS, m.tstS, m.sptS, m.solS, m.remLatencyS, m.wasoS, m.efficiency, + m.disturbances, m.deepMin, m.remMin, m.lightMin, m.deepPct, m.remPct, + m.lightPct, + ]; + +typedef _Agg = Aggregator, HypnogramMetrics>; + +ListBatch _batch(List xs) => ListBatch(xs); + +/// Whole run, then every chunked / sliced evaluation of [bounds], against [want]. +void _expectAllPaths(_Agg agg, List sorted, List bounds, + HypnogramMetrics want, String tag) { + final b = _batch(sorted); + _expectSame(runChunked(agg, b, bounds), want, '$tag chunks $bounds'); + _expectSame(runChunked(agg, b, bounds, copy: true), want, + '$tag copied chunks $bounds'); + _expectSame(runChunkedRight(agg, b, bounds), want, '$tag right $bounds'); + _expectSame(runSliced(agg, b, bounds), want, '$tag sliced $bounds'); +} + +const _stages = [ + 'wake', 'light', 'deep', 'rem', 'wake', 'light', 'deep', 'rem', // + 'unknown', 'REM', '', 'Wake', +]; + +/// A messy session: random bounds (sometimes end < start), up to [maxSegs] +/// segments placed anywhere (before, inside, after the session), random +/// durations (sometimes zero or negative), overlaps, unknown stage strings. +({int start, int end, List segs}) _messy(Rng r, int maxSegs) { + final start = r.intIn(-500, 1500); + final len = r.nextBool(.05) ? r.intIn(-50, 0) : r.intIn(0, 3000); + final end = start + len; + final n = r.intIn(0, maxSegs); + final segs = []; + for (var k = 0; k < n; k++) { + final s = r.intIn(start - 300, end + 300); + final d = r.nextBool(.05) ? r.intIn(-30, 0) : r.intIn(0, 600); + segs.add(StageSegment(s, s + d, _stages[r.nextInt(_stages.length)])); + } + return (start: start, end: end, segs: segs); +} + +/// A clean contiguous night starting at [start]: segments back to back. +({int start, int end, List segs}) _tidy(Rng r) { + final start = r.intIn(0, 1000); + var t = start + r.intIn(0, 120); + final segs = []; + final n = r.intIn(1, 28); + const names = ['wake', 'light', 'deep', 'rem', 'light', 'rem', 'wake']; + for (var k = 0; k < n; k++) { + final d = r.intIn(30, 900); + segs.add(StageSegment(t, t + d, names[r.nextInt(names.length)])); + t += d; + } + return (start: start, end: t + r.intIn(0, 200), segs: segs); +} + +void main() { + // Named scenarios, each with the sorted segments. + final scenarios = segs})>{ + 'empty stages': (start: 0, end: 1000, segs: []), + 'empty stages, zero-length session': (start: 50, end: 50, segs: []), + 'inverted session bounds': ( + start: 100, + end: 40, + segs: [StageSegment(100, 200, 'light')] + ), + 'only wake': ( + start: 0, + end: 600, + segs: [StageSegment(0, 300, 'wake'), StageSegment(300, 600, 'wake')] + ), + 'no sleep, only unknown stages': ( + start: 0, + end: 600, + segs: [StageSegment(0, 300, 'unknown'), StageSegment(300, 600, 'asleep')] + ), + 'wake before onset': ( + start: 0, + end: 1000, + segs: [ + StageSegment(0, 200, 'wake'), + StageSegment(200, 500, 'light'), + StageSegment(500, 900, 'rem'), + ] + ), + 'wake after last sleep': ( + start: 0, + end: 1000, + segs: [ + StageSegment(0, 400, 'light'), + StageSegment(400, 600, 'deep'), + StageSegment(600, 1000, 'wake'), + ] + ), + 'wake straddling onset and spt end': ( + start: 0, + end: 1000, + segs: [ + StageSegment(0, 300, 'wake'), + StageSegment(100, 250, 'light'), + StageSegment(250, 500, 'wake'), + StageSegment(500, 700, 'rem'), + StageSegment(650, 900, 'wake'), + ] + ), + 'wake in the middle of the night': ( + start: 0, + end: 3000, + segs: [ + StageSegment(0, 600, 'light'), + StageSegment(600, 700, 'wake'), + StageSegment(700, 1500, 'deep'), + StageSegment(1500, 1560, 'wake'), + StageSegment(1560, 2400, 'rem'), + StageSegment(2400, 3000, 'wake'), + ] + ), + 'rem only': ( + start: 10, + end: 700, + segs: [StageSegment(100, 400, 'rem')] + ), + 'single light segment': ( + start: 0, + end: 3600, + segs: [StageSegment(0, 3600, 'light')] + ), + 'tst exceeds time in bed (overlap)': ( + start: 0, + end: 100, + segs: [StageSegment(0, 100, 'light'), StageSegment(0, 100, 'deep')] + ), + 'overlapping sleep segments': ( + start: 0, + end: 1000, + segs: [ + StageSegment(0, 500, 'light'), + StageSegment(200, 800, 'rem'), + StageSegment(300, 400, 'wake'), + ] + ), + 'zero and negative length segments': ( + start: 0, + end: 500, + segs: [ + StageSegment(10, 10, 'light'), + StageSegment(20, 5, 'rem'), + StageSegment(30, 30, 'wake'), + StageSegment(40, 400, 'deep'), + ] + ), + 'segments entirely outside the session': ( + start: 1000, + end: 2000, + segs: [StageSegment(0, 100, 'light'), StageSegment(3000, 3100, 'rem')] + ), + 'tied starts keep input order': ( + start: 0, + end: 500, + segs: [ + StageSegment(10, 100, 'wake'), + StageSegment(10, 200, 'light'), + StageSegment(10, 50, 'rem'), + StageSegment(10, 300, 'wake'), + ] + ), + 'case-sensitive stage names': ( + start: 0, + end: 500, + segs: [StageSegment(0, 100, 'Light'), StageSegment(100, 200, 'REM')] + ), + }; + + List> allBounds(int n) => [ + [0, n], + for (var k = 0; k <= n; k++) [0, k, n], + [0, 0, 0, n, n], + if (n >= 3) [0, 1, 2, n], + if (n >= 3) [0, 1, 1, n - 1, n], + ]; + + group('named scenarios', () { + scenarios.forEach((name, sc) { + test('$name: aggregator == frozen reference == public function', () { + final sorted = _sorted(sc.segs); + final session = _session(sc.start, sc.end, sc.segs); + final want = _referenceHypnogram(session); + final _Agg agg = HypnogramAggregator(sc.start, sc.end); + _expectSame(agg.runList(sorted), want, 'run vs reference'); + _expectSame(AdvancedSleepStager.hypnogramMetrics(session), want, + 'public vs reference'); + _expectSame(agg.runList(sorted), + AdvancedSleepStager.hypnogramMetrics(session), 'run vs public'); + for (final b in allBounds(sorted.length)) { + _expectAllPaths(agg, sorted, b, want, name); + } + }); + }); + + test('no sleep: solS is null and remLatency is NaN', () { + final m = HypnogramAggregator(0, 600) + .runList([const StageSegment(0, 600, 'wake')]); + expect(m.solS, isNull); + expect(m.remLatencyS.isNaN, isTrue); + expect(m.tstS, 0); + expect(m.sptS, 0); + expect(m.wasoS, 0); + expect(m.efficiency, 0.0); + }); + + test('a known night, worked by hand', () { + // 0-100 wake, 100-400 light, 400-500 wake, 500-700 rem, 700-800 wake. + final m = HypnogramAggregator(0, 800).runList(const [ + StageSegment(0, 100, 'wake'), + StageSegment(100, 400, 'light'), + StageSegment(400, 500, 'wake'), + StageSegment(500, 700, 'rem'), + StageSegment(700, 800, 'wake'), + ]); + expect(m.tibS, 800); + expect(m.tstS, 500); + expect(m.sptS, 600); + expect(m.solS, 100); + expect(m.remLatencyS, 400.0); + expect(m.wasoS, 100); + expect(m.disturbances, 1); + expect(m.efficiency, 500 / 800); + expect(m.lightMin, 300 / 60); + expect(m.remMin, 200 / 60); + expect(m.deepMin, 0.0); + expect(m.remPct, 200 / 500 * 100); + }); + }); + + group('random parity (seeded)', () { + test('messy input: unchunked, both references', () { + final r = Rng(0xA11CE); + for (var t = 0; t < 120; t++) { + final g = _messy(r, 30); + final session = _session(g.start, g.end, g.segs); + final want = _referenceHypnogram(session); + final got = HypnogramAggregator(g.start, g.end).runList(_sorted(g.segs)); + _expectSame(got, want, 'case $t ${_show(want)}'); + _expectSame(AdvancedSleepStager.hypnogramMetrics(session), want, + 'public, case $t'); + } + }); + + test('tidy nights: unchunked, both references', () { + final r = Rng(0xB0B); + for (var t = 0; t < 60; t++) { + final g = _tidy(r); + final session = _session(g.start, g.end, g.segs); + final want = _referenceHypnogram(session); + _expectSame(HypnogramAggregator(g.start, g.end).runList(_sorted(g.segs)), + want, 'case $t'); + _expectSame(AdvancedSleepStager.hypnogramMetrics(session), want, + 'public, case $t'); + } + }); + + test('random chunkings and slicings, every evaluation path', () { + final r = Rng(0xC0DE); + for (var t = 0; t < 150; t++) { + final g = t.isEven ? _messy(r, 30) : _tidy(r); + final sorted = _sorted(g.segs); + final want = _referenceHypnogram(_session(g.start, g.end, g.segs)); + final _Agg agg = HypnogramAggregator(g.start, g.end); + for (var c = 0; c < 4; c++) { + _expectAllPaths(agg, sorted, randomBounds(r, sorted.length), want, + 'case $t'); + } + } + }); + + test('combine order: right-nested equals left-nested', () { + final r = Rng(0xD00D); + for (var t = 0; t < 40; t++) { + final g = _messy(r, 30); + final sorted = _sorted(g.segs); + final want = _referenceHypnogram(_session(g.start, g.end, g.segs)); + final _Agg agg = HypnogramAggregator(g.start, g.end); + final b = randomBounds(r, sorted.length); + _expectSame(runChunkedRight(agg, _batch(sorted), b), want, + 'case $t right-nested $b'); + _expectSame(runChunked(agg, _batch(sorted), b), want, + 'case $t left-nested $b'); + } + }); + }); + + test('persistable: a mid-stream checkpoint resumes to the one-pass result', () { + final g = _tidy(Rng(0xC0DE)); + expectCheckpointResumes( + HypnogramAggregator(g.start, g.end), _batch(_sorted(g.segs)), + equals: fieldsEqual(_fields)); + }); + + group('laws', () { + test('lawful on messy and tidy sorted inputs (exact equality)', () { + final r = Rng(0xFACE); + // The session bounds are an instance property, so one instance per + // session; the laws hold per instance. + for (var t = 0; t < 12; t++) { + final g = t.isEven ? _messy(r, 30) : _tidy(r); + final sorted = _sorted(g.segs); + final cases = >[ + _batch([]), + if (sorted.isNotEmpty) _batch([sorted.first]), + _batch(sorted), + _batch(sorted.sublist(0, sorted.length ~/ 2)), + _batch(sorted.sublist(sorted.length ~/ 2)), + ]; + expectLawful(HypnogramAggregator(g.start, g.end), cases, + equals: fieldsEqual(_fields)); + } + }); + + test('finish does not mutate: finishing twice and then adding more', () { + final _Agg agg = HypnogramAggregator(0, 1000); + const segs = [ + StageSegment(0, 100, 'wake'), + StageSegment(100, 300, 'light'), + StageSegment(300, 400, 'wake'), + StageSegment(400, 600, 'rem'), + ]; + final batch = _batch(segs); + var s = agg.fold(batch, 0, 3); + final a = agg.finish(s), b = agg.finish(s); + _expectSame(a, b, 'twice'); + s = agg.addSlice(s, batch, 3, 4); + _expectSame(agg.finish(s), agg.runList(segs), 'continued'); + }); + }); +} diff --git a/test/onehz/advanced_stager_test.dart b/test/onehz/advanced_stager_test.dart index 1a229aa3..08d39e6d 100644 --- a/test/onehz/advanced_stager_test.dart +++ b/test/onehz/advanced_stager_test.dart @@ -9,6 +9,8 @@ import 'dart:math' as math; import 'package:test/test.dart'; import 'package:openstrap_analytics/onehz.dart'; +import 'support/pre_bump_stager.dart'; + void main() { // Build a synthetic capture at 1 Hz: [dayH] active day, [nightH] still night // with low sleeping HR + physiological RR, then [tailH] active. @@ -315,6 +317,198 @@ void main() { }); }); + group('grid option 2: staging anchors its epochs at the session start', () { + // Owner decision 2026-10-23: the gap-anchored grid leaves the bump. The + // default stager's labels equal the pre-bump stager's (frozen as an oracle + // in support/pre_bump_stager.dart, logic of 83fc308), whatever run the + // session starts in. + test('detected sessions: labels equal the pre-bump stager exactly', () { + final d = build(2, 7, 1); + final sessions = AdvancedSleepStager.detectSleep(d.grav, d.hr, rr: d.rr); + expect(sessions, isNotEmpty); + for (final s in sessions) { + final want = preBumpStageSessionCardio(s.start, s.end, d.grav, d.hr, d.rr); + expect(showSegments(s.stages), showSegments(want), + reason: 'session ${s.start}..${s.end}'); + } + }); + + test('a session that starts mid-run: every boundary is on ITS grid ' + '(start + k * 30 s), not the run\'s', () { + final d = build(2, 7, 1); + for (final (from, to) in [(777, 9000), (1234, 20000), (4321, 25000)]) { + final start = d.nightStart + from, end = d.nightStart + to; + final got = AdvancedSleepStager.stageWindow(start, end, d.grav, d.hr, + rr: d.rr); + var checked = 0; + for (var i = 1; i < got.stages.length; i++) { + final g = got.stages[i]; + if (g.stage == 'unobserved' || got.stages[i - 1].stage == 'unobserved') { + continue; + } + expect((g.start - start) % 30, 0, + reason: 'window +$from..+$to, boundary ${g.start}'); + checked++; + } + expect(checked, greaterThan(5), reason: 'window +$from..+$to'); + } + }); + + test('a session that starts mid-run: labels equal the pre-bump stager ' + 'exactly (epochs from the start, features clipped at the session)', () { + // Offsets of 0 and 30 s share the run's phase but not its context: the + // pre-bump stager reads nothing outside [start, end). + final d = build(2, 7, 1); + for (final (from, to) in [(30, 9000), (777, 9000), (1234, 20000), (4321, 25000)]) { + final start = d.nightStart + from, end = d.nightStart + to; + final got = AdvancedSleepStager.stageWindow(start, end, d.grav, d.hr, + rr: d.rr); + final want = preBumpStageSessionCardio(start, end, d.grav, d.hr, d.rr); + expect(showSegments(got.stages), showSegments(want), + reason: 'window +$from..+$to'); + } + }); + + test('after a dropout longer than the carry the next sample re-anchors ' + '(pre-bump); before it the grid is the session start\'s', () { + // Two still blocks, 110 s apart (> the 61 s usable-run gap; the second + // starts at 8110, 8110 % 30 = 10). + final g = stillBlocks([(5000, 3000), (8110, 3000)]); + final rr = [ + for (var t = 5000; t < 11110; t++) + if (t < 8000 || t >= 8110) RrTs(t, 1150.0 + (t.isEven ? 20 : -20)) + ]; + for (final start in [5000, 5017, 5400]) { + final got = AdvancedSleepStager.stageWindow(start, 11110, g.grav, g.hr, + rr: rr); + var before = 0, after = 0; + for (var i = 1; i < got.stages.length; i++) { + final a = got.stages[i - 1], b = got.stages[i]; + if (a.stage == 'unobserved' || b.stage == 'unobserved') continue; + if (b.start < 8000) { + expect((b.start - start) % 30, 0, reason: 'start $start, run 1: ${b.start}'); + before++; + } else { + expect((b.start - 8110) % 30, 0, reason: 'start $start, run 2: ${b.start}'); + after++; + } + } + expect(before, greaterThan(0), reason: 'start $start'); + expect(after, greaterThan(0), reason: 'start $start'); + } + }); + }); + + group('pre-bump parity edge cases', () { + void parity(int start, int end, List grav, List hr, + [List rr = const []]) { + final got = AdvancedSleepStager.stageWindow(start, end, grav, hr, rr: rr); + final want = preBumpStageSessionCardio(start, end, grav, hr, rr); + expect(showSegments(got.stages), showSegments(want)); + } + + List flatGrav(int from, int to) => + [for (var t = from; t < to; t++) GravTs(t, 0, 0, 1)]; + + test('sparse HR (0, 30, 60 s) over [0,90): light, as before', () { + parity(0, 90, flatGrav(0, 90), [for (final t in [0, 30, 60]) HrTs(t, 60)]); + }); + + test('sparse HR (seconds 0..44 only) over [0,119)', () { + parity(0, 119, flatGrav(0, 119), [for (var t = 0; t < 45; t++) HrTs(t, 60)]); + }); + + test('gravity context across an accel gap [500,580)', () { + const bump = { + 19: 1.03, 20: 1.02, 22: 1.02, 23: 1.02, 27: 1.03, 29: 1.02, + 30: 1.02, 31: 1.01, 32: 1.02, 36: 1.02, 38: 1.03, 39: 1.03, + }; + final grav = [ + for (var t = 0; t < 1200; t++) + if (t < 500 || t >= 580) GravTs(t, 0, 0, bump[t ~/ 30] ?? 1.0) + ]; + parity(0, 1200, grav, [for (var t = 0; t < 1200; t++) HrTs(t, 60)]); + }); + + for (final (name, bad) in [('NaN', double.nan), ('infinite', double.infinity)]) { + test('non-finite RR ($name) does not throw; labels equal the oracle', () { + parity(0, 90, flatGrav(0, 90), [for (var t = 0; t < 90; t++) HrTs(t, 60)], + [RrTs(10, bad)]); + }); + } + + test('duplicate HR timestamps (60 then 0): last value wins as before', () { + parity(0, 90, flatGrav(0, 90), [ + for (var t = 0; t < 90; t++) ...[HrTs(t, 60), HrTs(t, 0)] + ]); + }); + + test('a pre-81b9b4b (EPOCH2) cache entry is never served', () { + // Keys and values as the old extractor stored them for gravity (0,0,1) + // each second and HR 60 then 0 at each second over [0,90): HR 60 where + // the oracle's last-wins reading has none (0-90:unobserved). + Map cacheJson(int version) => { + 'version': version, + 'rows': [ + for (final (s, fp) in [ + (0, '-518569928497453641'), + (30, '6424718457559164343'), + (60, '6000305869726000567'), + ]) + { + 'startSec': s, + 'epochSec': 30, + 'fingerprint': fp, + 'motion': 0.0, + 'hr': 60.0, + 'hrSd': 0.0, + 'rmssd': 'NaN', + 'lfhf': 'NaN', + 'rk': 'NaN', + 'sdnn': 'NaN', + } + ], + }; + // A cache written under the old identity (version 2) is refused on load. + expect(() => CardioRowCache.fromJson(cacheJson(2)), throwsFormatException); + // Even if the same keys sit under the current version they are old + // fingerprints: nothing matches the byte-wise ones, so nothing is served. + final stale = CardioRowCache.fromJson(cacheJson(3)); + final grav = flatGrav(0, 90); + final hr = [for (var t = 0; t < 90; t++) ...[HrTs(t, 60), HrTs(t, 0)]]; + final got = AdvancedSleepStager.stageWindow(0, 90, grav, hr, + rowCache: stale); + expect(stale.hits, 0, reason: 'old-identity entries must not hit'); + expect(showSegments(got.stages), + showSegments(preBumpStageSessionCardio(0, 90, grav, hr, const []))); + }); + + test('gravity in reverse time order: labels equal the oracle', () { + parity(0, 90, flatGrav(0, 90).reversed.toList(), + [for (var t = 0; t < 90; t++) HrTs(t, 60)]); + }); + + test('HR in reverse time order: labels equal the oracle', () { + parity(0, 90, flatGrav(0, 90), + [for (var t = 89; t >= 0; t--) HrTs(t, 60)]); + }); + + test('unsorted HR with duplicate seconds keeps list-order last-wins', () { + // Seconds descending, but within a second the records stay (0, 60): the + // oracle keeps 60 (the later record), so the night is stageable. + parity(0, 90, flatGrav(0, 90), [ + for (var t = 89; t >= 0; t--) ...[HrTs(t, 0), HrTs(t, 60)] + ]); + }); + + test('RR in reverse time order does not throw (Sol r3)', () { + // [0,30): the oracle abstains even with assertions on (longer windows + // assert sorted RR there). + parity(0, 30, flatGrav(0, 30), [for (var t = 0; t < 30; t++) HrTs(t, 60)], + [RrTs(1, 900), RrTs(0, 900)]); + }); + }); + group('AASM hypnogram metrics (hand-checked)', () { test('TIB/TST/SOL/WASO/REM-latency from a crafted hypnogram', () { // 1000-s session: 0-100 wake (latency), 100-400 light, 400-600 deep, diff --git a/test/onehz/cardio_epoch_features_test.dart b/test/onehz/cardio_epoch_features_test.dart new file mode 100644 index 00000000..36514c5f --- /dev/null +++ b/test/onehz/cardio_epoch_features_test.dart @@ -0,0 +1,260 @@ +// cardioEpochFeatures (round 15, phase 0) - RED until the implementation lands. +// +// ASSUMED API (derived from reading cardioStager; the implementer may change +// the CALL SHAPE below, never the assertions): +// +// CardioEpochFeatures cardioEpochFeatures( +// List hr1hz, +// List accel, { +// List rrMs = const [], +// List rrTsMs = const [], +// int epochSec = 30, +// int startEpoch = 0, // first epoch row to compute (inclusive) +// int? endEpoch, // one past the last row; null = through nEpoch +// }); +// +// Same inputs as `cardioStager` (so the stager is literally +// `classifyCardioEpochs(cardioEpochFeatures(...), ...)`). The whole-night inputs +// stay whole: the epoch grid (cadence, perEpoch, nEpoch = n ~/ perEpoch) is +// derived from them exactly as today, and [startEpoch, endEpoch) only selects +// which rows are computed, clamped to [0, nEpoch]. Row k of a sub-range call is +// epoch startEpoch + k of the full call. When the whole input cannot be staged +// (cadence abstain, nEpoch < 3) the result has zero-length lists for any range. +// +// Pinned by this file: +// 1. cardioStager(...) == classifyCardioEpochs(cardioEpochFeatures(...)) +// on every result field, bit for bit. +// 2. rows for random sub-ranges [a, b) == rows a..b-1 of the full computation, +// all seven feature lists, bit for bit (NaN-ness included). + +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +bool sameD(double a, double b) { + if (a.isNaN || b.isNaN) return a.isNaN && b.isNaN; + return a == b && a.isNegative == b.isNegative; +} + +Map> cols(CardioEpochFeatures f) => { + 'motion': f.motion, + 'hr': f.hr, + 'hrSd': f.hrSd, + 'rmssd': f.rmssd, + 'lfhf': f.lfhf, + 'rk': f.rk, + 'sdnn': f.sdnn, + }; + +void expectRowsEqual( + CardioEpochFeatures sub, CardioEpochFeatures full, int a, int b, String ctx) { + final s = cols(sub), f = cols(full); + for (final k in s.keys) { + expect(s[k]!.length, b - a, reason: '$ctx $k length for [$a,$b)'); + for (var i = 0; i < b - a; i++) { + if (!sameD(s[k]![i], f[k]![a + i])) { + fail('$ctx $k row ${a + i} of [$a,$b): ${s[k]![i]} vs ${f[k]![a + i]}'); + } + } + } +} + +void expectSameStager(CardioStagerResult a, CardioStagerResult b) { + expect(a.base.stages, b.base.stages); + expect(a.base.epochSec, b.base.epochSec); + expect(sameD(a.base.wakePct, b.base.wakePct), isTrue); + expect(sameD(a.base.nremPct, b.base.nremPct), isTrue); + expect(sameD(a.base.remPct, b.base.remPct), isTrue); + expect(a.deepFlag, b.deepFlag); + expect(sameD(a.confidence, b.confidence), isTrue); +} + +({List hr, List accel, List rr, List rrTs}) + genNight( + int seed, + int seconds, { + int cadenceSec = 1, + bool nanAccel = false, + bool hrDropouts = false, + bool rrOutliers = false, + bool withRr = true, +}) { + final r = math.Random(seed); + final n = seconds ~/ cadenceSec; + final hr = []; + final accel = []; + final rr = []; + final rrTs = []; + var posture = 1.0; + var moveLeft = 0; + var hrBase = 62.0; + for (var i = 0; i < n; i++) { + final ts = i * cadenceSec * 1000.0; + if (r.nextInt(900) == 0) posture = 0.95 + r.nextDouble() * 0.1; + if (moveLeft == 0 && r.nextInt(500) == 0) moveLeft = 5 + r.nextInt(40); + var m = posture + (r.nextDouble() - 0.5) * 0.002; + if (moveLeft > 0) { + m += r.nextDouble() * 0.3; + moveLeft--; + } + if (r.nextInt(2000) == 0) hrBase = 52 + r.nextDouble() * 25; + final cyc = 4 * math.sin(2 * math.pi * i * cadenceSec / 5400.0); + var h = hrBase + cyc + (r.nextDouble() - 0.5) * 4; + if (moveLeft > 0) h += 12; + if (hrDropouts && (i ~/ 200) % 7 == 3) h = 0; + if (hrDropouts && r.nextInt(300) == 0) h = double.nan; + hr.add(h); + var gx = m * 0.6; + final gy = m * 0.64, gz = m * 0.48; + if (nanAccel && r.nextInt(400) == 0) gx = double.nan; + accel.add(AccelSample(ts, gx, gy, gz)); + if (withRr) { + final nb = h.isNaN || h <= 0 ? 0 : (h * cadenceSec / 60).round(); + for (var b = 0; b < nb; b++) { + var v = 60000.0 / h + + (r.nextDouble() - 0.5) * 60 + + 40 * math.sin(2 * math.pi * i * cadenceSec / 25.0); + if (rrOutliers && r.nextInt(60) == 0) { + v = r.nextBool() ? 150 : 2600; + } else if (rrOutliers && r.nextInt(80) == 0) { + v += 350; + } + rr.add(v); + rrTs.add(ts); + } + } + } + return (hr: hr, accel: accel, rr: rr, rrTs: rrTs); +} + +void main() { + group('stager entry point == classify(features)', () { + void check(String name, int seed, int seconds, + {int cadenceSec = 1, + bool nanAccel = false, + bool hrDropouts = false, + bool rrOutliers = false, + bool withRr = true, + int epochSec = 30}) { + test(name, () { + final g = genNight(seed, seconds, + cadenceSec: cadenceSec, + nanAccel: nanAccel, + hrDropouts: hrDropouts, + rrOutliers: rrOutliers, + withRr: withRr); + final viaStager = cardioStager(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, epochSec: epochSec); + final feats = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, epochSec: epochSec); + final viaParts = classifyCardioEpochs(feats, epochSec: epochSec); + expectSameStager(viaStager, viaParts); + }); + } + + check('40 min with RR', 61, 2400); + check('50 min, outliers + dropouts', 62, 3000, + rrOutliers: true, hrDropouts: true); + check('40 min, NaN accel', 63, 2400, nanAccel: true); + check('40 min without RR', 64, 2400, withRr: false); + check('10 s cadence', 65, 5400, cadenceSec: 10); + check('epochSec 60', 66, 2400, epochSec: 60); + check('5 s cadence abstains', 67, 3600, cadenceSec: 5); + check('2 epochs abstains', 68, 70); + }); + + group('feature rows', () { + test('full call has one row per epoch in all seven lists', () { + final g = genNight(71, 2400); + final f = cardioEpochFeatures(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs); + expect(f.motion.length, 2400 ~/ 30); + for (final c in cols(f).values) { + expect(c.length, 80); + } + }); + + test('abstaining inputs give zero-length rows for any range', () { + final g = genNight(72, 70); + final f = cardioEpochFeatures(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs); + expect(f.length, 0); + final f2 = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, startEpoch: 0, endEpoch: 2); + expect(f2.length, 0); + }); + + void subRanges(String name, int seed, int seconds, + {int cadenceSec = 1, + bool nanAccel = false, + bool hrDropouts = false, + bool rrOutliers = false}) { + test(name, () { + final g = genNight(seed, seconds, + cadenceSec: cadenceSec, + nanAccel: nanAccel, + hrDropouts: hrDropouts, + rrOutliers: rrOutliers); + final full = + cardioEpochFeatures(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs); + final n = full.length; + expect(n, greaterThan(10)); + final r = math.Random(seed); + final ranges = >[ + [0, n], + [0, 1], + [n - 1, n], + [n, n], + [3, 3], + [0, 0], + [0, math.min(8, n)], + [math.max(0, n - 8), n], + for (var i = 0; i < 12; i++) + () { + final a = r.nextInt(n); + return [a, a + r.nextInt(n - a + 1)]; + }(), + ]; + for (final ab in ranges) { + final sub = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, + rrTsMs: g.rrTs, + startEpoch: ab[0], + endEpoch: ab[1]); + expectRowsEqual(sub, full, ab[0], ab[1], name); + } + }); + } + + subRanges('plain night', 81, 2700); + subRanges('RR outliers + HR dropouts', 82, 2700, + rrOutliers: true, hrDropouts: true); + subRanges('NaN accel magnitudes', 83, 2700, nanAccel: true); + subRanges('10 s cadence', 84, 5400, cadenceSec: 10); + + test('open-ended range and out-of-range bounds clamp', () { + final g = genNight(85, 2400); + final full = + cardioEpochFeatures(g.hr, g.accel, rrMs: g.rr, rrTsMs: g.rrTs); + final n = full.length; + final tail = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, startEpoch: 10); + expectRowsEqual(tail, full, 10, n, 'open-ended'); + final over = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, startEpoch: n - 5, endEpoch: n + 100); + expectRowsEqual(over, full, n - 5, n, 'clamped end'); + final past = cardioEpochFeatures(g.hr, g.accel, + rrMs: g.rr, rrTsMs: g.rrTs, startEpoch: n + 3, endEpoch: n + 9); + expect(past.length, 0); + }); + + test('no RR at all: rows are NaN for rmssd/sdnn/lfhf/rk, in any range', () { + final g = genNight(86, 2400, withRr: false); + final sub = cardioEpochFeatures(g.hr, g.accel, + startEpoch: 20, endEpoch: 30); + expect(sub.length, 10); + for (final c in [sub.rmssd, sub.sdnn, sub.lfhf, sub.rk]) { + expect(c.every((v) => v.isNaN), isTrue); + } + }); + }); +} diff --git a/test/onehz/cardio_row_cache_test.dart b/test/onehz/cardio_row_cache_test.dart new file mode 100644 index 00000000..9beb2bdb --- /dev/null +++ b/test/onehz/cardio_row_cache_test.dart @@ -0,0 +1,573 @@ +// Exact cardio epoch-row cache (round 16) - RED until the implementation lands. +// +// Spec: openstrap-analytics.research/r16-row-cache.md. The contract is that a +// `CardioRowCache` passed as `rowCache:` to segmentSleep changes NOTHING about +// the output (bit-identical to no cache, every field), only how many epoch rows +// are recomputed. +// +// DECISIONS (pinned here; the implementer may not change the assertions): +// * `CardioRowCache.fromJson` THROWS FormatException on malformed input and on +// a missing / non-matching `version` field. It never returns a cache holding +// guessed rows. (Callers that want "empty cache on failure" catch it.) +// * `hits` / `misses` may be cumulative over the cache's life; the tests only +// ever use deltas around one call. +// * `CardioRowKey` exposes `startSec` (absolute epoch start, the session +// clock) and `epochSec`. These two names are the only key members used here +// (retainWhere predicates). Everything else about the key is opaque. +// * Change detection is checked as: the changed sample forces at least one +// miss, and every epoch whose full context (own epoch +- 150 s, plus one +// epoch of slack for window edges) cannot contain it still hits. +// * "Most epochs hit when only the end moved" = hits >= 80 % of the epochs the +// first run staged that still lie inside the second session. +// +// Synthetic data: a few hours at 1 Hz for forced-window cases; ONE full night +// for the auto-detect case. + +import 'dart:convert'; +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +const int t0 = 1700000000; // absolute seconds, record start + +class Night { + final List accel; + final List hr; + final List rr; + final List rrTs; + Night(this.accel, this.hr, this.rr, this.rrTs); + + /// Samples with absolute second in [a, b). + Night slice(int a, int b) { + final ai = []; + final hi = []; + for (var i = 0; i < accel.length; i++) { + final s = accel[i].tsMs ~/ 1000; + if (s >= a && s < b) { + ai.add(accel[i]); + hi.add(hr[i]); + } + } + final r = [], rt = []; + for (var i = 0; i < rr.length; i++) { + final s = rrTs[i] ~/ 1000; + if (s >= a && s < b) { + r.add(rr[i]); + rt.add(rrTs[i]); + } + } + return Night(ai, hi, r, rt); + } + + /// Drop accel+hr positions and/or RR beats whose absolute second is in a + /// range (accel gaps are holes in the positional arrays, like real syncs). + Night withGaps({ + List<(int, int)> accelGaps = const [], + List<(int, int)> rrGaps = const [], + }) { + bool inAny(List<(int, int)> g, int s) => + g.any((r) => s >= r.$1 && s < r.$2); + final ai = []; + final hi = []; + for (var i = 0; i < accel.length; i++) { + if (inAny(accelGaps, accel[i].tsMs ~/ 1000)) continue; + ai.add(accel[i]); + hi.add(hr[i]); + } + final r = [], rt = []; + for (var i = 0; i < rr.length; i++) { + if (inAny(rrGaps, rrTs[i] ~/ 1000)) continue; + r.add(rr[i]); + rt.add(rrTs[i]); + } + return Night(ai, hi, r, rt); + } + + Night copy() => Night(List.of(accel), List.of(hr), List.of(rr), List.of(rrTs)); +} + +/// head minutes active, still minutes asleep, tail minutes active. +Night genNight(int seed, + {int headMin = 40, int stillMin = 150, int tailMin = 30, int start = t0}) { + final rnd = math.Random(seed); + final headS = headMin * 60, stillS = stillMin * 60, tailS = tailMin * 60; + final n = headS + stillS + tailS; + final accel = []; + final hr = []; + final rr = []; + final rrTs = []; + var hrBase = 50.0; + var moveLeft = 0; + var zBase = 1.0; + for (var i = 0; i < n; i++) { + final ts = (start + i) * 1000.0; + final active = i < headS || i >= headS + stillS; + double x, y = 0, z; + double h; + if (active) { + x = i.isEven ? 0.0 : 0.25; + z = 0.97; + h = 76 + (rnd.nextDouble() - 0.5) * 6; + } else { + if (rnd.nextInt(900) == 0) zBase = 0.97 + rnd.nextDouble() * 0.06; + if (moveLeft == 0 && rnd.nextInt(600) == 0) moveLeft = 5 + rnd.nextInt(30); + x = (rnd.nextDouble() - 0.5) * 0.002; + z = zBase + (rnd.nextDouble() - 0.5) * 0.002; + if (moveLeft > 0) { + x += rnd.nextDouble() * 0.3; + y += rnd.nextDouble() * 0.2; + moveLeft--; + } + if (rnd.nextInt(1500) == 0) hrBase = 47 + rnd.nextDouble() * 10; + h = hrBase + + 3 * math.sin(2 * math.pi * i / 5400.0) + + (rnd.nextDouble() - 0.5) * 3 + + (moveLeft > 0 ? 10 : 0); + } + accel.add(AccelSample(ts, x, y, z)); + hr.add(h); + final nb = (h / 60).round(); + for (var b = 0; b < nb; b++) { + rr.add(60000.0 / h + + (rnd.nextDouble() - 0.5) * 50 + + 35 * math.sin(2 * math.pi * i / 25.0)); + rrTs.add(ts); + } + } + return Night(accel, hr, rr, rrTs); +} + +SleepSegmentation stage(Night d, + {CardioRowCache? cache, int? onset, int? offset}) { + return segmentSleep( + d.accel, + d.hr, + rrMs: d.rr, + rrTsMs: d.rrTs, + tzOffsetSec: 0, + forcedWindow: onset == null || offset == null + ? null + : (onsetSec: onset, offsetSec: offset), + rowCache: cache, + ); +} + +String fingerprint(SleepSegmentation s) => jsonEncode({ + 'json': s.toJson(), + 'stages': s.stages.map((e) => e.toString()).toList(), + 'stages4': s.stages4, + 'conf': s.confidence.toString(), + 'eff': s.efficiencyPct.toString(), + }); + +/// Runs [stage] with [cache], returns (output, hitsDelta, missesDelta). +({SleepSegmentation out, int hits, int misses}) staged(Night d, + CardioRowCache cache, {int? onset, int? offset}) { + final h0 = cache.hits, m0 = cache.misses; + final out = stage(d, cache: cache, onset: onset, offset: offset); + return (out: out, hits: cache.hits - h0, misses: cache.misses - m0); +} + +CardioRowCache copyOf(CardioRowCache c) => + CardioRowCache.fromJson( + jsonDecode(jsonEncode(c.toJson())) as Map); + +void expectIdentical(SleepSegmentation a, SleepSegmentation b, String ctx) { + expect(fingerprint(a), fingerprint(b), reason: ctx); +} + +void main() { + // Shared 3.7 h record for the forced-window cases: 40 min head, 150 min + // still, 30 min tail. Window = the still part plus margins. + final base = genNight(7, headMin: 5, stillMin: 60, tailMin: 10); + final total = base.accel.length; + final winA0 = t0 + 600, winA1 = t0 + total - 600; + + test('sanity: the synthetic record stages without a cache', () { + final s = stage(base, onset: winA0, offset: winA1); + expect(s.present, isTrue); + expect(s.stages4.length, winA1 - winA0); + }); + + // why: a word-wise 64-bit mix lets two sign-bit flips cancel, so a row whose + // data changed in exactly two sign bits kept its key and a warm lookup served + // the old row. + // # catches: negating the x and y of one accel sample (two consecutive words + // of the key, magnitude and so the row unchanged) leaves every key identical + // under the word-wise mix; with byte-wise hashing the epochs that see the + // sample are looked up again. + test('two sign-bit changes in a row\'s data change its key', () { + final cache = CardioRowCache(); + staged(base, cache, onset: winA0, offset: winA1); + final i = base.accel.length ~/ 2; + final a = base.accel[i]; + expect(a.x != 0 || a.y != 0 || true, isTrue); + final changed = base.copy() + ..accel[i] = AccelSample(a.tsMs, -a.x, -a.y, a.z); + final warm = staged(changed, cache, onset: winA0, offset: winA1); + final ref = stage(changed, onset: winA0, offset: winA1); + expectIdentical(warm.out, ref, 'warm cache == no cache'); + // A 300 s key range holds about ten 30 s epochs; each that sees the sample + // is recomputed. + expect(warm.misses, greaterThanOrEqualTo(8)); + }); + + group('with cache == without cache', () { + test('forced window, cold cache, then warm cache', () { + final ref = stage(base, onset: winA0, offset: winA1); + final cache = CardioRowCache(); + final cold = staged(base, cache, onset: winA0, offset: winA1); + expectIdentical(cold.out, ref, 'cold'); + expect(cold.hits, 0); + expect(cold.misses, greaterThan(100)); + expect(cache.length, cold.misses); + final warm = staged(base, cache, onset: winA0, offset: winA1); + expectIdentical(warm.out, ref, 'warm'); + expect(warm.misses, 0); + expect(warm.hits, cold.misses); + }); + + test('RR gaps and accel gaps', () { + final d = base.withGaps( + rrGaps: [(t0 + 1500, t0 + 1900), (t0 + 2000, t0 + 2100)], + accelGaps: [(t0 + 2800, t0 + 2890), (t0 + 3300, t0 + 3310)], + ); + final ref = stage(d, onset: winA0, offset: winA1); + final cache = CardioRowCache(); + expectIdentical( + staged(d, cache, onset: winA0, offset: winA1).out, ref, 'cold'); + final warm = staged(d, cache, onset: winA0, offset: winA1); + expectIdentical(warm.out, ref, 'warm'); + expect(warm.misses, 0); + }); + + test('no RR at all', () { + final d = Night(base.accel, base.hr, const [], const []); + final ref = stage(d, onset: winA0, offset: winA1); + final cache = CardioRowCache(); + expectIdentical( + staged(d, cache, onset: winA0, offset: winA1).out, ref, 'cold'); + expectIdentical( + staged(d, cache, onset: winA0, offset: winA1).out, ref, 'warm'); + }); + + test('short sessions (under 3 epochs, a few epochs, ragged length)', () { + for (final len in [60, 100, 301, 905]) { + final a = t0 + 1800, b = a + len; + final ref = stage(base, onset: a, offset: b); + final cache = CardioRowCache(); + expectIdentical( + staged(base, cache, onset: a, offset: b).out, ref, 'cold $len'); + expectIdentical( + staged(base, cache, onset: a, offset: b).out, ref, 'warm $len'); + } + }); + + test('a cache full of rows from a DIFFERENT night never leaks in', () { + final other = genNight(99); + final cache = CardioRowCache(); + staged(other, cache, onset: winA0, offset: winA1); + expect(cache.length, greaterThan(100)); + final ref = stage(base, onset: winA0, offset: winA1); + final r = staged(base, cache, onset: winA0, offset: winA1); + expectIdentical(r.out, ref, 'same clock, other data'); + // Same timestamps, different values: essentially nothing may hit. + expect(r.hits, lessThan(r.misses ~/ 10 + 1)); + }); + + test('auto-detect path on one full night, with and without appended data', + () { + final night = genNight(11, headMin: 40, stillMin: 200, tailMin: 30); + final nightEnd = t0 + night.accel.length; + final ref = stage(night); + expect(ref.present, isTrue, reason: 'synthetic night must qualify'); + final cache = CardioRowCache(); + final cold = staged(night, cache); + expectIdentical(cold.out, ref, 'cold'); + expect(cold.misses, greaterThan(300)); + final warm = staged(night, cache); + expectIdentical(warm.out, ref, 'warm'); + expect(warm.misses, 0); + + // Same night plus 15 min of new data appended after the session. + // Build "appended" by concatenation (15 min of new data after the end). + final extra = genNight(12, headMin: 15, stillMin: 0, tailMin: 0, + start: nightEnd); + final appended = Night( + [...night.accel, ...extra.accel], + [...night.hr, ...extra.hr], + [...night.rr, ...extra.rr], + [...night.rrTs, ...extra.rrTs], + ); + final ref2 = stage(appended); + final app = staged(appended, cache); + expectIdentical(app.out, ref2, 'appended'); + expect(app.hits, greaterThan(0)); + expect(app.hits, greaterThanOrEqualTo((cold.misses * 0.8).floor())); + }); + }); + + group('reuse across windows', () { + test('same data in a window whose arrays start 1 h earlier', () { + // Edge stages D then D+1 windows over overlapping record ranges. + final cache = CardioRowCache(); + final early = base.slice(t0 + 20 * 60, t0 + total); // array starts later + final first = staged(early, cache, onset: winA0 + 600, offset: winA1); + expectIdentical(first.out, + stage(early, onset: winA0 + 600, offset: winA1), 'first'); + final second = staged(base, cache, onset: winA0 + 600, offset: winA1); + expectIdentical(second.out, + stage(base, onset: winA0 + 600, offset: winA1), 'second'); + expect(second.hits, greaterThan(0)); + }); + + test('session start moved by k*30 s', () { + for (final k in [1, 5, 20]) { + final cache = CardioRowCache(); + final first = staged(base, cache, onset: winA0, offset: winA1); + final staged1 = first.misses; + final newOnset = winA0 + k * 30; + final alive = copyOf(cache) + ..retainWhere((key) => key.startSec >= newOnset); + final second = staged(base, cache, onset: newOnset, offset: winA1); + expectIdentical(second.out, + stage(base, onset: newOnset, offset: winA1), 'start +$k'); + expect(second.hits, greaterThan(0)); + expect(second.hits, greaterThanOrEqualTo((alive.length * 0.8).floor()), + reason: 'k=$k of $staged1 first-run rows'); + } + }); + + test('session start moved by a non-multiple of the epoch (grid shifts)', + () { + final cache = CardioRowCache(); + staged(base, cache, onset: winA0, offset: winA1); + final r = staged(base, cache, onset: winA0 + 7, offset: winA1); + expectIdentical(r.out, stage(base, onset: winA0 + 7, offset: winA1), + 'start +7 s'); + }); + + test('only the end moved (earlier and later): most epochs hit', () { + for (final delta in [-1200, -90, 600]) { + final cache = CardioRowCache(); + staged(base, cache, onset: winA0, offset: winA1); + final newEnd = winA1 + delta; + final alive = copyOf(cache) + ..retainWhere((key) => key.startSec + key.epochSec <= newEnd); + final second = staged(base, cache, onset: winA0, offset: newEnd); + expectIdentical(second.out, + stage(base, onset: winA0, offset: newEnd), 'end $delta'); + expect(second.hits, greaterThanOrEqualTo((alive.length * 0.8).floor()), + reason: 'end moved by $delta s'); + expect(second.hits, greaterThan(0)); + } + }); + + test('extra data appended after the session (forced window unchanged)', () { + final extra = genNight(5, headMin: 20, stillMin: 0, tailMin: 0, + start: t0 + total); + final more = Night( + [...base.accel, ...extra.accel], + [...base.hr, ...extra.hr], + [...base.rr, ...extra.rr], + [...base.rrTs, ...extra.rrTs], + ); + final cache = CardioRowCache(); + final first = staged(base, cache, onset: winA0, offset: winA1); + final second = staged(more, cache, onset: winA0, offset: winA1); + expectIdentical(second.out, + stage(more, onset: winA0, offset: winA1), 'appended'); + expect(second.misses, 0); + expect(second.hits, first.misses); + }); + }); + + group('change detection', () { + // Context of an epoch: own 30 s, +-150 s (accel median, RR). One more epoch + // of slack either side for window edges, so at most 13 rows may be dirty. + const maxDirty = 13; + + Future mutate(String name, Night Function(Night) f, int atSec) async { + test(name, () { + final cache = CardioRowCache(); + final first = staged(base, cache, onset: winA0, offset: winA1); + final d = f(base.copy()); + final second = staged(d, cache, onset: winA0, offset: winA1); + expectIdentical(second.out, + stage(d, onset: winA0, offset: winA1), name); + expect(second.misses, greaterThanOrEqualTo(1)); + expect(second.misses, lessThanOrEqualTo(maxDirty), reason: name); + expect(second.hits, first.misses - second.misses); + }); + } + + const mid = 2400; // seconds into the record, deep inside the window + + mutate('one HR sample', (d) { + d.hr[mid] += 11; + return d; + }, mid); + + mutate('one accel sample', (d) { + final a = d.accel[mid]; + d.accel[mid] = AccelSample(a.tsMs, a.x + 0.4, a.y, a.z); + return d; + }, mid); + + mutate('one RR beat value', (d) { + final i = d.rrTs.indexWhere((t) => t >= (t0 + mid) * 1000.0); + d.rr[i] += 23; + return d; + }, mid); + + mutate('one RR beat removed', (d) { + final i = d.rrTs.indexWhere((t) => t >= (t0 + mid) * 1000.0); + d.rr.removeAt(i); + d.rrTs.removeAt(i); + return d; + }, mid); + + test('sample at the very start of the window only dirties near rows', () { + final cache = CardioRowCache(); + final first = staged(base, cache, onset: winA0, offset: winA1); + final d = base.copy(); + final i = winA0 - t0 + 3; + d.hr[i] += 9; + final second = staged(d, cache, onset: winA0, offset: winA1); + expectIdentical( + second.out, stage(d, onset: winA0, offset: winA1), 'edge'); + expect(second.misses, inInclusiveRange(1, maxDirty)); + expect(second.hits, first.misses - second.misses); + }); + }); + + group('json round trip', () { + test('populated cache -> json text -> cache: identical staging, all hits', + () { + final cache = CardioRowCache(); + final cold = staged(base, cache, onset: winA0, offset: winA1); + final text = jsonEncode(cache.toJson()); + final back = CardioRowCache.fromJson( + jsonDecode(text) as Map); + expect(back.length, cache.length); + expect(jsonEncode(back.toJson()), text, reason: 'lossless, stable'); + final warm = staged(base, back, onset: winA0, offset: winA1); + expectIdentical(warm.out, cold.out, 'revived'); + expect(warm.misses, 0); + expect(warm.hits, cold.misses); + }); + + test('toJson is deterministic across independent runs', () { + final a = CardioRowCache(), b = CardioRowCache(); + staged(base, a, onset: winA0, offset: winA1); + staged(base.copy(), b, onset: winA0, offset: winA1); + expect(jsonEncode(a.toJson()), jsonEncode(b.toJson())); + }); + + test('empty cache round trips', () { + final e = CardioRowCache.fromJson( + jsonDecode(jsonEncode(CardioRowCache().toJson())) + as Map); + expect(e.length, 0); + }); + + test('wrong or missing version throws FormatException', () { + final cache = CardioRowCache(); + staged(base, cache, onset: winA0, offset: winA0 + 1800); + final good = jsonDecode(jsonEncode(cache.toJson())) as Map; + expect(good.containsKey('version'), isTrue); + for (final v in [null, 'x', -1, 0, 9999, good['version'] is int ? (good['version'] as int) + 1 : 2]) { + final bad = Map.of(good)..['version'] = v; + expect(() => CardioRowCache.fromJson(bad), + throwsA(isA()), + reason: 'version=$v'); + } + final noVersion = Map.of(good)..remove('version'); + expect(() => CardioRowCache.fromJson(noVersion), + throwsA(isA())); + }); + + test('malformed payloads throw FormatException, never wrong rows', () { + final cache = CardioRowCache(); + staged(base, cache, onset: winA0, offset: winA0 + 1800); + final good = jsonDecode(jsonEncode(cache.toJson())) as Map; + expect(() => CardioRowCache.fromJson({}), + throwsA(isA())); + for (final k in good.keys.where((k) => k != 'version')) { + final bad = Map.of(good)..[k] = 'garbage'; + expect(() => CardioRowCache.fromJson(bad), + throwsA(isA()), + reason: 'key $k replaced by a string'); + final v = good[k]; + if (v is List && v.isNotEmpty) { + final badFirst = Map.of(good) + ..[k] = ['garbage', ...v.skip(1)]; + expect(() => CardioRowCache.fromJson(badFirst), + throwsA(isA()), + reason: 'key $k first element garbage'); + final truncated = Map.of(good) + ..[k] = v.sublist(0, v.length - 1); + // A shorter list is either rejected or yields strictly fewer rows + // (a parallel-array layout MUST reject); it must never add rows. + try { + final c = CardioRowCache.fromJson(truncated); + expect(c.length, lessThanOrEqualTo(cache.length)); + } on FormatException { + // acceptable + } + } + } + }); + }); + + group('retainWhere and merge', () { + test('retainWhere prunes, kept rows still hit, dropped rows miss', () { + final cache = CardioRowCache(); + final first = staged(base, cache, onset: winA0, offset: winA1); + final n = cache.length; + final cut = winA0 + 1800; + cache.retainWhere((k) => k.startSec < cut); + expect(cache.length, lessThan(n)); + expect(cache.length, greaterThan(0)); + final second = staged(base, cache, onset: winA0, offset: winA1); + expectIdentical(second.out, first.out, 'pruned'); + expect(second.hits, greaterThan(0)); + expect(second.misses, greaterThan(0)); + expect(second.hits + second.misses, first.misses); + cache.retainWhere((_) => false); + expect(cache.length, 0); + }); + + test('merge unions two caches; merged cache hits both windows', () { + final a = CardioRowCache(), b = CardioRowCache(); + final midS = winA0 + 1800; + final ra = staged(base, a, onset: winA0, offset: midS); + final rb = staged(base, b, onset: midS, offset: winA1); + // Wall-clock rows whose context is shared by both windows have the + // same key, including a row at the seam. Union must deduplicate them. + final unionKeys = { + for (final cache in [a, b]) + for (final row in cache.toJson()['rows'] as List) + '${row['startSec']}:${row['epochSec']}:${row['fingerprint']}', + }; + a.merge(b); + expect(a.length, unionKeys.length); + expect(a.length, lessThanOrEqualTo(ra.misses + rb.misses)); + final w1 = staged(base, a, onset: winA0, offset: midS); + final w2 = staged(base, a, onset: midS, offset: winA1); + expect(w1.misses, 0); + expect(w2.misses, 0); + expectIdentical(w1.out, stage(base, onset: winA0, offset: midS), 'w1'); + expectIdentical(w2.out, stage(base, onset: midS, offset: winA1), 'w2'); + }); + + test('merging a copy of itself changes nothing', () { + final a = CardioRowCache(); + staged(base, a, onset: winA0, offset: winA0 + 1800); + final n = a.length; + a.merge(copyOf(a)); + expect(a.length, n); + }); + }); +} diff --git a/test/onehz/cardio_run_cache_test.dart b/test/onehz/cardio_run_cache_test.dart new file mode 100644 index 00000000..cadeaccf --- /dev/null +++ b/test/onehz/cardio_run_cache_test.dart @@ -0,0 +1,26 @@ +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +void main() { + test('cached gravity references stop at run boundaries', () { + final hr = List.filled(180, 55); + final accel = [for (var i = 0; i < 180; i++) + AccelSample((i + (i >= 90 ? 1000 : 0)) * 1000.0, + 0, 0, i < 90 ? 1.0 : 1.1)]; + final cache = CardioRowCache(); + CardioEpochFeatures extract(List starts) => cardioEpochFeatures( + hr, accel, runStarts: starts, rowCache: cache); + // Two perfectly still postures, one on each side of a recording gap. + // Each run's own reference makes ENMO exactly zero in all six epochs. + expect(extract([90]).motion, List.filled(6, 0)); + expect(extract([90]).motion, List.filled(6, 0)); + expect(cache.hits, 6); + // Removing the boundary changes the dependencies even with identical raw + // arrays, so cached gap-clipped medians must not be reused. + final joined = extract([]); + expect(joined.motion.sublist(3), everyElement(closeTo(0.05, 1e-12))); + final fresh = cardioEpochFeatures(hr, accel); + expect(joined.motion, fresh.motion); + expect(extract([90]).motion, List.filled(6, 0)); + }); +} diff --git a/test/onehz/causal_stager_real_night_test.dart b/test/onehz/causal_stager_real_night_test.dart new file mode 100644 index 00000000..0ea55fc2 --- /dev/null +++ b/test/onehz/causal_stager_real_night_test.dart @@ -0,0 +1,245 @@ +// REGRESSION — the causal stager on the REAL WHOOP-4 overnight capture. +// +// Same fixture as real_night_cardio_stager_test.dart (8 h 54 min, anonymised, +// 1 Hz HR + accel + RR). There are no epoch labels for it — only the Apple +// Watch's whole-night minutes — so what is pinned here is AGREEMENT WITH THE +// RETROSPECTIVE STAGER, not accuracy: the online rules must keep tracking the +// offline ones, abstain only when they should, and keep the REM trigger from +// firing away from REM. `tool/causal_stager_validate.dart` prints the full +// report these bands were set from (kappa 0.369, agreement 70.3 %, causal wake +// 62.5 min vs offline 26.0, REM 78 vs 150 min). Bands are wide on purpose. + +import 'dart:io'; + +import 'package:test/test.dart'; +import 'package:openstrap_analytics/onehz.dart'; + +const double _t0Ms = 1700000010000; // 30 s-aligned + +class _Night { + final List hr = []; + final List accel = []; + final List rr = [], rrTs = []; +} + +_Night? _load() { + final a = File('test/onehz/fixtures/real_night_2026_07_onehz.csv'); + final b = File('test/onehz/fixtures/real_night_2026_07_rr.csv'); + if (!a.existsSync() || !b.existsSync()) return null; + final n = _Night(); + final l1 = a.readAsLinesSync(); + for (var i = 1; i < l1.length; i++) { + if (l1[i].trim().isEmpty) continue; + final p = l1[i].split(','); + n.hr.add(double.parse(p[1])); + n.accel.add(AccelSample(double.parse(p[0]) * 1000.0, double.parse(p[2]), + double.parse(p[3]), double.parse(p[4]))); + } + final l2 = b.readAsLinesSync(); + for (var i = 1; i < l2.length; i++) { + if (l2[i].trim().isEmpty) continue; + final p = l2[i].split(','); + n.rrTs.add(double.parse(p[0])); + n.rr.add(double.parse(p[1])); + } + return n; +} + +List _replay(_Night n, int toSec, + {double Function(int sec, double bpm)? hrOf}) { + final out = []; + CausalStagerState? st; + var rrI = 0; + for (var t = 30; t <= toSec; t += 30) { + final lo = t - 30; + final ts = [], v = []; + while (rrI < n.rrTs.length && n.rrTs[rrI] < t * 1000.0) { + ts.add(_t0Ms + n.rrTs[rrI]); + v.add(n.rr[rrI]); + rrI++; + } + final o = CausalStager.observe( + CausalSampleWindow( + nowMs: _t0Ms + t * 1000.0, + hr: [ + for (var i = lo; i < t; i++) + HrSample(_t0Ms + i * 1000.0, hrOf?.call(i, n.hr[i]) ?? n.hr[i]) + ], + accel: [ + for (var i = lo; i < t; i++) + AccelSample(_t0Ms + i * 1000.0, n.accel[i].x, n.accel[i].y, + n.accel[i].z) + ], + rr: RrSeries(ts, v), + ), + st); + out.add(o); + st = o.nextState; + } + return out; +} + +double _kappa(List a, List b) { + final n = a.length; + final m = List.generate(3, (_) => List.filled(3, 0)); + for (var i = 0; i < n; i++) { + m[a[i]][b[i]]++; + } + var po = 0.0, pe = 0.0; + for (var i = 0; i < 3; i++) { + po += m[i][i]; + var c = 0; + for (var j = 0; j < 3; j++) { + c += m[j][i]; + } + pe += m[i].reduce((x, y) => x + y) * c / n; + } + return (po / n - pe / n) / (1 - pe / n); +} + +void main() { + final night = _load(); + if (night == null) { + test('real night fixtures not found', () { + markTestSkipped('real night fixtures not found'); + }); + return; + } + final cfg = CausalStagerConfig.defaults; + final off = cardioStager(night.hr, night.accel, + rrMs: night.rr, rrTsMs: night.rrTs) + .base + .stages; + const idx = {CausalStage.wake: 0, CausalStage.nrem: 1, CausalStage.rem: 2}; + const base = 1700000010000 ~/ 30000; + late List obs; + setUpAll(() => obs = _replay(night, night.hr.length)); + + int offK(CausalStageObservation o) => + (o.epochStartMs! / 30000).round() - base; + + test('abstains only while warming up, then stages every epoch', () { + final absent = obs.where((o) => o.stage == CausalStage.absent).toList(); + expect(absent.length, cfg.warmupEpochs - 1); + expect(absent.every((o) => o.abstentionReason == CausalAbstention.warmup), + isTrue); + expect(obs.last.stage, isNot(CausalStage.absent)); + }); + + test('tracks the retrospective stager (kappa and agreement bands)', () { + final a = [], b = []; + for (final o in obs.where((o) => o.stage != CausalStage.absent)) { + final k = offK(o); + if (k < 0 || k >= off.length) continue; + a.add(idx[o.stage]!); + b.add(off[k].index); + } + var agree = 0; + for (var i = 0; i < a.length; i++) { + if (a[i] == b[i]) agree++; + } + expect(agree / a.length, greaterThan(0.62)); // measured 0.703 + expect(_kappa(a, b), greaterThan(0.28)); // measured 0.369 + }); + + test('stage minutes stay sane: no wake blow-up, REM not collapsed', () { + var wake = 0, rem = 0, nrem = 0; + for (final o in obs.where((o) => o.stage != CausalStage.absent)) { + switch (o.stage) { + case CausalStage.wake: + wake++; + case CausalStage.rem: + rem++; + case CausalStage.nrem: + nrem++; + case CausalStage.absent: + break; + } + } + // Causal wake is NOT Webster-bridged, so it runs above the offline 26 min + // (measured 62.5). The guard is against the posture-artifact blow-up the + // 1 g reference window exists to prevent (a 300 s reference gave 104 min). + expect(wake * 0.5, lessThan(90)); + expect(rem * 0.5, greaterThan(40)); // measured 78 + expect(nrem * 0.5, greaterThan(250)); // measured 374 + }); + + test('REM trigger: stable REM lands near offline REM, never far from it', + () { + bool offRem(int k) => k >= 0 && k < off.length && off[k] == SleepStage.rem; + bool near(int k) { + for (var d = -10; d <= 10; d++) { + if (offRem(k + d)) return true; + } + return false; + } + + for (final x in [120, 300]) { + final trig = obs + .where((o) => + o.stage == CausalStage.rem && o.runSec >= x) + .map(offK) + .toList(); + expect(trig, isNotEmpty, reason: 'X=$x never fired'); + final far = trig.where((k) => !near(k)).length; + // measured 0 at X=120 and 300 + expect(far / trig.length, lessThan(0.05), reason: 'X=$x false triggers'); + } + // X=120 s catches most offline REM bouts inside the bout (measured 11/16). + final bouts = <(int, int)>[]; + for (var i = 0; i < off.length;) { + if (!offRem(i)) { + i++; + continue; + } + var j = i; + while (j < off.length && offRem(j)) { + j++; + } + bouts.add((i, j)); + i = j; + } + final trig = { + for (final o in obs) + if (o.stage == CausalStage.rem && o.runSec >= 120) offK(o) + }; + var caught = 0; + for (final (s, e) in bouts) { + if (List.generate(e - s, (i) => s + i).any(trig.contains)) caught++; + } + expect(caught / bouts.length, greaterThan(0.5)); + }); + + test('the whole replay is deterministic (first 3 h, twice)', () { + final a = _replay(night, 3 * 3600); + final b = _replay(night, 3 * 3600); + for (var i = 0; i < a.length; i++) { + expect(b[i].toJson().toString(), a[i].toJson().toString()); + } + // ...and equal to the corresponding prefix of the full replay (causality). + for (var i = 0; i < a.length; i++) { + expect(obs[i].toJson().toString(), a[i].toJson().toString()); + } + }); + + test('a strap-off gap mid-night abstains with offWrist, then recovers', () { + // 4h00-4h10 off-skin (hr 0), everything else untouched. + const lo = 4 * 3600, hi = 4 * 3600 + 600; + final r = _replay(night, 5 * 3600, + hrOf: (s, bpm) => (s >= lo && s < hi) ? 0.0 : bpm); + var inside = 0; + for (final o in r) { + if (o.epochStartMs == null) continue; + final t = (o.epochStartMs! - _t0Ms) / 1000 + 30; // epoch end, s + if (t >= lo + 30 && t <= hi) { + inside++; + expect(o.stage, CausalStage.absent, reason: 'at ${t}s'); + expect(o.abstentionReason, CausalAbstention.offWrist); + } + } + expect(inside, 20, reason: '10 min = 20 epochs checked'); + final after = r.where((o) => + o.epochStartMs != null && (o.epochStartMs! - _t0Ms) / 1000 > hi + 600); + expect(after.every((o) => o.stage != CausalStage.absent), isTrue); + }); +} diff --git a/test/onehz/causal_stager_rollover_test.dart b/test/onehz/causal_stager_rollover_test.dart new file mode 100644 index 00000000..bcacec3e --- /dev/null +++ b/test/onehz/causal_stager_rollover_test.dart @@ -0,0 +1,105 @@ +// CAUSAL stager: label persistence across history rollover, the recent-coverage +// gate on stored labels, and persisted-row invariants. +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import 'support/causal_night.dart' show kT0Ms; + +/// Constant HR (60) and accel (0, 0, 1) for seconds [lo, hi); decision time at +/// the end of the window. +CausalSampleWindow constWindow(int lo, int hi) => CausalSampleWindow( + nowMs: kT0Ms + hi * 1000.0, + hr: [for (var s = lo; s < hi; s++) HrSample(kT0Ms + s * 1000.0, 60)], + accel: [ + for (var s = lo; s < hi; s++) + AccelSample(kT0Ms + s * 1000.0, 0, 0, 1) + ], + ); + +List labels(CausalStageObservation o) => + [for (final r in o.nextState.rows) r.label?.name]; + +void main() { + // why: once history rolls over, the rows an earlier epoch's baseline used are + // gone. Re-deciding that epoch on the shorter baseline cleared its label, so a + // window that merely re-sent 120 s of identical history changed the run and + // the state. + // # catches: warmup == history == 40, 40 epochs, then epoch 41 with and + // without a 120 s overlap: runSec 60 vs 30 and different labels. + test('re-sent identical history does not change labels after rollover', () { + const cfg = CausalStagerConfig(warmupEpochs: 40, historyEpochs: 40); + CausalStagerState? s; + for (var e = 0; e < 40; e++) { + s = CausalStager.observe(constWindow(e * 30, e * 30 + 30), s ?? CausalStagerState.initial(cfg)).nextState; + } + final plain = CausalStager.observe(constWindow(1200, 1230), s); + final overlap = CausalStager.observe(constWindow(1200 - 120, 1230), s); + expect(plain.stage, isNot(CausalStage.absent), reason: 'the setup stages'); + expect(overlap.stage, plain.stage); + expect(overlap.runSec, plain.runSec); + expect(labels(overlap), labels(plain)); + expect(overlap.nextState.toJson(), plain.nextState.toJson()); + }); + + // why: an epoch the observer abstains on for coverage must not keep a label + // that a later run can count. + // # catches: 40 usable epochs, 22 missing, 20 usable: the abstained epochs + // carry labels and the first covered observation reports a run that counts + // them (300 s here instead of 30 s). + test('an epoch that fails recent coverage gets no label to extend a run', () { + const cfg = CausalStagerConfig(warmupEpochs: 20, historyEpochs: 360); + CausalStagerState? s; + for (var e = 0; e < 40; e++) { + s = CausalStager.observe(constWindow(e * 30, e * 30 + 30), + s ?? CausalStagerState.initial(cfg)) + .nextState; + } + final byEpoch = {}; + for (var e = 62; e < 82; e++) { + final o = CausalStager.observe(constWindow(e * 30, e * 30 + 30), s); + s = o.nextState; + byEpoch[e] = o; + } + final firstCovered = byEpoch.entries + .firstWhere((e) => e.value.stage != CausalStage.absent) + .key; + expect(firstCovered, greaterThan(62), reason: 'the gap really abstains'); + for (var e = 62; e < firstCovered; e++) { + expect(byEpoch[e]!.abstentionReason, CausalAbstention.lowCoverage, + reason: 'epoch $e'); + } + // The run starts at the first covered epoch: abstained ones break it. + expect(byEpoch[firstCovered]!.runSec, 30); + expect(byEpoch[firstCovered + 1]!.runSec, 60); + for (final r in s!.rows.where((r) => r.epoch >= 62 && r.epoch < firstCovered)) { + expect(r.label, isNull, reason: 'epoch ${r.epoch}'); + } + }); + + // why: a usable row is read for its motion in every decision. + // # catches: a persisted row with motion == null passes `usable` and throws + // at r.motion! on the next decision. + group('persisted rows', () { + List row({Object? motion = 0.01, Object? hr = 60.0, Object? label}) => + [5, 30, 0, 30, motion, hr, 1.0, null, null, null, label]; + + test('a usable row without motion is refused', () { + expect(CausalEpochRow.fromJson(row()), isNotNull); + expect(CausalEpochRow.fromJson(row(motion: null)), isNull); + }); + test('counts and labels must be consistent', () { + expect(CausalEpochRow.fromJson([5, -1, 0, 30, 0.01, 60.0, 1.0, null, null, null, null]), isNull); + expect(CausalEpochRow.fromJson([5, 31, 0, 30, 0.01, 60.0, 1.0, null, null, null, null]), isNull); + // A label on a row that is not usable (no HR value). + expect(CausalEpochRow.fromJson(row(hr: null, label: 1)), isNull); + expect(CausalEpochRow.fromJson(row(label: 1)), isNotNull); + }); + test('a state holding such a row restarts instead of crashing', () { + final s = CausalStager.observe(constWindow(0, 30), null).nextState; + final json = s.toJson(); + final bad = Map.of(json); + bad['rows'] = [row(motion: null)]; + expect(CausalStagerState.fromJson(bad), isNull); + }); + }); +} diff --git a/test/onehz/causal_stager_test.dart b/test/onehz/causal_stager_test.dart new file mode 100644 index 00000000..c6d9050e --- /dev/null +++ b/test/onehz/causal_stager_test.dart @@ -0,0 +1,664 @@ +// CAUSAL (online) sleep stager — contract tests. +// +// What is pinned here, in the order the tests appear: +// 1. causality — nothing after `nowMs` can influence an output +// 2. determinism — same inputs, same bytes; survives JSON + an isolate hop +// 3. replacement — re-sent / overlapping / corrected windows converge on +// the state a clean feed would have produced +// 4. abstention — every reason fires, carries its reason, and never +// leaves a stage, a confidence or a run behind +// 5. warm-up — exact have/need accounting, then it stages +// 6. behaviour — on separable synthetic regimes it stages what is there +// 7. state — plain data, bounded, rejects garbage +// +// The synthetic night is in support/causal_night.dart. + +import 'dart:convert'; +import 'dart:isolate'; + +import 'package:test/test.dart'; +import 'package:openstrap_analytics/onehz.dart'; + +import 'package:openstrap_analytics/src/onehz/sleep/cardio_stager.dart' + show cleanBeatsInWindowForTest; + +import 'support/causal_night.dart'; + +/// Everything an observation decides, excluding bookkeeping counters in the +/// trace (e.g. how many future samples were ignored) — those legitimately +/// differ when a caller sends more than it should. +String core(CausalStageObservation o) => jsonEncode([ + o.stage.name, + o.confidence, + o.evidenceAgeMs, + o.abstentionReason?.name, + o.runSec, + o.epochStartMs, + o.note, + o.nextState.toJson(), + ]); + +/// Full signature, trace included. +String full(CausalStageObservation o) => + jsonEncode([o.toJson(), o.nextState.toJson()]); + +/// Stream a night in `stepSec` increments from `fromSec` to `toSec`, optionally +/// re-sending `overlapSec` of history in every window. Returns every +/// observation. +List stream( + SynthNight n, + int toSec, { + int fromSec = 0, + int stepSec = 30, + int overlapSec = 0, + CausalStagerState? start, +}) { + final out = []; + var state = start; + for (var t = fromSec + stepSec; t <= toSec; t += stepSec) { + final o = CausalStager.observe( + windowOf(n, t - stepSec - overlapSec, t), state); + out.add(o); + state = o.nextState; + } + return out; +} + +void expectAbsent(CausalStageObservation o, CausalAbstention why) { + expect(o.stage, CausalStage.absent); + expect(o.abstentionReason, why); + expect(o.confidence, 0, reason: 'absent ⇒ confidence 0'); + expect(o.runSec, 0); +} + +void main() { + final night = synthNight(standardNight()); + final warmSec = CausalStagerConfig.defaults.warmupEpochs * 30; + + group('exports', () { + test('the public barrel exposes the whole surface', () { + // Compiles only if every name is exported from onehz.dart. + expect(CausalStage.values.map((s) => s.name), + containsAll(['wake', 'nrem', 'rem', 'absent'])); + expect(CausalAbstention.values, isNotEmpty); + expect(CausalStagerConfig.defaults.warmupEpochs, greaterThan(0)); + }); + }); + + group('causality', () { + test('samples after nowMs never change any output or state', () { + CausalStagerState? sPast, sFut; + var sawIgnored = false; + for (var t = 30; t <= 100 * 60; t += 30) { + final past = windowOf(night, t - 30, t); + // same call, but the window also carries 10 min of FUTURE samples + final fut = windowOf(night, t - 30, t + 600, nowSec: t); + final a = CausalStager.observe(past, sPast); + final b = CausalStager.observe(fut, sFut); + expect(core(b), core(a), reason: 'diverged at t=${t}s'); + if (((b.trace['ignoredFuture'] as num?) ?? 0) > 0) sawIgnored = true; + sPast = a.nextState; + sFut = b.nextState; + } + expect(sawIgnored, isTrue, + reason: 'the guard must actually have been exercised'); + }); + + test('rewriting the future cannot move the present', () { + final base = stream(night, 90 * 60); + // Same history, then a window whose samples after `now` are garbage. + final t = 90 * 60; + final state = base[base.length - 2].nextState; + final clean = windowOf(night, t - 30, t); + final dirty = CausalSampleWindow( + nowMs: clean.nowMs, + hr: [ + ...clean.hr, + for (var i = 1; i <= 120; i++) + HrSample(clean.nowMs + i * 1000.0, 190), // a future "wake" + ], + accel: [ + ...clean.accel, + for (var i = 1; i <= 120; i++) + AccelSample(clean.nowMs + i * 1000.0, 3, 3, 3), + ], + rr: clean.rr, + ); + expect(core(CausalStager.observe(dirty, state)), + core(CausalStager.observe(clean, state))); + }); + + test('an output at time t is identical whether or not the stream continues', + () { + final short = stream(night, 80 * 60); + final long = stream(night, 120 * 60); + for (var i = 0; i < short.length; i++) { + expect(full(long[i]), full(short[i])); + } + }); + }); + + group('determinism', () { + test('the same inputs give byte-identical outputs and state', () { + final a = stream(night, 70 * 60); + final b = stream(night, 70 * 60); + for (var i = 0; i < a.length; i++) { + expect(full(b[i]), full(a[i])); + } + }); + + test('state survives JSON (restart / persistence) without changing results', + () { + CausalStagerState? plain, viaJson; + for (var t = 30; t <= 80 * 60; t += 30) { + final w = windowOf(night, t - 30, t); + final a = CausalStager.observe(w, plain); + final b = CausalStager.observe(w, viaJson); + expect(full(b), full(a), reason: 'diverged at t=${t}s'); + plain = a.nextState; + final j = jsonDecode(jsonEncode(b.nextState.toJson())); + viaJson = CausalStagerState.fromJson(j as Map); + expect(viaJson, isNotNull); + } + }); + + test('the result does not depend on how the stream is chunked', () { + final fine = stream(night, 3600, stepSec: 30).last; + final mid = stream(night, 3600, stepSec: 300).last; + final coarse = stream(night, 3600, stepSec: 3600).last; + expect(core(mid), core(fine)); + expect(core(coarse), core(fine)); + }); + + test('state and window cross an isolate boundary and give the same answer', + () async { + final upTo = stream(night, 60 * 60); + final state = upTo[upTo.length - 2].nextState; + final w = windowOf(night, 60 * 60 - 30, 60 * 60); + final local = CausalStager.observe(w, state); + final remote = await Isolate.run(() => CausalStager.observe(w, state)); + expect(full(remote), full(local)); + }); + }); + + group('incremental replacement', () { + test('re-sending the same window changes nothing', () { + final upTo = stream(night, 50 * 60); + final prior = upTo[upTo.length - 2].nextState; + final w = windowOf(night, 50 * 60 - 30, 50 * 60); + final first = CausalStager.observe(w, prior); + final again = CausalStager.observe(w, first.nextState); + expect(core(again), core(first)); + final thrice = CausalStager.observe(w, again.nextState); + expect(core(thrice), core(first)); + }); + + test('overlapping windows converge on the same state as a clean feed', () { + final clean = stream(night, 75 * 60); + final overlap = stream(night, 75 * 60, overlapSec: 120); + for (var i = 0; i < clean.length; i++) { + expect(core(overlap[i]), core(clean[i]), reason: 'step $i'); + } + }); + + test('a corrected re-send replaces earlier samples', () { + final t = 70 * 60; + final upTo = stream(night, t - 30); + final prior = upTo.last.nextState; + // First delivery of the last 3 min is corrupt (HR 40 bpm off-spec)... + final lo = t - 180; + final good = windowOf(night, lo, t); + // ...and every beat doubled, which only a span REPLACEMENT (not a + // per-key upsert) can undo. + final bad = CausalSampleWindow( + nowMs: good.nowMs, + hr: [for (final h in good.hr) HrSample(h.tsMs, 40)], + accel: good.accel, + rr: RrSeries([ + for (var i = 0; i < good.rr.length; i++) ...[ + good.rr.tsMs[i], + good.rr.tsMs[i] + ] + ], [ + for (var i = 0; i < good.rr.length; i++) ...[ + good.rr.rrMs[i], + good.rr.rrMs[i] + ] + ]), + ); + final afterBad = CausalStager.observe(bad, prior); + // ...then the producer re-sends the correct window. + final fixed = CausalStager.observe(good, afterBad.nextState); + final clean = CausalStager.observe(good, prior); + expect(core(fixed), core(clean)); + expect(jsonEncode(fixed.nextState.toJson()), + jsonEncode(clean.nextState.toJson())); + }); + + test('samples older than the replacement horizon are ignored, not applied', + () { + final t = 80 * 60; + final upTo = stream(night, t); + final state = upTo.last.nextState; + // Re-send an old minute with different content, with a fresh `now`. + final old = windowOf(night, 20 * 60, 21 * 60, nowSec: t); + final poisoned = CausalSampleWindow( + nowMs: old.nowMs, + hr: [for (final h in old.hr) HrSample(h.tsMs, 150)], + accel: old.accel, + rr: old.rr, + ); + final o = CausalStager.observe(poisoned, state); + expect(o.trace['ignoredStale'], greaterThan(0)); + expect(jsonEncode(o.nextState.toJson()), + jsonEncode(state.toJson())); + }); + + test('duplicate seconds inside one window: the later entry wins', () { + final t = 50 * 60; + final prior = stream(night, t - 30).last.nextState; + final w = windowOf(night, t - 30, t); + final withDup = CausalSampleWindow( + nowMs: w.nowMs, + hr: [for (final h in w.hr) HrSample(h.tsMs, 150), ...w.hr], + accel: w.accel, + rr: w.rr, + ); + expect(core(CausalStager.observe(withDup, prior)), + core(CausalStager.observe(w, prior))); + }); + }); + + group('abstention', () { + // Stream clean to `t`, then hand back state + the last observation. + CausalStagerState warm(int t) => stream(night, t).last.nextState; + final t = warmSec + 20 * 60; + + test('absent window, no prior: noEvidence, with a usable state', () { + final o = CausalStager.observe( + CausalSampleWindow(nowMs: kT0Ms, hr: const [], accel: const []), + null); + expectAbsent(o, CausalAbstention.noEvidence); + expect(o.evidenceAgeMs, isNull); + expect(o.epochStartMs, isNull); + expect(jsonEncode(o.nextState.toJson()), isNotEmpty); + }); + + test('data stopped (disconnect): staleEvidence, never a carried stage', () { + final s = warm(t); + final ok = CausalStager.observe(windowOf(night, t, t + 30), s); + expect(ok.stage, isNot(CausalStage.absent), reason: 'precondition'); + final o = CausalStager.observe( + CausalSampleWindow( + nowMs: night.t0Ms + (t + 30 + 300) * 1000.0, + hr: const [], + accel: const []), + ok.nextState); + expectAbsent(o, CausalAbstention.staleEvidence); + expect(o.evidenceAgeMs, greaterThanOrEqualTo(300 * 1000)); + }); + + test('evidence just inside the freshness limit still stages', () { + final s = warm(t); + final maxAge = CausalStagerConfig.defaults.maxEvidenceAgeSec; + final o = CausalStager.observe( + CausalSampleWindow( + nowMs: night.t0Ms + (t + maxAge - 5) * 1000.0, + hr: const [], + accel: const []), + s); + expect(o.stage, isNot(CausalStage.absent)); + expect(o.evidenceAgeMs, lessThanOrEqualTo(maxAge * 1000)); + }); + + test('clock going backwards: clockRegressed, state untouched', () { + final s = warm(t); + final o = CausalStager.observe( + windowOf(night, t - 120, t - 60, nowSec: t - 300), s); + expectAbsent(o, CausalAbstention.clockRegressed); + expect(jsonEncode(o.nextState.toJson()), jsonEncode(s.toJson())); + }); + + test('hr == 0 (off-skin) for the latest epoch: offWrist', () { + final s = warm(t); + final w = windowOf(night, t, t + 60); + final off = CausalSampleWindow( + nowMs: w.nowMs, + hr: [for (final h in w.hr) HrSample(h.tsMs, 0)], + accel: w.accel, + rr: w.rr, + ); + expectAbsent(CausalStager.observe(off, s), CausalAbstention.offWrist); + }); + + test('no HR at all in the latest epoch: missingHr', () { + final s = warm(t); + final w = windowOf(night, t, t + 60, hrOn: false); + expectAbsent(CausalStager.observe(w, s), CausalAbstention.missingHr); + }); + + test('non-finite HR is missing, not a value', () { + final s = warm(t); + final w = windowOf(night, t, t + 60); + final nan = CausalSampleWindow( + nowMs: w.nowMs, + hr: [for (final h in w.hr) HrSample(h.tsMs, double.nan)], + accel: w.accel, + rr: w.rr, + ); + expectAbsent(CausalStager.observe(nan, s), CausalAbstention.missingHr); + }); + + test('no accel in the latest epoch: missingAccel', () { + final s = warm(t); + final w = windowOf(night, t, t + 60, accelOn: false); + expectAbsent(CausalStager.observe(w, s), CausalAbstention.missingAccel); + }); + + test('accel flagged invalid: missingAccel', () { + final s = warm(t); + final w = windowOf(night, t, t + 60); + final inv = CausalSampleWindow( + nowMs: w.nowMs, + hr: w.hr, + accel: [ + for (final a in w.accel) + AccelSample(a.tsMs, a.x, a.y, a.z, valid: false) + ], + rr: w.rr, + ); + expectAbsent(CausalStager.observe(inv, s), CausalAbstention.missingAccel); + }); + + test('too few HR seconds in the epoch: lowCoverage', () { + final s = warm(t); + final w = windowOf(night, t, t + 60); + final sparse = CausalSampleWindow( + nowMs: w.nowMs, + hr: [ + for (var i = 0; i < w.hr.length; i++) + if (i % 30 < 4) w.hr[i] // 4 of every 30 seconds + ], + accel: w.accel, + rr: w.rr, + ); + expectAbsent( + CausalStager.observe(sparse, s), CausalAbstention.lowCoverage); + }); + + test('too few accel seconds in the epoch: lowCoverage', () { + final s = warm(t); + final w = windowOf(night, t, t + 60); + final sparse = CausalSampleWindow( + nowMs: w.nowMs, + hr: w.hr, + accel: [ + for (var i = 0; i < w.accel.length; i++) + if (i % 30 < 4) w.accel[i] + ], + rr: w.rr, + ); + expectAbsent( + CausalStager.observe(sparse, s), CausalAbstention.lowCoverage); + }); + + test('a staged stream recovers after a gap and does not carry old labels', + () { + final s = warm(t); + // 3 minutes of nothing (strap off the charger, say), then data again + final gapEnd = t + 180; + final resumed = CausalStager.observe( + windowOf(night, gapEnd, gapEnd + 60), s); + // the first closed epoch after the gap is fine; the point is that during + // the gap itself the answer was absent + final during = CausalStager.observe( + CausalSampleWindow( + nowMs: night.t0Ms + (t + 150) * 1000.0, + hr: const [], + accel: const []), + s); + expectAbsent(during, CausalAbstention.staleEvidence); + expect(resumed.stage, isNot(CausalStage.absent)); + }); + + test('a stream with no usable HR never produces a stage', () { + CausalStagerState? state; + for (var s = 30; s <= 60 * 60; s += 30) { + final w = windowOf(night, s - 30, s); + final blind = CausalSampleWindow( + nowMs: w.nowMs, + hr: [for (final h in w.hr) HrSample(h.tsMs, double.nan)], + accel: w.accel, + rr: w.rr, + ); + final o = CausalStager.observe(blind, state); + expect(o.stage, CausalStage.absent, reason: 'second $s'); + expect(o.abstentionReason, isNotNull); + state = o.nextState; + } + }); + + test('invariants: absent <=> reason; present ⇒ confidence in (0, 0.6]', () { + for (final o in stream(night, 100 * 60)) { + if (o.stage == CausalStage.absent) { + expect(o.abstentionReason, isNotNull); + expect(o.confidence, 0); + expect(o.runSec, 0); + } else { + expect(o.abstentionReason, isNull); + expect(o.confidence, inExclusiveRange(0, 0.6 + 1e-12)); + expect(o.runSec, greaterThan(0)); + expect(o.epochStartMs, isNotNull); + expect(o.evidenceAgeMs, isNotNull); + } + } + }); + }); + + group('warm-up', () { + test('abstains with an exact have/need count, then stages', () { + final need = CausalStagerConfig.defaults.warmupEpochs; + final obs = stream(night, (need + 6) * 30); + // After k calls, k epochs are closed (the stream starts epoch-aligned). + for (var k = 1; k <= need + 6; k++) { + final o = obs[k - 1]; + if (k < need) { + expectAbsent(o, CausalAbstention.warmup); + expect(o.note, 'warmup:have=$k,need=$need'); + } else { + expect(o.stage, isNot(CausalStage.absent), reason: 'k=$k'); + expect(o.note, isNull); + } + } + }); + + test('a thin start does not count epochs that were never usable', () { + // 10 min of off-skin first: none of it may count toward warm-up. + final need = CausalStagerConfig.defaults.warmupEpochs; + CausalStagerState? s; + for (var t = 30; t <= 600; t += 30) { + final w = windowOf(night, t - 30, t); + final off = CausalSampleWindow( + nowMs: w.nowMs, + hr: [for (final h in w.hr) HrSample(h.tsMs, 0)], + accel: w.accel, + rr: w.rr, + ); + s = CausalStager.observe(off, s).nextState; + } + final o = CausalStager.observe(windowOf(night, 600, 630), s); + expectAbsent(o, CausalAbstention.warmup); + expect(o.note, 'warmup:have=1,need=$need'); + }); + }); + + group('behaviour on separable regimes', () { + late List obs; + setUpAll(() => obs = stream(night, night.seconds)); + + // Ground truth for the epoch an observation describes. + String truthOf(CausalStageObservation o) { + final sec = ((o.epochStartMs! - night.t0Ms) / 1000).round(); + // majority regime across the 30 s epoch + final c = {}; + for (var i = sec; i < sec + 30; i++) { + c[night.truth[i]] = (c[night.truth[i]] ?? 0) + 1; + } + return c.entries.reduce((a, b) => a.value >= b.value ? a : b).key; + } + + test('REM is called inside REM blocks, mostly after the first minutes', () { + final inRem = obs.where((o) => + o.stage != CausalStage.absent && truthOf(o) == 'rem').toList(); + expect(inRem.length, greaterThan(20)); + final hit = inRem.where((o) => o.stage == CausalStage.rem).length; + expect(hit / inRem.length, greaterThan(0.6)); + }); + + test('NREM time is not called wake', () { + final inN = obs.where((o) => + o.stage != CausalStage.absent && truthOf(o) == 'nrem').toList(); + final wake = inN.where((o) => o.stage == CausalStage.wake).length; + expect(wake / inN.length, lessThan(0.05)); + }); + + // The REM "wake now" rule a caller would build: REM that has persisted for + // at least the shortest credible REM bout (remEpisodeMinMin = 5 min). + bool stableRem(CausalStageObservation o) => + o.stage == CausalStage.rem && o.runSec >= remEpisodeMinMin * 60; + + // [lo, hi) minute spans of the REM blocks in standardNight(). + const remBlocks = [(35, 47), (123, 138)]; + double minOf(CausalStageObservation o) => + (o.epochStartMs! - night.t0Ms) / 60000.0; + + test('every REM block raises a stable-REM candidate inside the block, ' + 'within 8 min of onset', () { + for (final (lo, hi) in remBlocks) { + final first = obs + .where((o) => stableRem(o) && minOf(o) >= lo && minOf(o) < hi) + .map(minOf) + .toList(); + expect(first, isNotEmpty, reason: 'REM block $lo-$hi min missed'); + expect(first.first - lo, lessThanOrEqualTo(8)); + } + }); + + test('stable-REM candidates far from any REM block are rare', () { + // "far" = more than 6 min after a block ended (trailing windows keep the + // tail of REM variability in view for a few minutes — the lag the doc + // states) and not inside one. + bool far(double m) => remBlocks.every((b) => m < b.$1 || m >= b.$2 + 6); + final farEpochs = obs + .where((o) => o.stage != CausalStage.absent && far(minOf(o))) + .toList(); + final falseTrig = farEpochs.where(stableRem).length; + expect(falseTrig / farEpochs.length, lessThan(0.06)); + }); + + test('a motion/HR wake block is called wake', () { + final inW = obs.where((o) => + o.stage != CausalStage.absent && truthOf(o) == 'wake').toList(); + expect(inW, isNotEmpty); + final hit = inW.where((o) => o.stage == CausalStage.wake).length; + expect(hit / inW.length, greaterThan(0.6)); + }); + + test('runSec counts how long the stage has persisted, in whole epochs', () { + var prev = CausalStage.absent; + var prevRun = 0.0; + for (final o in obs) { + if (o.stage == CausalStage.absent) { + prev = CausalStage.absent; + prevRun = 0; + continue; + } + expect(o.runSec % 30, 0); + if (o.stage == prev) { + expect(o.runSec, prevRun + 30); + } else { + expect(o.runSec, 30); + } + prev = o.stage; + prevRun = o.runSec; + } + }); + + test('evidence age is small for a live feed', () { + for (final o in obs.where((o) => o.stage != CausalStage.absent)) { + expect(o.evidenceAgeMs, lessThanOrEqualTo(2000)); + } + }); + }); + + group('shared building blocks (extracted from cardioStager, unchanged)', () { + test('weightedAxisScore: weighted mean over measurable axes only', () { + expect(weightedAxisScore([(1.0, 0.5), (null, 0.3), (double.nan, 0.2)]), + 1.0); + expect(weightedAxisScore([(2.0, 0.5), (-1.0, 0.5)]), 0.5); + expect(weightedAxisScore([(null, 0.5)]), isNull); + expect(weightedAxisScore([(1.0, -0.5), (3.0, 0.5)]), 1.0, + reason: 'a negative weight flips the axis; the denominator is |w|'); + }); + + test('cleanRrBeatsBetween is the same gate as the centred-window gather', + () { + final accel = [ + for (var i = 0; i < 30; i++) AccelSample(i * 1000.0, 0, 0, 1) + ]; + final ts = [9000, 10000, 10000, 15000, 20000, 20000, 21000]; + final rr = [900, 910, 5000, 920, 930, 240, 940]; // 5000/240 gated + final centred = + cleanBeatsInWindowForTest(rr, ts, accel, 0, 30, halfWinMs: 5000); + final direct = cleanRrBeatsBetween(rr, ts, 10000, 20000); + expect(direct.beats, centred); + expect(direct.tsSec.first, 0, reason: 'rebased to the window start'); + }); + + test('REM weights are the shipped effect sizes', () { + expect( + [kRemWeightRk, kRemWeightSdnn, kRemWeightHrSd, kRemWeightLfhf], + [0.43, 0.32, 0.27, 0.12]); + }); + }); + + group('state', () { + test('is plain JSON data', () { + final s = stream(night, 60 * 60).last.nextState; + final j = s.toJson(); + expect(jsonDecode(jsonEncode(j)), j); + }); + + test('is bounded no matter how long the stream runs', () { + final long = synthNight([('nrem', 7 * 3600)], seed: 5); + final s = stream(long, long.seconds, stepSec: 600).last.nextState; + final j = s.toJson(); + final cfg = CausalStagerConfig.defaults; + expect((j['rows'] as List).length, lessThanOrEqualTo(cfg.historyEpochs)); + expect((j['hr'] as List).length, lessThanOrEqualTo(1000)); + expect((j['accel'] as List).length, lessThanOrEqualTo(1000)); + }); + + test('a round-trip is the identity', () { + final s = stream(night, 45 * 60).last.nextState; + final back = CausalStagerState.fromJson( + jsonDecode(jsonEncode(s.toJson())) as Map); + expect(jsonEncode(back!.toJson()), jsonEncode(s.toJson())); + }); + + test('garbage and unknown versions are rejected (null), not guessed at', () { + final j = stream(night, 45 * 60).last.nextState.toJson(); + expect(CausalStagerState.fromJson({}), isNull); + expect(CausalStagerState.fromJson({...j, 'v': 999}), isNull); + expect(CausalStagerState.fromJson({...j, 'rows': 'nope'}), isNull); + expect(CausalStagerState.fromJson({...j, 'hr': [1, 2, 3]}), isNull); + }); + + test('an old state with a different config keeps its own config', () { + const cfg = CausalStagerConfig(warmupEpochs: 10); + final s0 = CausalStagerState.initial(cfg); + final o = CausalStager.observe(windowOf(night, 0, 30), s0); + expect(o.note, 'warmup:have=1,need=10'); + }); + }); +} diff --git a/test/onehz/causal_stager_validate_args_test.dart b/test/onehz/causal_stager_validate_args_test.dart new file mode 100644 index 00000000..94775660 --- /dev/null +++ b/test/onehz/causal_stager_validate_args_test.dart @@ -0,0 +1,47 @@ +// tool/causal_stager_validate.dart: option values are read with their options. +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../../tool/causal_stager_validate.dart'; + +void main() { + // # catches: `--step 60` leaves "60" among the positional arguments and the + // tool then opens "60" as the one-Hz fixture ("fixture not found"). + test('option values are not fixture paths', () { + final a = parseValidateArgs(['--step', '60']); + expect(a.error, isNull); + expect(a.pos, isEmpty); + expect(a.stepSec, 60); + expect(a.warmup, CausalStagerConfig.defaults.warmupEpochs); + + final b = parseValidateArgs(['--warmup', '20']); + expect(b.pos, isEmpty); + expect(b.warmup, 20); + expect(b.stepSec, 30); + + final c = parseValidateArgs( + ['a.csv', 'b.csv', '--step', '60', '--warmup', '20']); + expect(c.pos, ['a.csv', 'b.csv']); + expect((c.stepSec, c.warmup), (60, 20)); + + final d = parseValidateArgs(['--step=45', 'a.csv', '--warmup=10', 'b.csv']); + expect(d.pos, ['a.csv', 'b.csv']); + expect((d.stepSec, d.warmup), (45, 10)); + }); + + test('a bad command line is reported, not guessed at', () { + expect(parseValidateArgs(['--step']).error, contains('needs a value')); + expect(parseValidateArgs(['--step', 'x']).error, contains('positive whole')); + expect(parseValidateArgs(['--step', '0']).error, contains('positive whole')); + expect(parseValidateArgs(['--warmup=-3']).error, contains('positive whole')); + expect(parseValidateArgs(['--nope']).error, contains('unknown option')); + }); + + test('no arguments means the defaults', () { + final a = parseValidateArgs(const []); + expect(a.error, isNull); + expect(a.pos, isEmpty); + expect((a.stepSec, a.warmup), + (30, CausalStagerConfig.defaults.warmupEpochs)); + }); +} diff --git a/test/onehz/epoch_rows_anchor_test.dart b/test/onehz/epoch_rows_anchor_test.dart new file mode 100644 index 00000000..ef5c0fd2 --- /dev/null +++ b/test/onehz/epoch_rows_anchor_test.dart @@ -0,0 +1,352 @@ +// Combined bump, grid option 2 (owner decision 2026-10-23): onset-anchored +// staging plus a phase-keyed epoch-row memo. RED until the implementation lands. +// Contract: edge.research/combined-bump/option2-brief.md. +// +// New API under test (everything else already exists): +// +// EpochRows({CardioRowCache? rowCache, int? continuingAnchorMs, +// int? anchorMs}) +// anchorMs: the first run's grid passes through this absolute time, rows are +// anchorMs + k * kEpochMs. No row starts before anchorMs when it lies after +// the first sample (earlier samples only serve as feature context); when it +// lies before the first sample the first row is the epoch holding it. A run +// that follows a data gap longer than the usable-run gap still re-anchors at +// its first sample (this is what the pre-bump stager did; see +// test/onehz/support/pre_bump_stager.dart). +// int? EpochRows.phaseMs anchor mod kEpochMs of the first run's grid +// (null while no anchor/sample is known). +// toJson()['phase'] the same value, carried in the checkpoint; +// fromJson refuses a phase its rows contradict. +// +// EpochRowsMemo({int maxPhases = 2, CardioRowCache? rowCache}) +// EpochRows rowsFor(int anchorMs, {hr, accel, rrMs, rrTsMs}) +// Rows for the phase anchorMs % kEpochMs. The lists are the CUMULATIVE +// input from the start of the data (append-only between calls). A held +// phase folds only the samples it has not consumed; an unheld phase is +// built from scratch from the lists (exact recompute) and the least +// recently used phase beyond maxPhases is dropped. +// List phases held phases, least recently used first +// int phaseBuilds phases built from scratch (incl. rebuilds) +// int samplesFolded samples handed to EpochRows.fold, hr + accel + rr +// toJson() / EpochRowsMemo.fromJson(map) checkpoint; carries each held +// phase's rows and how many cumulative samples it consumed. A restored +// memo reuses a phase only when it is asked for the same phase. +import 'dart:convert'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/property.dart'; +import 'support/epoch_input.dart'; + +EpochRows _batch(EpochInput s, int anchorMs) { + final er = EpochRows(anchorMs: anchorMs); + s.foldInto(er, 0, s.seconds); + return er; +} + +EpochRows _chunked(EpochInput s, int anchorMs, List cuts, + {bool restart = false}) { + var er = EpochRows(anchorMs: anchorMs); + var at = 0; + for (final c in [...cuts, s.seconds]) { + s.foldInto(er, at, c); + at = c; + if (restart) { + er = EpochRows.fromJson( + jsonDecode(jsonEncode(er.toJson())) as Map); + } + } + return er; +} + +EpochRows _ask(EpochRowsMemo m, int anchorMs, EpochInput upTo) => + m.rowsFor(anchorMs, + hr: upTo.hr, accel: upTo.accel, rrMs: upTo.rrMs, rrTsMs: upTo.rrTs); + +void main() { + final t0 = kAlignedT0Ms + 13000; // phase 13000 + final night = EpochInput.make(nightSegs(), t0Ms: t0, seed: 41); + + group('EpochRows(anchorMs:) puts rows on anchorMs + k * 30 s', () { + for (final (name, a) in [ + ('before the first sample', t0 - 40000), + ('at the first sample', t0), + ('a few seconds in', t0 + 7000), + ('777 s in (a detected onset)', t0 + 777000), + ]) { + test('anchor $name', () { + final er = _batch(night, a); + final rows = er.allRows(); + expect(rows, isNotEmpty); + expect(er.phaseMs, a % kEpochMs); + for (final r in rows) { + expect((r.startMs - a) % kEpochMs, 0, reason: 'start ${r.startMs}'); + } + for (var i = 1; i < rows.length; i++) { + expect(rows[i].startMs - rows[i - 1].startMs, kEpochMs); + } + if (a >= t0) { + expect(rows.first.startMs, a, reason: 'no row before the anchor'); + } else { + expect(rows.first.startMs, lessThanOrEqualTo(t0)); + expect(rows.first.startMs + kEpochMs, greaterThan(t0), + reason: 'the epoch holding the first sample opens the grid'); + } + final lastEnd = t0 + night.seconds * 1000; + expect(rows.last.startMs + kEpochMs, + inInclusiveRange(lastEnd, lastEnd + kEpochMs)); + }); + } + + test('a different phase gives different row starts over the same samples', + () { + final a = _batch(night, t0 + 7000).allRows(); + final b = _batch(night, t0 + 19000).allRows(); + expect((a[3].startMs - b[3].startMs) % kEpochMs, isNot(0)); + }); + + forAll('any chunking, with JSON restarts (even before the first sample), ' + 'equals one fold on the same anchor', G.intIn(1, 100000), (seed) { + final r = Rng(seed); + final a = t0 + [0, 7000, 19000, 777000, -40000][r.nextInt(5)]; + final want = _batch(night, a).allRows(); + final cuts = randomCuts(r, night.seconds); + final got = _chunked(night, a, cuts, restart: r.nextBool()); + expectSameRows(got.allRows(), want, 'anchor $a cuts $cuts'); + }, cases: 10, budget: const Duration(seconds: 120)); + + test('an anchored checkpoint taken before any sample keeps the anchor', () { + final a = t0 + 777000; + final empty = EpochRows(anchorMs: a); + final back = EpochRows.fromJson( + jsonDecode(jsonEncode(empty.toJson())) as Map); + night.foldInto(back, 0, night.seconds); + expectSameRows(back.allRows(), _batch(night, a).allRows(), + 'resumed from an empty anchored checkpoint'); + }); + + test('resumed mid-night from a checkpoint = batch on that anchor', () { + final a = t0 + 7000; + final er = EpochRows(anchorMs: a); + night.foldInto(er, 0, 1700); + final back = EpochRows.fromJson( + jsonDecode(jsonEncode(er.toJson())) as Map); + night.foldInto(back, 1700, night.seconds); + expectSameRows(back.allRows(), _batch(night, a).allRows(), 'resumed'); + }); + + test('after a gap longer than the usable-run gap the next sample re-anchors ' + '(pre-bump behaviour): the onset grid is NOT kept across the gap', () { + // 100 s hole at +1300..+1400 s. Anchor +7 s; the second run starts at + // +1400 s, 1393 s = 13 s (mod 30) past the onset grid. + final s = EpochInput.make(nightSegs(), + t0Ms: t0, gaps: [(1300, 1400)], seed: 43); + final a = t0 + 7000; + final er = _batch(s, a); + final starts = er.allRows().map((r) => r.startMs).toList(); + for (final st in starts) { + final rel = st - (t0 + 1400000); + if (st < t0 + 1300000) { + expect((st - a) % kEpochMs, 0, reason: 'first run on the onset grid'); + } else { + expect(rel % kEpochMs, 0, reason: 'second run on its own grid: $st'); + } + } + expect(starts.contains(t0 + 1400000), isTrue); + expect(starts.contains(a + 1400000 ~/ 30000 * 30000 + 30000), isFalse, + reason: 'the old phase is gone after the gap'); + }); + + test('a hole the carry bridges keeps the onset grid', () { + final s = EpochInput.make(nightSegs(), + t0Ms: t0, gaps: [(1000, 1040)], seed: 45); + final a = t0 + 7000; + for (final r in _batch(s, a).allRows()) { + expect((r.startMs - a) % kEpochMs, 0, reason: 'start ${r.startMs}'); + } + }); + + test('the checkpoint carries the phase, and refuses one its rows contradict', + () { + final a = t0 + 7000; + final er = EpochRows(anchorMs: a); + night.foldInto(er, 0, 1200); + expect(er.phaseMs, a % kEpochMs); + final good = jsonDecode(jsonEncode(er.toJson())) as Map; + expect(good['phase'], a % kEpochMs); + expect(EpochRows.fromJson(good).phaseMs, a % kEpochMs); + final bad = {...good, 'phase': (a % kEpochMs + 1000) % kEpochMs}; + expect(() => EpochRows.fromJson(bad), throwsA(isA())); + }); + }); + + group('EpochRowsMemo: rows keyed by phase', () { + final a1 = t0 + 7000; // phase 20000 + final a2 = t0 + 19000; // phase 2000 (32000 mod 30000) + final a3 = t0 + 23000; // phase 6000 + + test('same phase: rows are reused and extended, never recomputed', () { + final m = EpochRowsMemo(); + final early = night.between(0, 1000); + final late = night.between(0, 1500); + final r1 = _ask(m, a1, early); + expect(m.phaseBuilds, 1); + expect(m.samplesFolded, early.sampleCount); + expectSameRows(r1.allRows(), _batch(early, a1).allRows(), 'first ask'); + final r2 = _ask(m, a1, late); + expect(m.phaseBuilds, 1, reason: 'same phase: no rebuild'); + expect(m.samplesFolded, late.sampleCount, + reason: 'every sample folded exactly once'); + expectSameRows(r2.allRows(), _batch(late, a1).allRows(), 'extended'); + // nothing new: nothing folded + _ask(m, a1, late); + expect(m.samplesFolded, late.sampleCount); + expect(m.phaseBuilds, 1); + expect(m.phases, [a1 % kEpochMs]); + }); + + test('an onset that moves by whole epochs keeps the phase and the rows', () { + final m = EpochRowsMemo(); + final all = night.between(0, 2000); + _ask(m, a1, all); + final moved = a1 + 20 * kEpochMs; + final r = _ask(m, moved, all); + expect(m.phaseBuilds, 1); + expect(m.samplesFolded, all.sampleCount); + final want = _batch(all, moved).allRows(); + expectSameRows( + [for (final x in r.allRows()) if (x.startMs >= moved) x], want, + 'rows from the new onset on'); + }); + + test('a different phase yields rows exactly equal to a fresh batch on that ' + 'phase, and the memo keeps the new phase', () { + final m = EpochRowsMemo(); + final all = night.between(0, 2400); + _ask(m, a1, all); + final r = _ask(m, a2, all); + expect(m.phaseBuilds, 2); + expect(m.phases.last, a2 % kEpochMs); + expectSameRows(r.allRows(), _batch(all, a2).allRows(), 'phase changed'); + for (final x in r.allRows()) { + expect((x.startMs - a2) % kEpochMs, 0); + } + // and the old phase is still exact when asked again with more data + final more = night.between(0, night.seconds); + final back = _ask(m, a1, more); + expectSameRows(back.allRows(), _batch(more, a1).allRows(), 'back to a1'); + }); + + test('the memo is bounded: at most 2 phases are kept, always exact', () { + final m = EpochRowsMemo(); + final all = night.between(0, 1800); + for (final a in [a1, a2, a3, a1, a3, a2]) { + final r = _ask(m, a, all); + expect(m.phases.length, lessThanOrEqualTo(2), reason: 'after $a'); + expect(m.phases.last, a % kEpochMs, reason: 'most recent is held'); + expectSameRows(r.allRows(), _batch(all, a).allRows(), 'anchor $a'); + } + expect(m.phaseBuilds, greaterThanOrEqualTo(4), + reason: 'a third phase evicts one; asking for it again rebuilds'); + }); + + test('checkpoint: same phase resumes without a rebuild', () { + final m = EpochRowsMemo(); + _ask(m, a1, night.between(0, 1000)); + final back = EpochRowsMemo.fromJson( + jsonDecode(jsonEncode(m.toJson())) as Map); + expect(back.phases, [a1 % kEpochMs]); + final more = night.between(0, 2200); + final r = _ask(back, a1, more); + expect(back.phaseBuilds, 0, reason: 'nothing rebuilt after the restore'); + expect(back.samplesFolded, more.sampleCount - night.between(0, 1000).sampleCount, + reason: 'only the samples the checkpoint had not consumed'); + expectSameRows(r.allRows(), _batch(more, a1).allRows(), 'resumed'); + }); + + test('checkpoint: a different phase rebuilds, never reusing the other ' + 'phase\'s rows, and still equals batch', () { + final m = EpochRowsMemo(); + _ask(m, a1, night.between(0, 1000)); + final back = EpochRowsMemo.fromJson( + jsonDecode(jsonEncode(m.toJson())) as Map); + final more = night.between(0, 2200); + final r = _ask(back, a2, more); + expect(back.phaseBuilds, 1); + expect(back.samplesFolded, more.sampleCount, + reason: 'built from all the samples'); + for (final x in r.allRows()) { + expect((x.startMs - a2) % kEpochMs, 0, reason: 'start ${x.startMs}'); + } + expectSameRows(r.allRows(), _batch(more, a2).allRows(), 'other phase'); + }); + + test('a memo embedding a version-3 row checkpoint is refused', () { + final m = EpochRowsMemo(); + _ask(m, a1, night.between(0, 1000)); + final good = jsonDecode(jsonEncode(m.toJson())) as Map; + for (final v in [1, 2, 3]) { + final bad = jsonDecode(jsonEncode(good)) as Map; + for (final p in bad['phases'] as List) { + (p['rows'] as Map)['version'] = v; + } + expect(() => EpochRowsMemo.fromJson(bad), + throwsA(isA()), reason: 'rows version $v'); + } + }); + + test('a checkpoint is refused when it is of another type or shape', () { + final m = EpochRowsMemo(); + _ask(m, a1, night.between(0, 600)); + final good = jsonDecode(jsonEncode(m.toJson())) as Map; + for (final bad in >[ + {...good, 'type': 'EpochRows'}, + {...good}..remove('type'), + {}, + ]) { + expect(() => EpochRowsMemo.fromJson(bad), + throwsA(isA()), reason: jsonEncode(bad)); + } + }); + }); + + group('staging from memo rows = batch staging from the onset', () { + for (final (name, onsetSec, endSec, firstAsk) in [ + ('onset 777 s in, fresh memo', 777, 2400, null), + ('onset 1234 s in, memo first asked on another phase', 1234, 2900, 3), + ('onset on a 30 s multiple of the data start', 600, 2100, null), + ]) { + test(name, () { + final base = t0 ~/ 1000; + final onset = base + onsetSec, end = base + endSec; + final want = AdvancedSleepStager.stageWindow( + onset, end, night.grav, night.hrTs, rr: night.rrTsSec) + .stages; + final m = EpochRowsMemo(); + final window = night.between(onsetSec, endSec); + if (firstAsk != null) { + _ask(m, t0 + 5000 + firstAsk * 1000, night.between(onsetSec, onsetSec + 900)); + } + final rows = _ask(m, onset * 1000, window); + expect(showStages(rows.stageWindow(onset, end)), showStages(want)); + }); + } + + test('staged in growing prefixes through one memo = batch at the end', () { + final base = t0 ~/ 1000; + final onset = base + 777, end = base + 2400; + final m = EpochRowsMemo(); + EpochRows? rows; + for (var hi = 1000; hi <= 2400; hi += 350) { + rows = _ask(m, onset * 1000, night.between(777, hi < 2400 ? hi : 2400)); + } + rows = _ask(m, onset * 1000, night.between(777, 2400)); + expect(m.phaseBuilds, 1); + final want = AdvancedSleepStager.stageWindow( + onset, end, night.grav, night.hrTs, rr: night.rrTsSec) + .stages; + expect(showStages(rows.stageWindow(onset, end)), showStages(want)); + }); + }); +} diff --git a/test/onehz/epoch_rows_cache_key_test.dart b/test/onehz/epoch_rows_cache_key_test.dart new file mode 100644 index 00000000..28e661d8 --- /dev/null +++ b/test/onehz/epoch_rows_cache_key_test.dart @@ -0,0 +1,45 @@ +// EpochRows cache fingerprint: byte-wise hashing. +// +// A row's cache key hashes the data the row can observe. Mixing a whole 64-bit +// word per step lets two sign-bit flips cancel, so a row whose data changed in +// exactly two sign bits kept its key and a warm lookup served the old row. +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import 'support/epoch_input.dart'; + +int _fold(EpochInput input, CardioRowCache cache) { + final er = EpochRows(rowCache: cache, anchorMs: input.t0Ms); + input.foldInto(er, 0, input.seconds); + er.allRows(); + return cache.misses; +} + +void main() { + // # catches: negating the x and y of one accel sample (two consecutive words + // of the key) leaves every key identical under the word-wise mix, so no row + // that sees the sample is looked up again. + test('two sign-bit changes in a row\'s data change its key', () { + final input = EpochInput.make(nightSegs(), t0Ms: kAlignedT0Ms, seed: 41); + final cache = CardioRowCache(); + final cold = _fold(input, cache); + expect(cold, greaterThan(20), reason: 'the first pass fills the cache'); + + final i = 600; + final a = input.accel[i]; + final accel = [...input.accel]..[i] = AccelSample(a.tsMs, -a.x, -a.y, a.z); + final changed = EpochInput( + input.t0Ms, input.seconds, input.hr, accel, input.rrMs, input.rrTs); + + final before = cache.misses; + _fold(changed, cache); + final misses = cache.misses - before; + // A 300 s key range holds about ten 30 s epochs; every one that sees the + // sample is recomputed. + expect(misses, greaterThanOrEqualTo(8)); + // The same data again is all hits. + final again = cache.misses; + _fold(changed, cache); + expect(cache.misses - again, 0); + }); +} diff --git a/test/onehz/epoch_rows_run_end_test.dart b/test/onehz/epoch_rows_run_end_test.dart new file mode 100644 index 00000000..954e2d3d --- /dev/null +++ b/test/onehz/epoch_rows_run_end_test.dart @@ -0,0 +1,41 @@ +// EpochRows: the gravity reference of a run that another run follows is read up +// to the batch staging boundary (whole epochs), not to the run's carry tail. +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import 'support/causal_night.dart' show kT0Ms; + +void main() { + // Run 1: seconds 0..999 (coverage runs 60 s past the last sample, to 1060). + // Run 2 starts at 1100. Batch staging cuts run 1 to whole epochs (1050), so + // the reference of the epoch at 900 s is the median over [750, 1050): 150 ones + // and 150 twos, 1.5, and its motion is 2 - 1.5 = 0.5. Reading to 1060 gave + // 150 ones and 160 twos: reference 2, motion 0. + // # catches: the sub-epoch carry tail of a non-last run entering the + // reference window. + for (final cached in [false, true]) { + test('a non-last run is cut to whole epochs for the reference ' + '(${cached ? 'with' : 'without'} a row cache)', () { + final hr = []; + final accel = []; + void second(int s, double mag) { + final ts = kT0Ms + s * 1000.0; + hr.add(HrSample(ts, 60)); + accel.add(AccelSample(ts, 0, 0, mag)); + } + + for (var s = 0; s < 1000; s++) { + second(s, s < 900 ? 1.0 : 2.0); + } + for (var s = 1100; s < 1500; s++) { + second(s, 1.0); + } + final er = EpochRows( + rowCache: cached ? CardioRowCache() : null, anchorMs: kT0Ms.toInt()); + er.fold(hr: hr, accel: accel, rrMs: const [], rrTsMs: const []); + final rows = er.allRows(); + final row = rows.firstWhere((r) => r.startMs == kT0Ms + 900000.0); + expect(row.motion, closeTo(0.5, 1e-9)); + }); + } +} diff --git a/test/onehz/epoch_rows_test.dart b/test/onehz/epoch_rows_test.dart new file mode 100644 index 00000000..dca11227 --- /dev/null +++ b/test/onehz/epoch_rows_test.dart @@ -0,0 +1,549 @@ +// The gap-anchored 30 s epoch grid and its `EpochRows` accumulator (combined +// bump, chunk B1-B3; owner decision 2026-10-22: anchor at the first sample after +// each data gap, replacing the wall-clock grid). +// +// B1 Epochs are 30,000 ms steps from the FIRST sample of each gap-free run +// (valid accelerometer coverage, with the stager's usable-run gap: the +// 60 s carry-forward). A run starts at the first sample and again at the +// first sample after any gap longer than that. The anchor depends on the +// data only, never on session detection, and is final as soon as the +// run's first sample arrives. An epoch is clipped only by a data gap: an +// epoch the data covers is a row (computed on the seconds that exist); +// an epoch with no data is not a row. +// B2 `EpochRows` (lookback +-165 s accel / HR, +-150 s RR) emits per-epoch +// feature rows. Chunk invariance is EXACT (any chunking, with JSON +// save/restore between chunks); rows ~3 min behind the newest sample are +// FINAL and never change; the last few are provisional; checkpointable. +// On an aligned contiguous stream the interior rows are today's +// `cardioEpochFeatures` rows (LF/HF excepted, see below). +// B3 Staging from the accumulated rows (`EpochRows.stageWindow`) equals +// `AdvancedSleepStager.stageWindow` in one batch, exactly, and that +// hypnogram's boundaries sit on the run's 30 s grid, whatever the session +// bounds. +// +// LF/HF is compared for presence only against today's rows: the span-dependent +// frequency grid (cardio_stager.dart `remFeaturesFromBeats`) may be fixed in +// the same bump, which moves the value but not whether it exists. +import 'dart:convert'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/property.dart'; +import 'support/causal_night.dart'; +import 'support/pre_bump_stager.dart'; + +const _alignedT0 = 1700000010000; // = kT0Ms, a multiple of 30,000 + +/// A night as raw input, optionally with holes (both streams lose the same +/// seconds, and the beats in them). +class _Input { + final int t0Ms, seconds; + final List hr; + final List accel; + final List rrMs, rrTs; + _Input(this.t0Ms, this.seconds, this.hr, this.accel, this.rrMs, this.rrTs); + + factory _Input.make(List<(String, int)> segs, + {int seed = 11, int t0Ms = _alignedT0, List<(int, int)> gaps = const []}) { + final n = synthNight(segs, seed: seed, t0Ms: t0Ms.toDouble()); + bool inGap(double ts) => gaps.any((g) => + ts >= t0Ms + g.$1 * 1000.0 && ts < t0Ms + g.$2 * 1000.0); + final rr = [], rt = []; + for (var i = 0; i < n.rrMs.length; i++) { + if (!inGap(n.rrTsMs[i])) { + rr.add(n.rrMs[i]); + rt.add(n.rrTsMs[i]); + } + } + return _Input(t0Ms, n.seconds, [for (final h in n.hr) if (!inGap(h.tsMs)) h], + [for (final a in n.accel) if (!inGap(a.tsMs)) a], rr, rt); + } + + /// Samples with `loSec <= second < hiSec` (seconds since t0). + void foldInto(EpochRows er, int loSec, int hiSec) { + final lo = t0Ms + loSec * 1000.0, hi = t0Ms + hiSec * 1000.0; + bool inside(double t) => t >= lo && t < hi; + final ri = [ + for (var i = 0; i < rrTs.length; i++) + if (inside(rrTs[i])) i + ]; + er.fold( + hr: [for (final h in hr) if (inside(h.tsMs)) h], + accel: [for (final a in accel) if (inside(a.tsMs)) a], + rrMs: [for (final i in ri) rrMs[i]], + rrTsMs: [for (final i in ri) rrTs[i]], + ); + } + + List get grav => [ + for (final a in accel) GravTs(a.tsMs ~/ 1000, a.x, a.y, a.z) + ]; + List get hrTs => [for (final h in hr) HrTs(h.tsMs ~/ 1000, h.hr)]; + List get rrTsSec => + [for (var i = 0; i < rrMs.length; i++) RrTs(rrTs[i] ~/ 1000, rrMs[i])]; +} + +EpochRows _oneFold(_Input s) { + final er = EpochRows(); + s.foldInto(er, 0, s.seconds); + return er; +} + +/// Folds [s] in chunks cut at [cuts] (seconds, ascending; empty chunks allowed), +/// saving to JSON TEXT and restoring between chunks when [restart]. +EpochRows _chunked(_Input s, List cuts, {bool restart = false}) { + var er = EpochRows(); + var at = 0; + for (final c in [...cuts, s.seconds]) { + s.foldInto(er, at, c); + at = c; + if (restart) { + er = EpochRows.fromJson(jsonDecode(jsonEncode(er.toJson())) as Map); + } + } + return er; +} + +List _cuts(Rng r, int seconds) { + const sizes = [1, 2, 7, 30, 31, 59, 100, 171, 500]; + final out = []; + var at = 0; + while (true) { + at += r.nextBool(.15) ? 0 : 1 + r.nextInt(sizes[r.nextInt(sizes.length)]); + if (at >= seconds) return out; + out.add(at); + } +} + +bool _sameRow(EpochFeatureRow a, EpochFeatureRow b) => + a.startMs == b.startMs && + [ + [a.motion, b.motion], [a.hr, b.hr], [a.hrSd, b.hrSd], + [a.rmssd, b.rmssd], [a.lfhf, b.lfhf], [a.rk, b.rk], [a.sdnn, b.sdnn], + ].every((p) => p[0].compareTo(p[1]) == 0); + +void _expectSameRows(List got, List want, String why) { + expect(got.length, want.length, reason: '$why: row count'); + for (var i = 0; i < got.length; i++) { + if (!_sameRow(got[i], want[i])) { + fail('$why: row $i (start ${want[i].startMs}) differs: ' + 'got ${_show(got[i])} want ${_show(want[i])}'); + } + } +} + +String _show(EpochFeatureRow r) => + '[${r.startMs} motion ${r.motion} hr ${r.hr} hrSd ${r.hrSd} rmssd ${r.rmssd} ' + 'lfhf ${r.lfhf} rk ${r.rk} sdnn ${r.sdnn}]'; + +List<(String, int)> _night() => [ + ('nrem', 900), ('rem', 600), ('nrem', 900), ('wake', 300), ('nrem', 300), + ]; + +void main() { + group('B1: the gap-anchored grid', () { + test('kEpochMs is 30,000 ms', () => expect(kEpochMs, 30000)); + + test('no data: no runs, no epoch', () { + final er = EpochRows(); + expect(er.runAnchorsMs, isEmpty); + expect(er.epochStartMsFor(_alignedT0), isNull); + }); + + for (final offsetMs in [0, 13000, 29000, 13250]) { + test('data starting ${offsetMs} ms past a Unix boundary: epochs are 30 s ' + 'steps from the first sample, every covered epoch is a row', () { + final s = _Input.make([('nrem', 700), ('wake', 140)], t0Ms: _alignedT0 + offsetMs); + final er = _oneFold(s); + expect(er.runAnchorsMs, [s.t0Ms]); + final rows = er.allRows(); + expect(rows, isNotEmpty); + expect(rows.first.startMs, s.t0Ms, reason: 'the first sample opens the grid'); + for (final r in rows) { + expect((r.startMs - s.t0Ms) % kEpochMs, 0, reason: 'start ${r.startMs}'); + } + for (var i = 1; i < rows.length; i++) { + expect(rows[i].startMs - rows[i - 1].startMs, kEpochMs, + reason: 'contiguous data: no epoch skipped or repeated'); + } + final lastEnd = s.t0Ms + s.seconds * 1000; + expect(rows.last.startMs + kEpochMs, + inInclusiveRange(lastEnd, lastEnd + kEpochMs), + reason: 'the epoch holding the last sample is the last row'); + // the per-timestamp lookup agrees with the rows + for (final r in rows) { + expect(er.epochStartMsFor(r.startMs), r.startMs); + expect(er.epochStartMsFor(r.startMs + kEpochMs - 1), r.startMs); + } + expect(er.epochStartMsFor(s.t0Ms - 1), isNull, reason: 'before the data'); + }); + } + + test('the anchor is final as soon as the first sample arrives', () { + final s = _Input.make([('nrem', 900)], t0Ms: _alignedT0 + 13250); + final er = EpochRows(); + s.foldInto(er, 0, 5); + expect(er.runAnchorsMs, [s.t0Ms]); + s.foldInto(er, 5, 400); + s.foldInto(er, 400, s.seconds); + expect(er.runAnchorsMs, [s.t0Ms], reason: 'later data never moves it'); + // and it does not depend on a session: nothing has been asked of the rows + expect(er.allRows().first.startMs, s.t0Ms); + }); + + test('a gap longer than the usable-run gap re-anchors at the first sample ' + 'after it; a shorter one does not', () { + // next valid accel at +61 s after the last: the carry bridges it (one run); + // at +62 s it does not (second run). Same threshold as the stager's + // usable-run split. + for (final (holeSec, runs) in [(60, 1), (61, 2), (100, 2), (600, 2)]) { + // a hole of `holeSec` missing seconds after +599 (both streams) + final s = _Input.make([('nrem', 1500)], + gaps: [(600, 600 + holeSec)], seed: 17, t0Ms: _alignedT0 + 13000); + final er = _oneFold(s); + expect(er.runAnchorsMs.length, runs, reason: 'hole $holeSec s'); + if (runs == 2) { + expect(er.runAnchorsMs.last, s.t0Ms + (600 + holeSec) * 1000); + // the new grid steps from the new anchor, not the old one + final starts = er.allRows().map((r) => r.startMs).toSet(); + expect(starts.contains(s.t0Ms + (600 + holeSec) * 1000), isTrue); + if ((holeSec) % 30 != 0) { + expect(starts.contains(s.t0Ms + (630 + (holeSec ~/ 30) * 30) * 1000), + isFalse, reason: 'the old phase is gone after the gap'); + } + } else { + for (final r in er.allRows()) { + expect((r.startMs - s.t0Ms) % kEpochMs, 0); + } + } + } + }); + + test('gap-only clipping: an epoch inside a gap is not a row, an epoch the ' + 'gap only nibbles still is (on the seconds that exist)', () { + // gap A: 117 s (a new run, anchored at +417); gap B: 10 s in the middle + // of one epoch (bridged). Offsets are seconds since t0. + final s = _Input.make([('nrem', 1500)], + gaps: [(300, 417), (810, 820)], seed: 17); + final er = _oneFold(s); + expect(er.runAnchorsMs, [s.t0Ms, s.t0Ms + 417000]); + final byStart = {for (final r in er.allRows()) r.startMs: r}; + // the first run's grid ends with its coverage (last sample + the carry) + for (var e = 360; e < 417; e += 1) { + expect(byStart.containsKey(s.t0Ms + e * 1000), isFalse, + reason: 'second +${e}s lies in the gap, no epoch starts there'); + } + for (final e in [270, 417, 447, 777, 837]) { + expect(byStart.containsKey(s.t0Ms + e * 1000), isTrue, + reason: 'epoch at +${e}s has data'); + } + final nibbled = byStart[s.t0Ms + 807 * 1000]!; // 417 + 13 * 30 = 807 + final present = [ + for (final h in s.hr) + if (h.tsMs >= s.t0Ms + 807000 && h.tsMs < s.t0Ms + 837000 && h.valid) + h.hr + ]; + expect(present.length, 20); + expect(nibbled.hr, closeTo(present.reduce((a, b) => a + b) / present.length, 1e-9), + reason: 'mean HR of the 20 seconds that exist'); + }); + + test('nothing is trimmed to a "session": the first and last wake epochs of a ' + 'night are rows', () { + final s = _Input.make([('wake', 300), ('nrem', 600), ('wake', 300)]); + final rows = _oneFold(s).allRows(); + expect(rows.first.startMs, _alignedT0); + expect(rows.last.startMs, _alignedT0 + 1200 * 1000 - kEpochMs); + }); + + test('a run that continues data folded earlier (seeded anchor) keeps the ' + 'earlier grid', () { + final s = _Input.make([('nrem', 2400)], t0Ms: _alignedT0 + 13250); + final full = _oneFold(s); + // fold only the part from +400 s on, telling the accumulator that the run + // it joins was anchored at the true first sample + final part = EpochRows(continuingAnchorMs: s.t0Ms); + s.foldInto(part, 400, s.seconds); + expect(part.runAnchorsMs, [s.t0Ms]); + final starts = part.allRows().map((r) => r.startMs).toList(); + expect(starts, isNotEmpty); + for (final st in starts) { + expect((st - s.t0Ms) % kEpochMs, 0); + } + // rows well inside the fed part are the full-night rows + final want = {for (final r in full.allRows()) r.startMs: r}; + var same = 0; + for (final r in part.allRows()) { + if (r.startMs < s.t0Ms + 400000 + 200000) continue; + expect(_sameRow(r, want[r.startMs]!), isTrue, reason: 'row ${r.startMs}'); + same++; + } + expect(same, greaterThan(50)); + }); + }); + + group('B2: EpochRows', () { + final night = _Input.make(_night(), t0Ms: _alignedT0 + 13000); + late final whole = _oneFold(night).allRows(); + + test('rows exist and are not all provisional', () { + expect(whole.length, greaterThanOrEqualTo(100), reason: '3000 s, 30 s epochs'); + final er = _oneFold(night); + expect(er.finalRows, isNotEmpty); + expect(er.provisionalRows(), isNotEmpty); + expect(er.finalRows.length + er.provisionalRows().length, whole.length); + }); + + forAll('chunk invariance: any chunking (empty chunks, JSON restarts) = one fold', + G.intIn(1, 100000), (seed) { + final r = Rng(seed); + final cuts = _cuts(r, night.seconds); + final got = _chunked(night, cuts, restart: r.nextBool()); + _expectSameRows(got.allRows(), whole, 'cuts $cuts'); + }, cases: 12, budget: const Duration(seconds: 120)); + + test('one-second chunks give the same rows', () { + final s = _Input.make([('nrem', 600), ('wake', 120)], t0Ms: _alignedT0 + 7000); + final want = _oneFold(s).allRows(); + final er = EpochRows(); + for (var t = 0; t < s.seconds; t++) { + s.foldInto(er, t, t + 1); + } + _expectSameRows(er.allRows(), want, '1 s chunks'); + }); + + test('rows ~3 min behind the newest sample are final and never change; the ' + 'last few are provisional', () { + final er = EpochRows(); + var prevFinal = 0; + var at = 0; + for (final c in [..._cuts(Rng(5), night.seconds), night.seconds]) { + night.foldInto(er, at, c); + at = c; + final fin = er.finalRows; + // never retracted, never rewritten: a prefix of the full-night rows + expect(fin.length, greaterThanOrEqualTo(prevFinal)); + _expectSameRows(fin, whole.sublist(0, fin.length), 'final rows at +${c}s'); + prevFinal = fin.length; + // liveness: everything 4+ min behind the newest sample is final + final newest = er.newestMs!; + final settledBy = whole.where((r) => r.startMs <= newest - 240000).length; + expect(fin.length, greaterThanOrEqualTo(settledBy), + reason: 'rows 4 min behind the newest sample must be final (at +${c}s)'); + // the provisional rows continue the final ones, and are few + final prov = er.provisionalRows(); + expect(prov.length, lessThanOrEqualTo(14)); + if (fin.isNotEmpty && prov.isNotEmpty) { + expect(prov.first.startMs, fin.last.startMs + kEpochMs); + } + expect(er.allRows().length, fin.length + prov.length); + } + expect(er.newestMs, night.t0Ms + (night.seconds - 1) * 1000); + }); + + test('provisional rows match a fold of the data so far, clipped at its end', () { + final cut = 1500; + final er = EpochRows(); + night.foldInto(er, 0, cut); + final prefixOnly = _Input.make(_night(), t0Ms: _alignedT0 + 13000); + final fresh = EpochRows(); + prefixOnly.foldInto(fresh, 0, cut); + _expectSameRows(er.allRows(), fresh.allRows(), 'same prefix, same rows'); + // asking does not change anything + final before = jsonEncode(er.toJson()); + er.provisionalRows(); + er.allRows(); + expect(jsonEncode(er.toJson()), before); + }); + + test('checkpoint: toJson -> JSON text -> fromJson is lossless and resumes', () { + final er = EpochRows(); + night.foldInto(er, 0, 1700); + final text = jsonEncode(er.toJson()); + final back = EpochRows.fromJson(jsonDecode(text) as Map); + expect(jsonEncode(back.toJson()), text); + _expectSameRows(back.allRows(), er.allRows(), 'restored'); + night.foldInto(back, 1700, night.seconds); + _expectSameRows(back.allRows(), whole, 'resumed'); + }); + + test('an empty accumulator checkpoints and resumes', () { + final back = EpochRows.fromJson( + jsonDecode(jsonEncode(EpochRows().toJson())) as Map); + expect(back.allRows(), isEmpty); + expect(back.newestMs, isNull); + night.foldInto(back, 0, night.seconds); + _expectSameRows(back.allRows(), whole, 'from an empty checkpoint'); + }); + + test('a checkpoint of another type, version or shape is refused', () { + final er = EpochRows(); + night.foldInto(er, 0, 400); + final good = jsonDecode(jsonEncode(er.toJson())) as Map; + for (final bad in >[ + {...good, 'type': 'RrCorrector'}, + {...good, 'version': 999}, + {...good}..remove('type'), + {'type': good['type'], 'version': good['version']}, + {}, + ]) { + expect(() => EpochRows.fromJson(bad), throwsA(isA()), + reason: jsonEncode(bad)); + } + }); + + test('checkpoints written before 81b9b4b (versions 1-3) are refused', () { + final er = EpochRows(); + night.foldInto(er, 0, 1000); + final good = jsonDecode(jsonEncode(er.toJson())) as Map; + expect(good['version'], 4, reason: 'row semantics changed: new version'); + for (final v in [1, 2, 3]) { + // Finalized rows carry the old (wrong) values; the shape is otherwise + // what version 3 wrote. + final bad = {...good, 'version': v}; + expect(() => EpochRows.fromJson(bad), throwsA(isA()), + reason: 'version $v'); + } + }); + + test('on an aligned contiguous stream the interior rows are today\'s ' + 'cardioEpochFeatures rows', () { + final s = _Input.make(_night()); // aligned t0 + final rows = {for (final r in _oneFold(s).allRows()) r.startMs: r}; + final f = cardioEpochFeatures( + [for (final h in s.hr) h.hr], + s.accel, + rrMs: s.rrMs, + rrTsMs: s.rrTs, + ); + var compared = 0; + for (var e = 0; e < f.length; e++) { + final startSec = e * 30; + if (startSec < 165 || startSec + 30 + 165 > s.seconds) continue; + final r = rows[_alignedT0 + startSec * 1000]; + expect(r, isNotNull, reason: 'row for epoch $e'); + bool same(double a, double b) => a.compareTo(b) == 0; + expect(same(r!.motion, f.motion[e]), isTrue, reason: 'motion e=$e ${r.motion} vs ${f.motion[e]}'); + expect(same(r.hr, f.hr[e]), isTrue, reason: 'hr e=$e'); + expect(same(r.hrSd, f.hrSd[e]), isTrue, reason: 'hrSd e=$e'); + expect(same(r.rmssd, f.rmssd[e]), isTrue, reason: 'rmssd e=$e'); + expect(same(r.sdnn, f.sdnn[e]), isTrue, reason: 'sdnn e=$e'); + expect(same(r.rk, f.rk[e]), isTrue, reason: 'rk e=$e'); + expect(r.lfhf.isNaN, f.lfhf[e].isNaN, reason: 'lfhf presence e=$e'); + compared++; + } + expect(compared, greaterThan(80)); + }); + }); + + group('B3: staging on the gap-anchored grid', () { + SleepSession batch(_Input s, [int fromSec = 0, int? toSec]) => + AdvancedSleepStager.stageWindow(s.t0Ms ~/ 1000 + fromSec, + s.t0Ms ~/ 1000 + (toSec ?? s.seconds), s.grav, s.hrTs, rr: s.rrTsSec); + + String show(List l) => jsonEncode([for (final g in l) g.toJson()]); + + for (final (name, offsetMs, gaps) in [ + ('unaligned start, no gaps', 13000, <(int, int)>[]), + ('aligned start', 0, <(int, int)>[]), + ('unaligned start, a 100 s hole in accel and HR', 13000, [(1300, 1400)]), + ]) { + test('staging from accumulated rows = one batch, exactly ($name)', () { + final s = _Input.make(_night(), t0Ms: _alignedT0 + offsetMs, gaps: gaps, seed: 29); + final want = batch(s).stages; + final r = Rng(offsetMs + gaps.length); + final er = _chunked(s, _cuts(r, s.seconds), restart: true); + final got = er.stageWindow(s.t0Ms ~/ 1000, s.t0Ms ~/ 1000 + s.seconds); + expect(show(got), show(want)); + }); + } + + test('a session that begins inside a gap-free run is staged on ITS OWN ' + 'start (grid option 2: phase 0 at the session start, as pre-bump)', () { + // Replaces the gap-anchored expectation (owner decision 2026-10-23): the + // run\'s grid no longer decides where a session\'s epochs sit. + final s = _Input.make(_night(), t0Ms: _alignedT0 + 13000, seed: 37); + for (final (from, to) in [(1000, 2400), (777, 2111), (1234, 2999)]) { + final t0 = s.t0Ms ~/ 1000; + final got = batch(s, from, to).stages; + var checked = 0; + for (var i = 1; i < got.length; i++) { + if (got[i].stage == 'unobserved' || got[i - 1].stage == 'unobserved') { + continue; + } + expect((got[i].start - (t0 + from)) % 30, 0, + reason: 'session +$from..+${to}s: boundary ${got[i].start}'); + checked++; + } + // +777..+2111 has two boundaries in the pre-bump labels too. + expect(checked, greaterThan(1), reason: 'session +$from..+${to}s'); + expect(show(got), + show(preBumpStageSessionCardio(t0 + from, t0 + to, s.grav, s.hrTs, s.rrTsSec)), + reason: 'session +$from..+${to}s equals the pre-bump labels'); + } + }); + + test('the hypnogram\'s boundaries sit on the 30 s grid of the first sample', () { + for (final offsetMs in [13000, 21000]) { + final s = _Input.make(_night(), t0Ms: _alignedT0 + offsetMs, seed: 31); + final st = batch(s); + final stages = st.stages; + expect(stages.first.start, st.start); + expect(stages.last.end, st.end); + for (var i = 1; i < stages.length; i++) { + expect(stages[i].start, stages[i - 1].end, reason: 'tiles without holes'); + } + var checked = 0; + for (var i = 1; i < stages.length; i++) { + final edge = stages[i].start; + if (stages[i].stage == 'unobserved' || stages[i - 1].stage == 'unobserved') { + continue; + } + expect((edge - s.t0Ms ~/ 1000) % 30, 0, + reason: 'boundary at $edge (${stages[i - 1].stage} -> ' + '${stages[i].stage}), data starts at +${offsetMs ~/ 1000}s'); + checked++; + } + expect(checked, greaterThan(2), reason: 'the night must change stage'); + } + }); + + test('a hole shorter than the carry-forward does not move the grid', () { + final s = _Input.make(_night(), + t0Ms: _alignedT0 + 13000, gaps: [(1000, 1040)], seed: 33); + final stages = batch(s).stages; + for (var i = 1; i < stages.length; i++) { + if (stages[i].stage == 'unobserved' || stages[i - 1].stage == 'unobserved') { + continue; + } + expect((stages[i].start - s.t0Ms ~/ 1000) % 30, 0, + reason: 'boundary ${stages[i].start}'); + } + }); + + test('a hole longer than the carry-forward re-anchors the grid after it', () { + // 1100 % 30 = 20: the second run's grid is 20 s out of phase with the first + final s = _Input.make(_night(), + t0Ms: _alignedT0 + 13000, gaps: [(1000, 1100)], seed: 35); + final t0 = s.t0Ms ~/ 1000; + final stages = batch(s).stages; + var before = 0, after = 0; + for (var i = 1; i < stages.length; i++) { + if (stages[i].stage == 'unobserved' || stages[i - 1].stage == 'unobserved') { + continue; + } + final edge = stages[i].start - t0; + if (edge < 1000) { + expect(edge % 30, 0, reason: 'first run, boundary at +$edge'); + before++; + } else { + expect((edge - 1100) % 30, 0, reason: 'second run, boundary at +$edge'); + after++; + } + } + expect(before, greaterThan(0)); + expect(after, greaterThan(0)); + }); + }); +} diff --git a/test/onehz/fixtures/real_night_2026_07_hypnogram_v1.csv b/test/onehz/fixtures/real_night_2026_07_hypnogram_v1.csv new file mode 100644 index 00000000..6253fe1c --- /dev/null +++ b/test/onehz/fixtures/real_night_2026_07_hypnogram_v1.csv @@ -0,0 +1,57 @@ +rel_start_s,rel_end_s,stage +0,1980,light +1980,2670,rem +2670,5760,light +5760,6210,deep +6210,6240,light +6240,6540,deep +6540,7050,light +7050,7260,deep +7260,8130,light +8130,8370,deep +8370,8880,light +8880,9630,rem +9630,10140,light +10140,10350,deep +10350,13260,light +13260,14070,deep +14070,14160,light +14160,14850,deep +14850,15060,light +15060,15360,rem +15360,15420,wake +15420,15510,rem +15510,15630,wake +15630,15690,rem +15690,15780,wake +15780,16680,rem +16680,17220,light +17220,17520,rem +17520,17820,light +17820,18600,rem +18600,18900,light +18900,19500,rem +19500,21480,light +21480,22110,deep +22110,22680,light +22680,23880,rem +23880,23940,wake +23940,24060,light +24060,24180,wake +24180,24360,rem +24360,24510,wake +24510,24960,rem +24960,25050,wake +25050,25170,rem +25170,25230,wake +25230,25320,rem +25320,25470,wake +25470,26820,light +26820,27150,wake +27150,28260,light +28260,28290,wake +28290,29910,light +29910,31050,rem +31050,31290,wake +31290,31980,rem +31980,32041,wake diff --git a/test/onehz/sleep_honesty_test.dart b/test/onehz/sleep_honesty_test.dart index 2ac024c6..42a0335e 100644 --- a/test/onehz/sleep_honesty_test.dart +++ b/test/onehz/sleep_honesty_test.dart @@ -258,6 +258,24 @@ void main() { } final s = AdvancedSleepStager.stageWindow(_t0, _t0 + span, grav, hr, rr: rr); + final cache = CardioRowCache(); + final cold = AdvancedSleepStager.stageWindow(_t0, _t0 + span, grav, hr, + rr: rr, rowCache: cache); + final warm = AdvancedSleepStager.stageWindow(_t0, _t0 + span, grav, hr, + rr: rr, rowCache: cache); + expect(cache.hits, greaterThan(0)); + final expected = [for (final seg in s.stages) seg.toJson()]; + expect([for (final seg in cold.stages) seg.toJson()], expected); + expect([for (final seg in warm.stages) seg.toJson()], expected); + final rows = EpochRows(anchorMs: _t0 * 1000); + rows.fold( + hr: [for (final h in hr) HrSample(h.ts * 1000.0, h.bpm)], + accel: [for (final g in grav) AccelSample(g.ts * 1000.0, g.x, g.y, g.z)], + rrMs: [for (final r in rr) r.rrMs], + rrTsMs: [for (final r in rr) r.ts * 1000.0], + throughMs: (_t0 + span) * 1000 - 1); + expect([for (final seg in rows.stageWindow(_t0, _t0 + span)) seg.toJson()], + expected); final inBlock = {}; for (final seg in s.stages) { final a = math.max(seg.start, wS), b = math.min(seg.end, wE); diff --git a/test/onehz/stage_minute_ranges_test.dart b/test/onehz/stage_minute_ranges_test.dart new file mode 100644 index 00000000..31b842ca --- /dev/null +++ b/test/onehz/stage_minute_ranges_test.dart @@ -0,0 +1,298 @@ +// SLEEP — per-stage minute RANGES (D27 revised, 2026-10-19). RED: new API. +// +// The cardio stager has no per-epoch probabilities, only robust-z scores +// against two fixed cutoffs plus a night confidence from RR coverage. So a +// stage's minutes get a BOUND, not a statistical interval: +// +// lo[s] = epochs that can only be s: covered, labelled s, and every score +// that decided the label clear of its cutoff by >= margin; +// hi[s] = lo[s] + epochs that could still be s: +// • uncovered epochs (no usable HR in the epoch, or no stage label at +// all) could be ANY stage; +// • an asleep epoch whose score is within `margin` of a cutoff counts +// toward both neighbouring stages (remScore ↔ REM / its NREM +// substage; deepScore ↔ light / deep). +// nrem = light ∪ deep (an epoch counts toward nrem.lo when every stage it +// could be is light or deep). +// +// API chosen here (lib/src/onehz/sleep/stage_ranges.dart, exported through +// `package:openstrap_analytics/onehz.dart`): +// +// class StageEpochEvidence { +// const StageEpochEvidence({SleepStage? stage, bool deep = false, +// bool covered = true, double? remScore, double? deepScore}); +// } +// class MinuteRange { final double lo, hi; } +// class StageMinuteRanges { +// final MinuteRange wake, light, deep, rem, nrem; +// final double windowMin; +// Map> toJson(); // wake/light/deep/rem → {lo,hi} +// } +// const double kStageScoreMargin; // calibrated, > 0 +// StageMinuteRanges? stageMinuteRanges(List? epochs, +// {int epochSec = 30, double remScoreCut = kDefaultRemScoreCut, +// double deepScoreCut = kDefaultDeepScoreCut, +// double margin = kStageScoreMargin}); +// extension on CardioStagerResult: List get epochEvidence +// — one entry per base.stages epoch, stage/deep equal to the FINAL labels, +// covered == the epoch carried at least one valid HR sample. +// +// "Point" minutes below are the stager's own labelled minutes (what the edge +// already shows); the range never moves the point, it brackets it. + +import 'dart:math' as math; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +const int _t0 = 1700000000; + +StageEpochEvidence _w() => const StageEpochEvidence(stage: SleepStage.wake); +StageEpochEvidence _l() => const StageEpochEvidence( + stage: SleepStage.nrem, remScore: -1.5, deepScore: -1.5); +StageEpochEvidence _d() => const StageEpochEvidence( + stage: SleepStage.nrem, deep: true, remScore: -1.5, deepScore: 1.5); +StageEpochEvidence _r() => const StageEpochEvidence( + stage: SleepStage.rem, remScore: 2.0, deepScore: -1.5); +StageEpochEvidence _u() => + const StageEpochEvidence(stage: SleepStage.nrem, covered: false); + +void _expectRange(MinuteRange r, double lo, double hi, String what) { + expect(r.lo, closeTo(lo, 1e-9), reason: '$what.lo'); + expect(r.hi, closeTo(hi, 1e-9), reason: '$what.hi'); +} + +void main() { + test('the margin is a real, positive number', () { + expect(kStageScoreMargin, greaterThan(0)); + expect(kStageScoreMargin.isFinite, isTrue); + }); + + group('absent input abstains', () { + test('null ⇒ null', () => expect(stageMinuteRanges(null), isNull)); + test('empty ⇒ null', () => expect(stageMinuteRanges(const []), isNull)); + }); + + test('every epoch covered and clear ⇒ lo == hi == point for every stage', () { + // 20 wake, 60 light, 30 deep, 40 rem epochs = 10/30/15/20 min. + final ev = [ + for (var i = 0; i < 20; i++) _w(), + for (var i = 0; i < 60; i++) _l(), + for (var i = 0; i < 30; i++) _d(), + for (var i = 0; i < 40; i++) _r(), + ]; + final r = stageMinuteRanges(ev, margin: 0.2)!; + _expectRange(r.wake, 10, 10, 'wake'); + _expectRange(r.light, 30, 30, 'light'); + _expectRange(r.deep, 15, 15, 'deep'); + _expectRange(r.rem, 20, 20, 'rem'); + _expectRange(r.nrem, 45, 45, 'nrem'); + expect(r.windowMin, closeTo(75, 1e-9)); + }); + + test('nothing covered ⇒ every stage is [0, window]', () { + final ev = [for (var i = 0; i < 120; i++) _u()]; + final r = stageMinuteRanges(ev)!; + for (final (name, range) in [ + ('wake', r.wake), + ('light', r.light), + ('deep', r.deep), + ('rem', r.rem), + ('nrem', r.nrem), + ]) { + _expectRange(range, 0, 60, name); + } + }); + + test('an epoch with no stage label at all is uncovered, not dropped', () { + final ev = [ + for (var i = 0; i < 10; i++) _l(), + for (var i = 0; i < 10; i++) const StageEpochEvidence(), + ]; + final r = stageMinuteRanges(ev, margin: 0.2)!; + expect(r.windowMin, closeTo(10, 1e-9)); + _expectRange(r.light, 5, 10, 'light'); + _expectRange(r.wake, 0, 5, 'wake'); + _expectRange(r.rem, 0, 5, 'rem'); + }); + + test('uncovered epochs widen every stage by their minutes', () { + // 40 covered light + 20 uncovered (stager fell through to NREM there). + final ev = [ + for (var i = 0; i < 40; i++) _l(), + for (var i = 0; i < 20; i++) _u(), + ]; + final r = stageMinuteRanges(ev, margin: 0.2)!; + _expectRange(r.light, 20, 30, 'light'); + _expectRange(r.wake, 0, 10, 'wake'); + _expectRange(r.deep, 0, 10, 'deep'); + _expectRange(r.rem, 0, 10, 'rem'); + _expectRange(r.nrem, 20, 30, 'nrem'); + }); + + test('a REM score inside the margin counts toward REM and its NREM neighbour', + () { + const cut = kDefaultRemScoreCut; + final ev = [ + for (var i = 0; i < 10; i++) _l(), + // labelled REM, but only 0.05 above the cutoff (margin 0.2) + for (var i = 0; i < 10; i++) + const StageEpochEvidence( + stage: SleepStage.rem, remScore: cut + 0.05, deepScore: -1.5), + // labelled light, 0.05 below the cutoff + for (var i = 0; i < 10; i++) + const StageEpochEvidence( + stage: SleepStage.nrem, remScore: cut - 0.05, deepScore: -1.5), + ]; + final r = stageMinuteRanges(ev, margin: 0.2)!; + _expectRange(r.rem, 0, 10, 'rem'); + _expectRange(r.light, 5, 15, 'light'); + _expectRange(r.deep, 0, 0, 'deep'); + _expectRange(r.wake, 0, 0, 'wake'); + }); + + test('a deep score inside the margin counts toward light and deep', () { + const cut = kDefaultDeepScoreCut; + final ev = [ + for (var i = 0; i < 10; i++) + const StageEpochEvidence( + stage: SleepStage.nrem, + deep: true, + remScore: -1.5, + deepScore: cut + 0.05), + for (var i = 0; i < 10; i++) _d(), + ]; + final r = stageMinuteRanges(ev, margin: 0.2)!; + _expectRange(r.deep, 5, 10, 'deep'); + _expectRange(r.light, 0, 5, 'light'); + // An epoch that is light-or-deep is certainly NREM. + _expectRange(r.nrem, 10, 10, 'nrem'); + }); + + // A score that is missing decided nothing, so the decision it would have made + // stays open: it is never read as "clear of the cutoff". + group('a missing score is undecided', () { + const m = {'null': null, 'NaN': double.nan}; + for (final e in m.entries) { + final missing = e.value; + test('${e.key} on both scores of an asleep epoch', () { + final r = stageMinuteRanges([ + StageEpochEvidence( + stage: SleepStage.nrem, remScore: missing, deepScore: missing) + ])!; + _expectRange(r.rem, 0, 0.5, 'rem'); + _expectRange(r.light, 0, 0.5, 'light'); + _expectRange(r.deep, 0, 0.5, 'deep'); + _expectRange(r.wake, 0, 0, 'wake'); + _expectRange(r.nrem, 0, 0.5, 'nrem'); + }); + test('${e.key} on the REM score only: REM stays open, substage is known', + () { + final r = stageMinuteRanges([ + StageEpochEvidence( + stage: SleepStage.nrem, remScore: missing, deepScore: -1.5) + ])!; + _expectRange(r.rem, 0, 0.5, 'rem'); + _expectRange(r.light, 0, 0.5, 'light'); + _expectRange(r.deep, 0, 0, 'deep'); + }); + test('${e.key} on the deep score only: light/deep stays open, REM is out', + () { + final r = stageMinuteRanges([ + StageEpochEvidence( + stage: SleepStage.nrem, remScore: -1.5, deepScore: missing) + ])!; + _expectRange(r.rem, 0, 0, 'rem'); + _expectRange(r.light, 0, 0.5, 'light'); + _expectRange(r.deep, 0, 0.5, 'deep'); + _expectRange(r.nrem, 0.5, 0.5, 'nrem'); + }); + test('${e.key} on a REM epoch\'s scores', () { + final r = stageMinuteRanges([ + StageEpochEvidence( + stage: SleepStage.rem, remScore: missing, deepScore: missing) + ])!; + _expectRange(r.rem, 0, 0.5, 'rem'); + _expectRange(r.light, 0, 0.5, 'light'); + _expectRange(r.deep, 0, 0.5, 'deep'); + }); + } + }); + + test('scores exactly at the margin are clear', () { + const cut = kDefaultRemScoreCut; + final ev = [ + for (var i = 0; i < 10; i++) + const StageEpochEvidence( + stage: SleepStage.rem, remScore: cut + 0.25, deepScore: -1.5), + ]; + final r = stageMinuteRanges(ev, margin: 0.25)!; + _expectRange(r.rem, 5, 5, 'rem'); + }); + + test('epochSec scales minutes', () { + final ev = [for (var i = 0; i < 12; i++) _w()]; + final r = stageMinuteRanges(ev, epochSec: 60)!; + _expectRange(r.wake, 12, 12, 'wake'); + expect(r.windowMin, closeTo(12, 1e-9)); + }); + + test('toJson carries the four display stages as {lo, hi}', () { + final r = stageMinuteRanges([_w(), _l(), _d(), _r(), _u()], margin: 0.2)!; + final j = r.toJson(); + expect(j.keys.toSet(), {'wake', 'light', 'deep', 'rem'}); + for (final v in j.values) { + expect(v.keys.toSet(), {'lo', 'hi'}); + expect(v['lo']!, lessThanOrEqualTo(v['hi']!)); + } + expect(j['wake'], {'lo': 0.5, 'hi': 1.0}); + }); + + group('CardioStagerResult.epochEvidence', () { + // 3 h still night at a steady 55 bpm with a 30-min off-skin gap in the + // middle (HR 0), RR absent: the gap's epochs must come out uncovered. + final n = 3 * 3600; + final accel = [ + for (var i = 0; i < n; i++) + AccelSample((_t0 + i) * 1000.0, 0.0, 0.0, 1.0) + ]; + final rng = math.Random(7); + final hr = [ + for (var i = 0; i < n; i++) + (i >= 3600 && i < 5400) ? 0.0 : 55 + rng.nextDouble() * 4 + ]; + final res = cardioStager(hr, accel); + + test('one entry per staged epoch, labels equal the final stages', () { + expect(res.base.stages, isNotEmpty); + final ev = res.epochEvidence; + expect(ev.length, res.base.stages.length); + for (var e = 0; e < ev.length; e++) { + expect(ev[e].stage, res.base.stages[e], reason: 'epoch $e stage'); + expect(ev[e].deep, res.deepFlag[e], reason: 'epoch $e deep'); + } + }); + + test('epochs with no valid HR are uncovered, the rest covered', () { + final ev = res.epochEvidence; + for (var e = 0; e < ev.length; e++) { + final inGap = e * 30 >= 3600 && (e + 1) * 30 <= 5400; + expect(ev[e].covered, !inGap, reason: 'epoch $e'); + } + }); + + test('the gap widens every stage range by its 30 minutes', () { + final r = stageMinuteRanges(res.epochEvidence)!; + expect(r.windowMin, closeTo(res.base.stages.length * 0.5, 1e-9)); + for (final range in [r.wake, r.light, r.deep, r.rem]) { + expect(range.hi - range.lo, greaterThanOrEqualTo(30 - 1e-9)); + } + }); + + test('an abstaining stager yields no evidence and therefore no range', () { + final off = cardioStager(List.filled(n, 0.0), accel); + expect(off.epochEvidence, isEmpty); + expect(stageMinuteRanges(off.epochEvidence), isNull); + }); + }); +} diff --git a/test/onehz/support/causal_night.dart b/test/onehz/support/causal_night.dart new file mode 100644 index 00000000..2c6486d6 --- /dev/null +++ b/test/onehz/support/causal_night.dart @@ -0,0 +1,132 @@ +// Deterministic synthetic 1 Hz nights for the causal-stager tests. +// +// Not a physiological simulator — just three separable regimes so the tests can +// check that the causal stager sees what is there: +// nrem : low, steady HR; quiet RR; still wrist +// rem : slightly higher HR, jittery instantaneous HR, wide RR spread; still +// wake : high HR; bursts of wrist motion +// RR beats carry the SAME second-quantised timestamps the device produces +// (`rr_ts_ms = rec_ts * 1000`, so several beats tie on one second). + +import 'dart:math' as math; +import 'package:openstrap_analytics/onehz.dart'; + +/// 30-s-aligned absolute start (1_700_000_010_000 / 30_000 is an integer). +const double kT0Ms = 1700000010000; + +class SynthNight { + final double t0Ms; + final List hr; + final List accel; + final List rrTsMs; + final List rrMs; + + /// Ground-truth regime per second ('nrem' | 'rem' | 'wake'). + final List truth; + const SynthNight( + this.t0Ms, this.hr, this.accel, this.rrTsMs, this.rrMs, this.truth); + int get seconds => truth.length; +} + +class _Rng { + int _s; + _Rng(int seed) : _s = seed == 0 ? 0x9E3779B9 : seed; + double next() { + _s ^= (_s << 13) & 0xFFFFFFFF; + _s ^= _s >> 17; + _s ^= (_s << 5) & 0xFFFFFFFF; + _s &= 0xFFFFFFFF; + return _s / 4294967296.0; + } + + double gauss() { + var a = 0.0; + for (var i = 0; i < 12; i++) { + a += next(); + } + return a - 6.0; + } +} + +SynthNight synthNight(List<(String, int)> segments, + {int seed = 11, double t0Ms = kT0Ms}) { + final rng = _Rng(seed); + final hr = []; + final accel = []; + final rrTs = []; + final rr = []; + final truth = []; + var beatClockMs = 0.0; // ms since t0, the next beat's time + var sec = 0; + for (final (kind, len) in segments) { + for (var k = 0; k < len; k++, sec++) { + final base = switch (kind) { 'rem' => 59.0, 'wake' => 80.0, _ => 54.0 }; + final hrSd = switch (kind) { 'rem' => 3.2, 'wake' => 2.0, _ => 0.9 }; + final rrSd = switch (kind) { 'rem' => 55.0, 'wake' => 25.0, _ => 14.0 }; + final bpm = (base + hrSd * rng.gauss()).roundToDouble(); + final ts = t0Ms + sec * 1000.0; + hr.add(HrSample(ts, bpm)); + // posture drifts every 20 min (a static change, not motion) + final posture = (sec ~/ 1200) % 3; + final g = switch (posture) { + 0 => const [0.30, 0.80, 0.50], + 1 => const [0.70, 0.20, 0.68], + _ => const [-0.20, 0.55, 0.81], + }; + final gn = math.sqrt(g[0] * g[0] + g[1] * g[1] + g[2] * g[2]); + var burst = 0.0; + if (kind == 'wake' && rng.next() < 0.35) burst = 0.25 + 0.2 * rng.next(); + accel.add(AccelSample( + ts, + g[0] / gn + 0.0015 * rng.gauss() + burst, + g[1] / gn + 0.0015 * rng.gauss(), + g[2] / gn + 0.0015 * rng.gauss(), + )); + // beats whose start falls inside this second + while (beatClockMs < (sec + 1) * 1000.0) { + final inst = 60000.0 / math.max(35.0, bpm); + final v = (inst + rrSd * rng.gauss()).clamp(380.0, 1900.0).toDouble(); + if (beatClockMs >= sec * 1000.0) { + rrTs.add(ts); + rr.add(v.roundToDouble()); + } + beatClockMs += v; + } + truth.add(kind); + } + } + return SynthNight(t0Ms, hr, accel, rrTs, rr, truth); +} + +/// A window of the night: samples with `lo <= second < hi` (seconds since t0), +/// decision time `nowSec`. `hi` may exceed `nowSec` to carry FUTURE samples. +CausalSampleWindow windowOf(SynthNight n, int loSec, int hiSec, + {int? nowSec, bool hrOn = true, bool accelOn = true, bool rrOn = true}) { + final hiC = math.min(hiSec, n.seconds); + final loC = math.max(0, loSec); + final loMs = n.t0Ms + loC * 1000.0, hiMs = n.t0Ms + hiC * 1000.0; + final rrIdx = [ + for (var i = 0; i < n.rrTsMs.length; i++) + if (n.rrTsMs[i] >= loMs && n.rrTsMs[i] < hiMs) i + ]; + return CausalSampleWindow( + nowMs: n.t0Ms + (nowSec ?? hiSec) * 1000.0, + hr: hrOn ? n.hr.sublist(loC, hiC) : const [], + accel: accelOn ? n.accel.sublist(loC, hiC) : const [], + rr: rrOn + ? RrSeries([for (final i in rrIdx) n.rrTsMs[i]], + [for (final i in rrIdx) n.rrMs[i]]) + : RrSeries(const [], const []), + ); +} + +/// A reasonably realistic ~3 h night used by most tests. +List<(String, int)> standardNight() => [ + ('nrem', 35 * 60), + ('rem', 12 * 60), + ('nrem', 40 * 60), + ('wake', 6 * 60), + ('nrem', 30 * 60), + ('rem', 15 * 60), + ('nrem', 25 * 60), + ]; diff --git a/test/onehz/support/epoch_input.dart b/test/onehz/support/epoch_input.dart new file mode 100644 index 00000000..f97e36a2 --- /dev/null +++ b/test/onehz/support/epoch_input.dart @@ -0,0 +1,105 @@ +// Shared fixtures for the option-2 (onset-anchored, phase-keyed) EpochRows +// tests: a synthetic night as raw samples, optionally with holes. +import 'dart:convert'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../../support/property.dart'; +import 'causal_night.dart'; + +const int kAlignedT0Ms = 1700000010000; // a multiple of 30,000 + +class EpochInput { + final int t0Ms, seconds; + final List hr; + final List accel; + final List rrMs, rrTs; + EpochInput(this.t0Ms, this.seconds, this.hr, this.accel, this.rrMs, this.rrTs); + + /// Both streams lose the same seconds [gaps] (seconds since t0), and the + /// beats in them. + factory EpochInput.make(List<(String, int)> segs, + {int seed = 11, int t0Ms = kAlignedT0Ms, List<(int, int)> gaps = const []}) { + final n = synthNight(segs, seed: seed, t0Ms: t0Ms.toDouble()); + bool inGap(double ts) => gaps + .any((g) => ts >= t0Ms + g.$1 * 1000.0 && ts < t0Ms + g.$2 * 1000.0); + final rr = [], rt = []; + for (var i = 0; i < n.rrMs.length; i++) { + if (!inGap(n.rrTsMs[i])) { + rr.add(n.rrMs[i]); + rt.add(n.rrTsMs[i]); + } + } + return EpochInput(t0Ms, n.seconds, [for (final h in n.hr) if (!inGap(h.tsMs)) h], + [for (final a in n.accel) if (!inGap(a.tsMs)) a], rr, rt); + } + + /// Samples with `loSec <= second < hiSec` (seconds since t0), as a new input + /// (same t0; `seconds` becomes hiSec - loSec only for bookkeeping). + EpochInput between(int loSec, int hiSec) { + final lo = t0Ms + loSec * 1000.0, hi = t0Ms + hiSec * 1000.0; + bool inside(double t) => t >= lo && t < hi; + final ri = [ + for (var i = 0; i < rrTs.length; i++) + if (inside(rrTs[i])) i + ]; + return EpochInput( + t0Ms, + seconds, + [for (final h in hr) if (inside(h.tsMs)) h], + [for (final a in accel) if (inside(a.tsMs)) a], + [for (final i in ri) rrMs[i]], + [for (final i in ri) rrTs[i]]); + } + + void foldInto(EpochRows er, int loSec, int hiSec) { + final p = between(loSec, hiSec); + er.fold(hr: p.hr, accel: p.accel, rrMs: p.rrMs, rrTsMs: p.rrTs); + } + + int get sampleCount => hr.length + accel.length + rrMs.length; + + List get grav => + [for (final a in accel) GravTs(a.tsMs ~/ 1000, a.x, a.y, a.z)]; + List get hrTs => [for (final h in hr) HrTs(h.tsMs ~/ 1000, h.hr)]; + List get rrTsSec => + [for (var i = 0; i < rrMs.length; i++) RrTs(rrTs[i] ~/ 1000, rrMs[i])]; +} + +List<(String, int)> nightSegs() => [ + ('nrem', 900), ('rem', 600), ('nrem', 900), ('wake', 300), ('nrem', 300), + ]; + +List randomCuts(Rng r, int seconds) { + const sizes = [1, 2, 7, 30, 31, 59, 100, 171, 500]; + final out = []; + var at = 0; + while (true) { + at += r.nextBool(.15) ? 0 : 1 + r.nextInt(sizes[r.nextInt(sizes.length)]); + if (at >= seconds) return out; + out.add(at); + } +} + +bool sameRow(EpochFeatureRow a, EpochFeatureRow b) => + a.startMs == b.startMs && + [ + [a.motion, b.motion], [a.hr, b.hr], [a.hrSd, b.hrSd], + [a.rmssd, b.rmssd], [a.lfhf, b.lfhf], [a.rk, b.rk], [a.sdnn, b.sdnn], + ].every((p) => p[0].compareTo(p[1]) == 0); + +void expectSameRows( + List got, List want, String why) { + expect(got.length, want.length, reason: '$why: row count'); + for (var i = 0; i < got.length; i++) { + if (!sameRow(got[i], want[i])) { + fail('$why: row $i differs: got start ${got[i].startMs} hr ${got[i].hr} ' + 'motion ${got[i].motion}; want start ${want[i].startMs} ' + 'hr ${want[i].hr} motion ${want[i].motion}'); + } + } +} + +String showStages(List l) => + jsonEncode([for (final g in l) g.toJson()]); diff --git a/test/onehz/support/pre_bump_stager.dart b/test/onehz/support/pre_bump_stager.dart new file mode 100644 index 00000000..7a502abf --- /dev/null +++ b/test/onehz/support/pre_bump_stager.dart @@ -0,0 +1,154 @@ +// Target upstream main 9db83ce default staging path, frozen as a test oracle. +// The former pre-bump oracle now uses main's pooled night baselines, per-run +// coverage gate and run-boundary continuity. Only private names are qualified. +import 'package:openstrap_analytics/onehz.dart'; + +List preBumpStageSessionCardio(int start, int end, + List grav, List hr, List rr) { + final span = end - start; + if (span <= 0) return const []; + final epSec = AdvancedSleepStager.epochS.round(); + final minStageableSec = 3 * epSec; + if (span < minStageableSec) return [StageSegment(start, end, 'unobserved')]; + + // Invalid samples are NOT samples: they must not seed `usable`, and they + // must not become the source of a carry-forward. A second with only an + // undecoded vector falls through to the same unstaged→unobserved path as a + // second with no row at all (cardioStager itself never reads + // [AccelSample.valid], so it can only be protected here). + final gByTs = { + for (final g in grav) + if (g.valid && g.ts >= start && g.ts < end) g.ts: g + }; + final hByTs = {for (final h in hr) if (h.ts >= start && h.ts < end) h.ts: h}; + final accel = List.filled( + span, AccelSample(start * 1000.0, 0, 0, 1.0)); + final hr1hz = List.filled(span, 0.0); + // usable[i] — second i has a real accel sample or a BOUNDED carry-forward. + final usable = List.filled(span, false); + var lastRealIdx = -1; + for (var i = 0; i < span; i++) { + final ts = start + i; + final g = gByTs[ts]; + if (g != null) { + accel[i] = AccelSample(ts * 1000.0, g.x, g.y, g.z); + lastRealIdx = i; + usable[i] = true; + } else if (lastRealIdx >= 0 && (i - lastRealIdx) <= AdvancedSleepStager.maxAccelCarryForwardSec) { + // Bounded carry-forward — see [maxAccelCarryForwardSec]. The TIMESTAMP + // is this second's, not the stale sample's: cardioStager centres its RR + // windows on `accel[mid].tsMs`, so copying the old sample wholesale + // mis-centred every RMSSD/LF-HF window inside a gap. + final p = accel[i - 1]; + accel[i] = AccelSample(ts * 1000.0, p.x, p.y, p.z); + usable[i] = true; + } + hr1hz[i] = hByTs[ts]?.bpm ?? 0.0; // 0 = off-skin, cardioStager's own contract. + } + + final rSeg = [for (final r in rr) if (r.ts >= start && r.ts < end) r]; + final rrMs = [for (final r in rSeg) r.rrMs]; + final rrTsMs = [for (final r in rSeg) r.ts * 1000.0]; + + // Everything not staged below stays 'unobserved' — no stage, no wake. + final perSec = List.filled(span, 'unobserved'); + // Stageable runs. All but the last are trimmed to whole epochs so every run + // boundary sits on the epoch grid; cardioStager scores no sub-epoch tail + // anyway, and the tail takes the run's last label below. + final runs = <({int from, int to, int nEp})>[]; + var i = 0; + while (i < span) { + if (!usable[i]) { + i++; + continue; + } + var j = i; + while (j < span && usable[j]) { + j++; + } + if ((j - i) >= minStageableSec) { + final nEp = (j - i) ~/ epSec; + // cardioStager's HR-coverage abstention, applied per run as it was + // when each run was staged alone: pooled with well-covered runs, a run + // with no HR would pass the window-wide gate and fall through to NREM. + var withHr = 0; + for (var e = 0; e < nEp; e++) { + final lo = i + e * epSec; + for (var k = lo; k < lo + epSec; k++) { + if (hr1hz[k] > 0) { + withHr++; + break; + } + } + } + if (withHr >= minHrCoverage * nEp) { + runs.add((from: i, to: j, nEp: nEp)); + } + } + i = j; + } + + if (runs.isNotEmpty) { + final hrCat = []; + final accelCat = []; + final runStarts = []; + for (var ri = 0; ri < runs.length; ri++) { + final r = runs[ri]; + runStarts.add(hrCat.length); + // The last run keeps its tail (no boundary follows it), so a window + // with a single run is staged exactly as before. + final stop = ri == runs.length - 1 ? r.to : r.from + r.nEp * epSec; + hrCat.addAll(hr1hz.sublist(r.from, stop)); + accelCat.addAll(accel.sublist(r.from, stop)); + } + final result = cardioStager( + hrCat, + accelCat, + rrMs: rrMs, + rrTsMs: rrTsMs, + runStarts: runStarts, + ); + final stages = result.base.stages; + // Empty = cardioStager abstained: everything stays 'unobserved'. + if (stages.isNotEmpty) { + var g = 0; // epoch index into the concatenated result + for (final r in runs) { + for (var e = 0; e < r.nEp; e++, g++) { + if (g >= stages.length) break; + final label = switch (stages[g]) { + SleepStage.wake => 'wake', + SleepStage.rem => 'rem', + SleepStage.nrem => + (g < result.deepFlag.length && result.deepFlag[g]) + ? 'deep' + : 'light', + }; + final lo = r.from + e * epSec; + // The last epoch absorbs the run's sub-epoch remainder, exactly as + // [_buildSegments] extends the final segment to the window end. + final hi = e == r.nEp - 1 ? r.to : lo + epSec; + for (var k = lo; k < hi; k++) { + perSec[k] = label; + } + } + } + } + } + + // Coalesce the per-second labels into contiguous [StageSegment]s tiling + // [start, end) — same segment semantics [_buildSegments] produces. + final segments = []; + var k = 0; + while (k < span) { + var m = k; + while (m < span && perSec[m] == perSec[k]) { + m++; + } + segments.add(StageSegment(start + k, start + m, perSec[k])); + k = m; + } + return segments; + } + +String showSegments(List l) => + [for (final g in l) '${g.start}-${g.end}:${g.stage}'].join(' '); diff --git a/test/onehz/wallclock_agreement_test.dart b/test/onehz/wallclock_agreement_test.dart new file mode 100644 index 00000000..53d120c1 --- /dev/null +++ b/test/onehz/wallclock_agreement_test.dart @@ -0,0 +1,227 @@ +// GUARD: the gap-anchored grid against today's labels on the checked-in real +// night (combined bump, chunk B3 + B4; owner decision 2026-10-22). +// +// * `fixtures/real_night_2026_07_hypnogram_v1.csv` is today's hypnogram for +// that night (AdvancedSleepStager.stageWindow at analytics 83fc308, the +// run-start-anchored grid), computed for a window starting at the +// UNALIGNED second 1700000013, which is also the night's first sample. It is +// frozen: regenerate it only as a deliberate decision. +// * AGREEMENT GUARD (B4): per 30 s epoch of that grid (steps of 30 s from the +// first sample), the new label equals the frozen one at least +// [kMinAgreement] = 0.97 of the time (both staged; the epoch's midpoint +// second is looked up). Anchoring at the first sample of each gap-free run +// is the old grid for a window that starts at the data, so this is a near +// identity; the wall-clock grid it replaced scored 0.905 here. +// * The harness (tool/stager_harness.dart) cannot stand in for this: it +// scores the decision layer on an external PSG corpus (DREAMT feature +// export) that is not in the repo, and it has no grid-comparison mode. The +// skipped test at the end says what that needs. +import 'dart:convert'; +import 'dart:io'; + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import 'support/pre_bump_stager.dart'; + +const kMinAgreement = 0.97; +const _base = 1700000013; // unaligned: 1700000013 % 30 == 3 + +class _Night { + final List hr = []; + final List accel = []; + final List rrMs = [], rrTs = []; + int seconds = 0; + List get grav => + [for (final a in accel) GravTs(a.tsMs ~/ 1000, a.x, a.y, a.z)]; + List get hrTs => [for (final h in hr) HrTs(h.tsMs ~/ 1000, h.hr)]; + List get rrSec => + [for (var i = 0; i < rrMs.length; i++) RrTs(rrTs[i] ~/ 1000, rrMs[i])]; +} + +_Night? _load() { + final a = File('test/onehz/fixtures/real_night_2026_07_onehz.csv'); + final b = File('test/onehz/fixtures/real_night_2026_07_rr.csv'); + if (!a.existsSync() || !b.existsSync()) return null; + final n = _Night(); + for (final l in a.readAsLinesSync().skip(1)) { + if (l.trim().isEmpty) continue; + final p = l.split(','); + final s = int.parse(p[0]); + final ts = (_base + s) * 1000.0; + n.seconds = s + 1; + n.hr.add(HrSample(ts, double.parse(p[1]))); + n.accel.add(AccelSample( + ts, double.parse(p[2]), double.parse(p[3]), double.parse(p[4]))); + } + for (final l in b.readAsLinesSync().skip(1)) { + if (l.trim().isEmpty) continue; + final p = l.split(','); + n.rrTs.add(_base * 1000.0 + double.parse(p[0])); + n.rrMs.add(double.parse(p[1])); + } + return n; +} + +List<(int, int, String)> _golden() => [ + for (final l in File('test/onehz/fixtures/real_night_2026_07_hypnogram_v1.csv') + .readAsLinesSync() + .skip(1)) + if (l.trim().isNotEmpty) + ( + int.parse(l.split(',')[0]), + int.parse(l.split(',')[1]), + l.split(',')[2] + ) + ]; + +String? _at(List<(int, int, String)> segs, int rel) { + for (final s in segs) { + if (rel >= s.$1 && rel < s.$2) return s.$3; + } + return null; +} + +void main() { + final night = _load(); + + test('fixtures are present', () { + expect(night, isNotNull); + expect(night!.seconds, greaterThan(32000)); + expect(_golden().length, greaterThan(10)); + expect(_base % 30, isNot(0)); + }); + + final nightOrNull = night; + if (nightOrNull == null) return; + + SleepSession stageBatch() => AdvancedSleepStager.stageWindow( + _base, _base + nightOrNull.seconds, nightOrNull.grav, nightOrNull.hrTs, + rr: nightOrNull.rrSec); + + late final SleepSession now = stageBatch(); + + test('per-epoch agreement with today\'s labels is at least $kMinAgreement', () { + final old = _golden(); + final stages = [ + for (final g in now.stages) (g.start - _base, g.end - _base, g.stage) + ]; + var epochs = 0, compared = 0, agree = 0; + for (var start = _base; start + 30 <= _base + nightOrNull.seconds; start += 30) { + epochs++; + final mid = start + 15 - _base; + final a = _at(old, mid), b = _at(stages, mid); + if (a == null || b == null || a == 'unobserved' || b == 'unobserved') continue; + compared++; + if (a == b) agree++; + } + expect(compared, greaterThan(epochs * 0.9), + reason: 'most of the night must be staged under both grids'); + final agreement = agree / compared; + expect(agreement, greaterThanOrEqualTo(kMinAgreement), + reason: 'agreement $agree/$compared = ${agreement.toStringAsFixed(4)}'); + }); + + // Grid option 2 (owner decision 2026-10-23): the window start is phase 0 of + // the epoch grid, as before the bump. On this night the start is the first + // sample, so agreement with the frozen hypnogram is exactly 1.0 (it is 1.0 + // under the gap-anchored grid too; this keeps it there). + test('onset grid: agreement with the frozen hypnogram is exactly 1.0', () { + final old = _golden(); + final stages = [ + for (final g in now.stages) (g.start - _base, g.end - _base, g.stage) + ]; + var compared = 0, agree = 0; + for (var start = _base; start + 30 <= _base + nightOrNull.seconds; start += 30) { + final mid = start + 15 - _base; + final a = _at(old, mid), b = _at(stages, mid); + // Complete label equality: 'unobserved' (and missing) epochs count too. + compared++; + if (a == b) agree++; + } + expect(compared, greaterThan(1000)); + expect(agree, compared, reason: 'agreement $agree/$compared'); + }); + + test('a window that starts mid-night (unaligned to the first sample) is ' + 'staged on the onset grid: labels equal the pre-bump stager', () { + // 5 h in, 17 s off the 30 s steps of the first sample, 3 h long. + final start = _base + 18000 + 17, end = start + 3 * 3600; + final got = AdvancedSleepStager.stageWindow( + start, end, nightOrNull.grav, nightOrNull.hrTs, rr: nightOrNull.rrSec) + .stages; + final want = preBumpStageSessionCardio( + start, end, nightOrNull.grav, nightOrNull.hrTs, nightOrNull.rrSec); + expect(showSegments(got), showSegments(want)); + }); + + test('the real night\'s hypnogram boundaries sit on the grid of their ' + 'gap-free run (30 s steps from the run\'s first sample)', () { + // The run anchors, found independently of the code under test: a new run + // starts at the first valid sample more than 61 s after the previous one + // (the stager's 60 s carry-forward bridges anything shorter). + final anchors = []; + int? prev; + for (final g in nightOrNull.grav) { + if (prev == null || g.ts > prev + 61) anchors.add(g.ts); + prev = g.ts; + } + expect(anchors.first, _base); + final stages = now.stages; + var checked = 0; + for (var i = 1; i < stages.length; i++) { + if (stages[i].stage == 'unobserved' || stages[i - 1].stage == 'unobserved') { + continue; + } + final edge = stages[i].start; + final anchor = anchors.lastWhere((a) => a <= edge); + expect((edge - anchor) % 30, 0, + reason: 'boundary at $edge (${stages[i - 1].stage} -> ' + '${stages[i].stage}), run anchored at $anchor'); + checked++; + } + expect(checked, greaterThan(20)); + }); + + test('EpochRows over the whole night in 15-minute chunks with checkpoints ' + 'stages exactly like one batch, and its checkpoint stays small', () { + var er = EpochRows(); + var worst = 0; + for (var lo = 0; lo < nightOrNull.seconds; lo += 900) { + final hi = lo + 900 < nightOrNull.seconds ? lo + 900 : nightOrNull.seconds; + final t0 = (_base + lo) * 1000.0, t1 = (_base + hi) * 1000.0; + bool inside(double t) => t >= t0 && t < t1; + final ri = [ + for (var i = 0; i < nightOrNull.rrTs.length; i++) + if (inside(nightOrNull.rrTs[i])) i + ]; + er.fold( + hr: nightOrNull.hr.sublist(lo, hi), + accel: nightOrNull.accel.sublist(lo, hi), + rrMs: [for (final i in ri) nightOrNull.rrMs[i]], + rrTsMs: [for (final i in ri) nightOrNull.rrTs[i]], + ); + final text = jsonEncode(er.toJson()); + if (text.length > worst) worst = text.length; + er = EpochRows.fromJson(jsonDecode(text) as Map); + } + // ~1,070 rows of 8 numbers plus an unsettled tail of a few minutes of + // samples; the raw night (32,000 s x 5 channels) would be several times this. + expect(worst, lessThan(800000), reason: 'checkpoint chars'); + final got = er.stageWindow(_base, _base + nightOrNull.seconds); + expect(jsonEncode([for (final g in got) g.toJson()]), + jsonEncode([for (final g in now.stages) g.toJson()])); + }); + + test('DREAMT harness: per-epoch agreement of the gap-anchored grid with the ' + 'run-start grid on the PSG corpus', () { + // Needs (1) the DREAMT feature export (stager_harness.dart's fixture JSON), + // which is not in this repo, and (2) a grid-comparison mode in + // tool/stager_harness.dart: classify the same subject once on epochs cut at + // the subject's start and once on epochs cut 0..29 s later, report per-epoch + // agreement and kappa for both against the PSG labels. Threshold: 0.97. + fail('unreachable: skipped until the corpus and the harness mode exist'); + }, skip: 'DREAMT fixture is not in the repo and tool/stager_harness.dart has no ' + 'grid-comparison mode; run it on the machine that holds the corpus ' + '(B4 of combined-bump/brief.md).'); +} diff --git a/test/properties/stage_minute_ranges_laws_test.dart b/test/properties/stage_minute_ranges_laws_test.dart new file mode 100644 index 00000000..77cdfd73 --- /dev/null +++ b/test/properties/stage_minute_ranges_laws_test.dart @@ -0,0 +1,226 @@ +// LAWS — per-stage minute ranges (D27 revised). RED: new API. +// +// See test/onehz/stage_minute_ranges_test.dart for the contract and the API. +// These laws hold for any night the stager could produce: +// L1 bounds: 0 ≤ lo ≤ point ≤ hi ≤ window per stage; Σlo ≤ window ≤ Σhi; +// deep and light sit inside nrem (lo and hi both). +// L2 coverage: covering more epochs never widens a range (lo never falls, +// hi never rises), whatever the newly covered epochs say. +// L3 clear: every epoch covered and every score clear of the margin ⇒ +// lo == hi == point for every stage. +// +// Run: dart test test/properties/stage_minute_ranges_laws_test.dart + +import 'package:openstrap_analytics/onehz.dart'; +import 'package:test/test.dart'; + +import '../support/law_registry.dart'; +import '../support/property.dart'; + +const double _margin = 0.2; +const double _remCut = kDefaultRemScoreCut; +const double _deepCut = kDefaultDeepScoreCut; + +/// One generated epoch: label index (0 none, 1 wake, 2 light, 3 deep, 4 rem), +/// (remScore, deepScore), and a coverage level: 0 uncovered in both nights of +/// L2, 1 uncovered in A but covered in B, 2 covered in both. +typedef _Ep = (int, (double?, double?), int); + +final Gen _score = G.doubleIn(-3, 3, boundaries: [ + _remCut, + _remCut - _margin, + _remCut + _margin, + _deepCut, + _deepCut - _margin, + _deepCut + _margin, +]); + +final Gen<_Ep> _ep = G.triple( + G.intIn(0, 4, origin: 2), + G.pair(G.nullable(_score, nullProbability: .15), + G.nullable(_score, nullProbability: .15)), + G.intIn(0, 2, origin: 2), +); + +final Gen> _night = + _SometimesAllCovered(G.listOf(_ep, minLen: 1, maxLen: 240)); + +/// Per-epoch coverage is drawn independently, so a night of any length is +/// almost never covered end to end. A fifth of the nights are forced fully +/// covered so L1 meets that case (it is the shape a good night has). +class _SometimesAllCovered extends Gen> { + _SometimesAllCovered(this.inner); + final Gen> inner; + + @override + List<_Ep> generate(Rng r, int size) { + final n = inner.generate(r, size); + if (!r.nextBool(.2)) return n; + return [for (final (l, s, _) in n) (l, s, 2)]; + } + + @override + Iterable> shrink(List<_Ep> v) => inner.shrink(v); + + @override + String show(List<_Ep> v) => inner.show(v); +} + +StageEpochEvidence _evidence(_Ep e, {required bool covered}) { + final (label, (rem, deep), _) = e; + final SleepStage? stage = switch (label) { + 1 => SleepStage.wake, + 2 || 3 => SleepStage.nrem, + 4 => SleepStage.rem, + _ => null, + }; + return StageEpochEvidence( + stage: stage, + deep: label == 3, + covered: covered, + remScore: stage == SleepStage.wake ? null : rem, + deepScore: stage == SleepStage.wake ? null : deep, + ); +} + +List _nightA(List<_Ep> n) => + [for (final e in n) _evidence(e, covered: e.$3 == 2)]; +List _nightB(List<_Ep> n) => + [for (final e in n) _evidence(e, covered: e.$3 >= 1)]; + +/// The stager's own labelled minutes per display stage. +Map _points(List ev) { + final p = {'wake': 0.0, 'light': 0.0, 'deep': 0.0, 'rem': 0.0, 'nrem': 0.0}; + for (final e in ev) { + switch (e.stage) { + case SleepStage.wake: + p['wake'] = p['wake']! + .5; + case SleepStage.nrem: + p[e.deep ? 'deep' : 'light'] = p[e.deep ? 'deep' : 'light']! + .5; + p['nrem'] = p['nrem']! + .5; + case SleepStage.rem: + p['rem'] = p['rem']! + .5; + case null: + break; + } + } + return p; +} + +Map _byName(StageMinuteRanges r) => { + 'wake': r.wake, + 'light': r.light, + 'deep': r.deep, + 'rem': r.rem, + 'nrem': r.nrem, + }; + +bool _near(double? s, double cut) => s != null && (s - cut).abs() < _margin; + +void _observe(List<_Ep> n, void Function(String) bump) { + if (n.any((e) => e.$3 == 0)) bump('uncovered epoch'); + if (n.any((e) => e.$3 == 1)) bump('newly covered epoch'); + if (n.every((e) => e.$3 == 2)) bump('all covered'); + if (n.any((e) => e.$1 == 0)) bump('unlabelled epoch'); + if (n.any((e) => e.$1 == 1)) bump('wake'); + if (n.any((e) => e.$1 == 3)) bump('deep'); + if (n.any((e) => e.$1 == 4)) bump('rem'); + if (n.any((e) => e.$1 >= 2 && _near(e.$2.$1, _remCut))) bump('near rem cut'); + if (n.any((e) => e.$1 >= 2 && _near(e.$2.$2, _deepCut))) { + bump('near deep cut'); + } +} + +/// L3's input: the same night with every epoch covered, labelled, and both +/// scores pushed clear of their cutoffs. +List _clear(List<_Ep> n) => [ + for (final (label, (rem, deep), _) in n) + _evidence(( + label == 0 ? 2 : label, + (_clearOf(rem, _remCut), _clearOf(deep, _deepCut)), + 2, + ), covered: true) + ]; + +double? _clearOf(double? s, double cut) { + // A missing score decided nothing, so it is undecided and never clear: L3 + // gives it a score well clear of the cutoff, like any other. + if (s == null || s.isNaN) return cut + 2 * _margin; + if (!_near(s, cut)) return s; + return s >= cut ? cut + _margin : cut - _margin; +} + +void main() { + final laws = LawSet(); + final reach = Reach>({ + 'uncovered epoch': .5, + 'newly covered epoch': .5, + 'all covered': .02, + 'unlabelled epoch': .3, + 'wake': .5, + 'deep': .5, + 'rem': .5, + 'near rem cut': .3, + 'near deep cut': .3, + }, _observe); + + laws.law>('L1 ranges bracket the point and the window', _night, + (n) { + for (final ev in [_nightA(n), _nightB(n)]) { + final r = stageMinuteRanges(ev, margin: _margin)!; + final window = ev.length * .5; + expect(r.windowMin, closeTo(window, 1e-9)); + final pts = _points(ev); + var sumLo = 0.0, sumHi = 0.0; + _byName(r).forEach((name, range) { + expect(range.lo, greaterThanOrEqualTo(0), reason: '$name lo ≥ 0'); + expect(range.hi, lessThanOrEqualTo(window + 1e-9), + reason: '$name hi ≤ window'); + expect(range.lo, lessThanOrEqualTo(pts[name]! + 1e-9), + reason: '$name lo ≤ point'); + expect(range.hi, greaterThanOrEqualTo(pts[name]! - 1e-9), + reason: '$name point ≤ hi'); + if (name != 'nrem') { + sumLo += range.lo; + sumHi += range.hi; + } + }); + expect(sumLo, lessThanOrEqualTo(window + 1e-9), reason: 'Σlo ≤ window'); + expect(sumHi, greaterThanOrEqualTo(window - 1e-9), + reason: 'window ≤ Σhi'); + for (final sub in [r.deep, r.light]) { + expect(sub.lo, lessThanOrEqualTo(r.nrem.lo + 1e-9)); + expect(sub.hi, lessThanOrEqualTo(r.nrem.hi + 1e-9)); + } + } + }, reach: reach, examples: [ + [(0, (null, null), 0)], + [(4, (_remCut + .01, null), 2), (3, (null, _deepCut - .01), 2)], + ]); + + laws.law>('L2 more coverage never widens a range', _night, (n) { + final a = _byName(stageMinuteRanges(_nightA(n), margin: _margin)!); + final b = _byName(stageMinuteRanges(_nightB(n), margin: _margin)!); + for (final k in a.keys) { + expect(b[k]!.lo, greaterThanOrEqualTo(a[k]!.lo - 1e-9), + reason: '$k lo fell with more coverage'); + expect(b[k]!.hi, lessThanOrEqualTo(a[k]!.hi + 1e-9), + reason: '$k hi rose with more coverage'); + } + }, reach: reach, examples: [ + [(1, (null, null), 1), (2, (-1, -1), 0)], + ]); + + laws.law>('L3 covered and clear ⇒ lo == hi == point', _night, + (n) { + final ev = _clear(n); + final r = stageMinuteRanges(ev, margin: _margin)!; + final pts = _points(ev); + _byName(r).forEach((name, range) { + expect(range.lo, closeTo(pts[name]!, 1e-9), reason: '$name lo'); + expect(range.hi, closeTo(pts[name]!, 1e-9), reason: '$name hi'); + }); + }, reach: reach); + + laws.registerReachTest(); +} diff --git a/tool/causal_stager_validate.dart b/tool/causal_stager_validate.dart new file mode 100644 index 00000000..4b3b337d --- /dev/null +++ b/tool/causal_stager_validate.dart @@ -0,0 +1,508 @@ +// VALIDATION HARNESS — replay a recorded night through the CAUSAL stager and +// score it against the retrospective one. +// +// WHAT THIS CAN AND CANNOT SAY. The only recorded night in the repo +// (test/onehz/fixtures/real_night_2026_07_*.csv) has no epoch labels at all — +// just the Apple Watch's whole-night stage minutes, quoted in +// test/onehz/real_night_cardio_stager_test.dart. There is no PSG-labelled set +// in-tree (the DREAMT corpus behind `tool/stager_harness.dart` lives outside +// the repo). So everything below is AGREEMENT WITH THE OFFLINE STAGER, which +// is itself a kappa-0.13 wrist estimate against PSG. It measures how closely +// the online rules track the retrospective ones and how the "REM, wake now" +// trigger behaves against that reference — NOT stage accuracy. One night, one +// subject: read the numbers as a smoke test with a ruler, not as a result. +// +// Usage: +// dart run tool/causal_stager_validate.dart [onehz.csv rr.csv] [--step S] +// [--warmup N] +// defaults: the in-tree real night, 30 s steps, the shipped warm-up. +// +// Fixture columns: onehz.csv `rel_sec,hr,ax,ay,az`; rr.csv `rel_ms,rr_ms`. + +import 'dart:convert'; +import 'dart:io'; + +import 'package:openstrap_analytics/onehz.dart'; + +const _fixtures = 'test/onehz/fixtures/'; + +/// Shift the fixture's 0-based clock to a realistic, 30 s-aligned absolute one. +const double _t0Ms = 1700000010000; + +/// The parsed command line: fixture paths plus the two options. +class ValidateArgs { + /// Arguments no option consumed: the one-Hz and RR fixture paths. + final List pos; + final int stepSec, warmup; + + /// Set when the command line is not usable. + final String? error; + const ValidateArgs(this.pos, this.stepSec, this.warmup, [this.error]); +} + +/// Each option is read together with its value (`--step 60` or `--step=60`), so +/// a value is never mistaken for a fixture path. +ValidateArgs parseValidateArgs(List args, + {int defaultStep = 30, int? defaultWarmup}) { + final pos = []; + var step = defaultStep; + var warm = defaultWarmup ?? CausalStagerConfig.defaults.warmupEpochs; + ValidateArgs bad(String why) => ValidateArgs(pos, step, warm, why); + for (var i = 0; i < args.length; i++) { + final a = args[i]; + if (!a.startsWith('--')) { + pos.add(a); + continue; + } + final eq = a.indexOf('='); + final name = eq >= 0 ? a.substring(0, eq) : a; + if (name != '--step' && name != '--warmup') return bad('unknown option $name'); + String? raw = eq >= 0 ? a.substring(eq + 1) : null; + if (raw == null) { + if (i + 1 >= args.length) return bad('$name needs a value'); + raw = args[++i]; + } + final n = int.tryParse(raw); + if (n == null || n <= 0) { + return bad('$name needs a positive whole number, got "$raw"'); + } + if (name == '--step') { + step = n; + } else { + warm = n; + } + } + return ValidateArgs(pos, step, warm); +} + +void main(List args) { + final parsed = parseValidateArgs(args); + if (parsed.error != null) { + stderr.writeln(parsed.error); + exitCode = 64; + return; + } + final pos = parsed.pos; + final stepSec = parsed.stepSec; + final warm = parsed.warmup; + final onehz = pos.isNotEmpty ? pos[0] : '${_fixtures}real_night_2026_07_onehz.csv'; + final rrPath = pos.length > 1 ? pos[1] : '${_fixtures}real_night_2026_07_rr.csv'; + if (!File(onehz).existsSync() || !File(rrPath).existsSync()) { + stderr.writeln('fixture not found: $onehz / $rrPath'); + exitCode = 66; + return; + } + final night = _load(onehz, rrPath); + stdout.writeln('night: ${night.hr.length} s ' + '(${(night.hr.length / 3600).toStringAsFixed(2)} h), ' + '${night.rrTs.length} RR beats'); + + // ── offline reference ────────────────────────────────────────────────────── + final off = cardioStager(night.hr, night.accel, + rrMs: night.rr, rrTsMs: night.rrTs); + final offStages = off.base.stages; + final nEp = offStages.length; + stdout.writeln('offline epochs: $nEp confidence ${off.confidence}'); + _minutes('offline cardioStager', offStages.map(_name).toList()); + stdout.writeln('Apple Watch reference (summary only): wake 3 REM 162 ' + 'light 330 deep 38 min'); + stdout.writeln(''); + + // ── causal replay ────────────────────────────────────────────────────────── + final cfg = CausalStagerConfig(warmupEpochs: warm); + final run = _replay(night, stepSec, cfg); + final again = _replay(night, stepSec, cfg); + final deterministic = _digest(run) == _digest(again); + stdout.writeln('causal replay: step ${stepSec}s, warm-up $warm epochs ' + '(${warm * 30 / 60} min), deterministic on replay: $deterministic'); + + // Latest observation per epoch (a finer step can describe an epoch twice). + final perEpoch = {}; + for (final o in run) { + if (o.stage != CausalStage.absent) { + perEpoch[(o.epochStartMs! / 30000).round()] = o; + } + } + final reasons = {}; + for (final o in run) { + if (o.stage == CausalStage.absent) { + reasons[o.abstentionReason!.name] = + (reasons[o.abstentionReason!.name] ?? 0) + 1; + } + } + final base = (_t0Ms / 30000).round(); + stdout.writeln('observations ${run.length}: staged ${run.length - reasons.values.fold(0, (a, b) => a + b)}, ' + 'abstained $reasons'); + final causal = List.filled(nEp, null); + for (final e in perEpoch.entries) { + final i = e.key - base; + if (i >= 0 && i < nEp) causal[i] = e.value.stage.name; + } + _minutes('causal (staged epochs only)', + [for (final c in causal) if (c != null) c]); + stdout.writeln(''); + + // ── agreement, by lag ────────────────────────────────────────────────────── + // causal(k) is built from TRAILING windows, so it describes the physiology a + // couple of minutes earlier. Compare causal(k) with offline(k - d). + stdout.writeln('AGREEMENT with offline cardioStager (staged epochs only)'); + stdout.writeln(' lag d: causal(k) vs offline(k-d) n agree kappa'); + var bestD = 0; + var bestK = -2.0; + for (final d in [0, 1, 2, 3, 4, 6, 8]) { + final a = [], b = []; + for (var k = d; k < nEp; k++) { + final c = causal[k]; + if (c == null) continue; + a.add(_idx(c)); + b.add(_idx(_name(offStages[k - d]))); + } + final kap = _kappa(a, b); + if (kap > bestK) { + bestK = kap; + bestD = d; + } + stdout.writeln(' d=$d (${d * 30}s)${' ' * (21 - '$d (${d * 30}s)'.length)}' + '${a.length.toString().padLeft(6)} ${_pct(_agree(a, b))} ${_f(kap)}'); + } + stdout.writeln(' best lag: d=$bestD'); + { + final a = [], b = []; + for (var k = 0; k < nEp; k++) { + final c = causal[k]; + if (c == null) continue; + a.add(_idx(c)); + b.add(_idx(_name(offStages[k]))); + } + stdout.writeln(''); + stdout.writeln('CONFUSION at d=0 (rows = offline, cols = causal)'); + _confusion(b, a); + } + + // ── the REM "wake now" use case ──────────────────────────────────────────── + _remTrigger(perEpoch, base, offStages, nEp, causal); + + // ── warm-up sensitivity ──────────────────────────────────────────────────── + stdout.writeln(''); + stdout.writeln('WARM-UP sensitivity (d=0) warm-up staged% agree kappa'); + for (final w in [20, 40, 60, 120]) { + final r = _replay(night, 30, CausalStagerConfig(warmupEpochs: w)); + final a = [], b = []; + var staged = 0; + for (final o in r) { + if (o.stage == CausalStage.absent) continue; + staged++; + final i = (o.epochStartMs! / 30000).round() - base; + if (i < 0 || i >= nEp) continue; + a.add(_idx(o.stage.name)); + b.add(_idx(_name(offStages[i]))); + } + stdout.writeln(' ${(w * 0.5).toStringAsFixed(0).padLeft(4)} min ' + '${_pct(staged / nEp)} ${_pct(_agree(a, b))} ${_f(_kappa(a, b))}'); + } +} + +// ── REM trigger analysis ─────────────────────────────────────────────────────── + +/// "Wake now" = REM that has persisted for X seconds (`runSec >= X`), the rule +/// a caller would build from the observation. 300 s is [remEpisodeMinMin], the +/// retrospective stager's shortest credible REM bout; the table below shows +/// what shorter thresholds trade. +void _remTrigger(Map perEpoch, int base, + List offStages, int nEp, List causal) { + bool offRem(int k) => k >= 0 && k < nEp && offStages[k] == SleepStage.rem; + // Offline REM within +-5 min (10 epochs): trailing windows keep the edges of a + // bout in view for a few minutes, so "near" is the fair no-false-trigger test. + bool nearRem(int k) { + for (var d = -10; d <= 10; d++) { + if (offRem(k + d)) return true; + } + return false; + } + + bool trigAt(int k, int x) { + final o = perEpoch[k + base]; + return o != null && o.stage == CausalStage.rem && o.runSec >= x; + } + + final bouts = <(int, int)>[]; + for (var i = 0; i < nEp;) { + if (!offRem(i)) { + i++; + continue; + } + var j = i; + while (j < nEp && offRem(j)) { + j++; + } + bouts.add((i, j)); + i = j; + } + + var nStaged = 0, nOffRem = 0, nNear = 0; + for (var k = 0; k < nEp; k++) { + if (causal[k] == null) continue; + nStaged++; + if (offRem(k)) nOffRem++; + if (nearRem(k)) nNear++; + } + stdout.writeln(''); + stdout.writeln('REM "WAKE NOW" TRIGGER reference = ${bouts.length} offline REM ' + 'bouts as the retrospective stager emits them (several are under 5 min)'); + stdout.writeln(' chance level for a trigger landing on offline REM: ' + '${_pct(nOffRem / nStaged)}; within +-5 min of it: ${_pct(nNear / nStaged)}' + ' <- the "no false trigger" test is only as strict as this'); + stdout.writeln(' rule: causal stage = rem AND run >= X s'); + stdout.writeln(' X s trigger epochs @offlineREM @+-5min falseEpochs ' + 'bouts caught (of ${bouts.length}) lateness min (median) ' + 'late-tail epochs'); + for (final x in [60, 120, 180, 300, 420]) { + final eps = [for (var k = 0; k < nEp; k++) if (trigAt(k, x)) k]; + final onRem = eps.where(offRem).length; + final near = eps.where(nearRem).length; + var caught = 0; + final late = []; + for (final (s, e) in bouts) { + for (var k = s; k < e; k++) { + if (trigAt(k, x)) { + caught++; + late.add((k - s) * 0.5); + break; + } + } + } + // Trigger epochs that are not REM but follow a bout within 5 min: the + // late tail, i.e. "woke you just after REM ended". + final tail = eps.where((k) { + if (offRem(k)) return false; + for (var d = 1; d <= 10; d++) { + if (offRem(k - d)) return true; + } + return false; + }).length; + String p(int v) => eps.isEmpty ? ' n/a' : _pct(v / eps.length); + stdout.writeln(' ${x.toString().padLeft(5)} ${eps.length.toString().padLeft(13)}' + ' ${p(onRem)} ${p(near)} ${(eps.length - near).toString().padLeft(11)}' + ' ${caught.toString().padLeft(16)}' + ' ${late.isEmpty ? "n/a".padLeft(20) : _median(late).toStringAsFixed(1).padLeft(20)}' + ' ${tail.toString().padLeft(16)}'); + } + + stdout.writeln(''); + stdout.writeln(' per offline bout at X=300 / X=120 (first trigger INSIDE the bout):'); + for (final (s, e) in bouts) { + String f(int x) { + for (var k = s; k < e; k++) { + if (trigAt(k, x)) return '+${((k - s) * 0.5).toStringAsFixed(1)} min'; + } + return 'none'; + } + + stdout.writeln(' ${_hm(s)}-${_hm(e)} (${((e - s) * 0.5).toStringAsFixed(1).padLeft(4)} min)' + ' X=300: ${f(300).padRight(9)} X=120: ${f(120)}'); + } + + // Natural-Wake-shaped windows: every must-be-up time T on the 30 s grid whose + // whole window [T-N, T) was observable; fire once, at the first trigger epoch. + for (final x in [120, 300]) { + stdout.writeln(''); + stdout.writeln('WINDOW SIMULATION, X=$x s (fire once at the first trigger ' + 'epoch in [T-N, T))'); + stdout.writeln(' N min windows fired @offlineREM @+-5min ' + 'fired w/o REM near missed(REM in window) quiet(no REM)'); + for (final nMin in [15, 30, 60, 120]) { + final nW = nMin * 2; + var windows = 0, fired = 0, onRem = 0, near = 0, falseFire = 0; + var missed = 0, quiet = 0; + for (var t = nW; t <= nEp; t++) { + var ok = true; + for (var k = t - nW; k < t; k++) { + if (causal[k] == null) { + ok = false; + break; + } + } + if (!ok) continue; + windows++; + int? f; + var remInWin = false; + for (var k = t - nW; k < t; k++) { + if (f == null && trigAt(k, x)) f = k; + if (offRem(k)) remInWin = true; + } + if (f != null) { + fired++; + if (offRem(f)) onRem++; + if (nearRem(f)) { + near++; + } else { + falseFire++; + } + } else if (remInWin) { + missed++; + } else { + quiet++; + } + } + String c(int v, [int w = 9]) => v.toString().padLeft(w); + stdout.writeln(' ${nMin.toString().padLeft(4)} ${c(windows)}${c(fired)}' + '${c(onRem, 13)}${c(near, 9)}${c(falseFire, 19)}${c(missed, 21)}${c(quiet, 15)}'); + } + } +} + +// ── replay + metrics ────────────────────────────────────────────────────────── + +List _replay( + _Night n, int stepSec, CausalStagerConfig cfg) { + final out = []; + CausalStagerState? st = CausalStagerState.initial(cfg); + final secs = n.hr.length; + var rrI = 0; + for (var t = stepSec; t <= secs; t += stepSec) { + final lo = t - stepSec; + final loMs = _t0Ms + lo * 1000.0, hiMs = _t0Ms + t * 1000.0; + final rrTs = [], rrV = []; + while (rrI < n.rrTs.length && _t0Ms + n.rrTs[rrI] < hiMs) { + if (_t0Ms + n.rrTs[rrI] >= loMs) { + rrTs.add(_t0Ms + n.rrTs[rrI]); + rrV.add(n.rr[rrI]); + } + rrI++; + } + final o = CausalStager.observe( + CausalSampleWindow( + nowMs: hiMs, + hr: [ + for (var i = lo; i < t; i++) HrSample(_t0Ms + i * 1000.0, n.hr[i]) + ], + accel: [ + for (var i = lo; i < t; i++) + AccelSample(_t0Ms + i * 1000.0, n.accel[i].x, n.accel[i].y, + n.accel[i].z) + ], + rr: RrSeries(rrTs, rrV), + ), + st); + out.add(o); + st = o.nextState; + } + return out; +} + +String _digest(List r) { + var h = 0; + for (final o in r) { + for (final c in jsonEncode(o.toJson()).codeUnits) { + h = (h * 31 + c) & 0x7fffffff; + } + } + return '$h'; +} + +class _Night { + final List hr; + final List accel; + final List rr, rrTs; + _Night(this.hr, this.accel, this.rr, this.rrTs); +} + +_Night _load(String onehz, String rrPath) { + final hr = []; + final accel = []; + final l1 = File(onehz).readAsLinesSync(); + for (var i = 1; i < l1.length; i++) { + if (l1[i].trim().isEmpty) continue; + final p = l1[i].split(','); + hr.add(double.parse(p[1])); + accel.add(AccelSample(double.parse(p[0]) * 1000.0, double.parse(p[2]), + double.parse(p[3]), double.parse(p[4]))); + } + final rr = [], rrTs = []; + final l2 = File(rrPath).readAsLinesSync(); + for (var i = 1; i < l2.length; i++) { + if (l2[i].trim().isEmpty) continue; + final p = l2[i].split(','); + rrTs.add(double.parse(p[0])); + rr.add(double.parse(p[1])); + } + return _Night(hr, accel, rr, rrTs); +} + +String _name(SleepStage s) => s.name; +int _idx(String s) => const {'wake': 0, 'nrem': 1, 'rem': 2}[s]!; +String _pct(double v) => '${(100 * v).toStringAsFixed(1).padLeft(5)}%'; +String _f(double v) => v.isNaN ? ' n/a' : v.toStringAsFixed(3).padLeft(6); +String _hm(int epoch) { + final m = epoch ~/ 2; + return '${m ~/ 60}h${(m % 60).toString().padLeft(2, '0')}'; +} + +double _median(List v) { + final s = [...v]..sort(); + return s[s.length ~/ 2]; +} + +void _minutes(String label, List stages) { + final c = {'wake': 0, 'nrem': 0, 'rem': 0}; + for (final s in stages) { + c[s] = c[s]! + 1; + } + stdout.writeln('$label: wake ${c['wake']! / 2} min nrem ${c['nrem']! / 2} min ' + 'rem ${c['rem']! / 2} min (${stages.length / 2} min)'); +} + +double _agree(List a, List b) { + if (a.isEmpty) return double.nan; + var m = 0; + for (var i = 0; i < a.length; i++) { + if (a[i] == b[i]) m++; + } + return m / a.length; +} + +double _kappa(List a, List b) { + final n = a.length; + if (n == 0) return double.nan; + final m = List.generate(3, (_) => List.filled(3, 0)); + for (var i = 0; i < n; i++) { + m[a[i]][b[i]]++; + } + var po = 0.0, pe = 0.0; + for (var i = 0; i < 3; i++) { + po += m[i][i]; + final r = m[i].reduce((x, y) => x + y); + var c = 0; + for (var j = 0; j < 3; j++) { + c += m[j][i]; + } + pe += r * c / n; + } + po /= n; + pe /= n; + return pe >= 1 ? double.nan : (po - pe) / (1 - pe); +} + +/// rows = `ref`, cols = `got`. +void _confusion(List ref, List got) { + const names = ['wake', 'nrem', 'rem']; + final m = List.generate(3, (_) => List.filled(3, 0)); + for (var i = 0; i < ref.length; i++) { + m[ref[i]][got[i]]++; + } + stdout.writeln(' ${names.map((s) => s.padLeft(7)).join()} recall'); + for (var i = 0; i < 3; i++) { + final tot = m[i].reduce((a, b) => a + b); + stdout.writeln(' ${names[i].padRight(9)} ${m[i].map((v) => v.toString().padLeft(7)).join()}' + ' ${tot == 0 ? " n/a" : _pct(m[i][i] / tot)}'); + } + final prec = []; + for (var j = 0; j < 3; j++) { + var col = 0; + for (var i = 0; i < 3; i++) { + col += m[i][j]; + } + prec.add(col == 0 ? ' n/a' : _pct(m[j][j] / col).padLeft(7)); + } + stdout.writeln(' precision ${prec.join()}'); +}