Repository navigation
sleep: per-night HMM provenance, cardio parity and setting in cache identity (follow-up to #756) - #763
sleep: per-night HMM provenance, cardio parity and setting in cache identity (follow-up to #756)#763mikenrafter wants to merge 12 commits into
Conversation
Wake was under-detected because the emission used only HR/RMSSD and 'motion_artifact' was hardcoded false — movement never reached the classifier. Now: - DayBundleInput carries sleepEnmo (per-second ENMO, g; 0 = absent sentinel = neutral, never 'still'); derivation_engine computes it from the sleep substrate's accel vector via accelPresentAt. - Emission is a noisy-OR over autonomic (HR up / RMSSD down), motion (vs the night's own ENMO median), and skin temp (vs night median) — any strong cue can carry wake, so brief movement-only awakenings are caught. Sleep states are penalised by movement. - Unusable HR/RR falls back to motion-only evidence instead of uniform neutral (which used to decode unmeasured-but-moving nights as wake via transition ties). - Transitions less sticky into wake: light->wake 0.05->0.12, deep->wake 0.02->0.06, rem->wake 0.05->0.10. - 5 new hybrid tests (motion-only arousal, quiet sleep, HR arousal, absent-ENMO neutrality, motion-only fallback). Still default-off. Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
…ng-term baseline Extends the hybrid HMM with per-user multi-night calibration: - computeMultiNightBaselines() builds a 7-day rolling and a long-term baseline (medians) from the persisted nightly rhr/rmssd histories - the emission now uses a blended effective baseline (nightly + 7d), so one restless night no longer dominates the thresholds - outlier detection: if tonight's baseline deviates >2 robust z (MAD) from the long-term baseline, the night is flagged atypicalNight and persisted as stager_atypical_night in sleepJson (lower confidence) - no history -> falls back to the nightly baseline alone - 4 new tests in test/sleep_hmm_multinight_test.dart Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
…in temp Medical grounding (AASM Scoring Manual v2.x): - an EEG arousal lasts >=3 s and needs >=10 s stable sleep before it; an epoch is stage W only when >50% of the 30 s epoch is awake -> wake runs shorter than 15 s in our per-second decode are arousals, not wake. _hmmRestage now reverts wake runs <15 s to the surrounding sleep stage. - long wake runs (>=15 s) stay wake and count towards WASO. - staging after a wake phase continues sequentially (no restart from 0) — the Viterbi path already respects continuity; pinned by a test. Also: skin temp now self-calibrates across nights too (7d rolling + long-term medians from skinTempAdcHistory, blended into the emission like HR/RMSSD). ENMO stays nightly-only: no persisted per-night motion history exists yet. 3 new AASM wake-rule tests. Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
…e opening window The first 120 min contain settling-in/turning and are systematically more active, so a motion median taken from them overstated the person's normal sleep motion and pushed the wake-vs-arousal threshold too high. New computeMotionBaseline() takes the ENMO median from the middle 50 % of the night by elapsed time. HR/RMSSD keep the opening-window baseline (autonomic settling is intended there). Absent ENMO (0 sentinel) is still excluded. 2 new tests. Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
- transition rows normalized to sum 1.0 (deep light 0.14->0.11, rem self 0.85->0.80); REM->deep hard-forbid preserved - fully missing epochs (no HR/RR AND absent ENMO) return uniform neutral log(0.25) again, so gaps cannot bias adjacent epochs towards wake - absent optional sensors are excluded from the wake noisy-OR (factor 0), never the 0.5 that silently boosted wake - ENMO absent sentinel is now -1.0 (measured zero motion is >= 0), so epoch means count real still seconds in the denominator - skinTempHistory forwarded to computeMultiNightBaselines on the prod path - stager_atypical_night forwarded into the output sleep map - 2 new neutrality regression tests Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
…nStrap#593 pattern) Lying still and awake (low motion, HR clearly above the personal sleep baseline, RMSSD barely moved) was scored light because the autonomic cue is an AND of HR-up and RMSSD-down and motion was absent. Added a capped HR-only term to the wake noisy-OR: sigmoid(hrNorm, 1.22, 0.07) * 0.3. Guarded against REM (wake-like HR/HRV, Zemaityte et al.): high center 1.22x + cap 0.3 + REM's own emission and 0.80 self-transition keep REM-like still epochs non-wake — pinned by a regression test. A second test pins that still + HR ~1.25x reads as wake. 2 new tests. Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
…ross nights Closes the last calibration gap: motion now has a persisted per-night history like HR/RMSSD/skin temp. - HMM-restaged nights store their stable-core ENMO median as stager_enmo_median in sleepJson, forwarded into the bundle sleep block and persisted to metric_series as 'enmo_median_night' (only for HMM-staged nights, so the series fills from real staging) - _BaselineHistoryCache gains the 'enmo_median_night' key; _attachHistory feeds it as enmo_history into DayBundleInput - computeMultiNightBaselines builds enmoMedian7d/enmoMedianLong (absent sentinel -1 excluded; measured zero motion is a valid 0) and the emission blends nightly core + 7d via effectiveEnmoMedian - 2 new tests Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Reviewer's GuideData structuresAdd the captured HMM choice to prepared-day data and add sleep ENMO and ENMO history to the bundle input so the worker receives the inputs needed for staging.
Define the optional stored staging-provenance fields and their tolerant typed read boundary, preserving unknown strings while omitting malformed values.
APIExpose valid staging-provenance fields in the sleep detail repository response without inventing values for older payloads.
Uses of the APICapture the staging preference for each prepared night, pass it through the normal and imported worker paths, and build per-second sleep ENMO from the substrate.
ImplementationMake the HMM setting part of staging-cache identity so changing the requested method invalidates the relevant cached candidate.
Restage with hybrid HR, RMSSD, motion and temperature evidence; add multi-night calibration, wake-run post-processing, and fallback accounting and provenance in the produced sleep bundle.
Persist HMM-night ENMO medians into metric history and attach that history to later bundle inputs for motion self-calibration.
Bump the producing algorithm version so retained nights can be re-derived with the new provenance output.
DocumentationDefine staging provenance terminology and distinguish requested method, effective method, and observed fallback seconds.
TestsPin cardio byte parity, provenance semantics for whole-night and partial fallback, paired-device cases, stale-key clearing, and tolerant parsing.
Cover repository reads and selected-day and full-export restore of staging provenance, including legacy and malformed payloads.
Exercise hybrid emission, AASM wake handling, stable-core motion baselines, and multi-night calibration behavior.
Provide the deterministic HMM-night input fixture used to verify worker output and provenance.
Keep the WHOOP freeze comparison compatible with version 125 by projecting only version columns back to the pinned version.
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Door: one-way, because this writes stored payload keys and bumps the producing algorithm version.
Blast radius: data
Stack: 2 of 2, after #756
Upstream review order: #756, open, first; then this draft follow-up.
What it does and why
Each experimental sleep night records which method supplied its labels. It also records how many observed seconds kept the original labels when the hidden Markov model (HMM) could not stage them. Before this change, the HMM source tag could not distinguish a successful HMM pass from a partial fallback, and a whole-night fallback carried no method record.
The setting travels with the prepared worker input and participates in staging cache identity. Toggling it recomputes requested nights with the captured choice. Cardio output remains byte-identical to the #756 fixture. Algorithm version 125 makes retained nights eligible for the changed stored output. The HMM remains in #756's existing Edge compute layout.
Why this door
The optional keys become part of stored and exported day payloads, and version 125 creates new derived rows. A code revert leaves those rows and exported copies intact. For a practical rollback, disable the experimental setting and re-analyze nights whose inputs remain available; preserve monotonically increasing algorithm versions when shipping a follow-up. No database schema migration is needed.
Known limits
hmm_unavailablegroups whole-night baseline and usable-input abstentions. lib/compute/onehz_pipeline.dart:720hmm_epoch_unavailableidentifies partial epoch fallback. The metadata records the original source even for paired-device staging. The existing whole-night all-wake fallback is namedhmm_no_sleep.Questions for reviewers
Please check that the effective-method names make partial fallback clear, especially when the original labels came from a paired device. Confirm that version 125 is still the next version when this stack merges.
Review check (10 questions, 30 points)
If you can answer these at 80% or better (24 of 30 points) without looking, you have probably reviewed it well enough.
1. What does a whole-night HMM abstention record?
Hint
lib/compute/onehz_pipeline.dart:718
Answer
2. How does a partially staged night account for missing epochs?
Hint
lib/compute/onehz_pipeline.dart:2769
Answer
3. Why must a device-staged night have distinct fallback provenance?
Hint
test/hmm_staging_provenance_test.dart:69
Answer
4. What happens when an older or hand-edited payload is read?
Hint
lib/compute/staging_provenance.dart:4
Answer
5. Can a successful rerun retain an old fallback reason?
Hint
lib/compute/onehz_pipeline.dart:705
Answer
6. Where is the setting captured and how does it reach compute?
Hint
lib/compute/derivation_engine.dart:4458
Answer
7. Which cache entries move when the setting changes?
Hint
test/hmm_staging_flag_cache_test.dart:67
Answer
8. How is provenance preserved across backup and restore?
Hint
test/hmm_staging_provenance_read_test.dart:204
Answer
9. What proves cardio compatibility?
Hint
test/hmm_staging_provenance_test.dart:16
Answer
10. Why bump the version and how can this be rolled back?
Hint
lib/compute/derivation_engine.dart:1899
Answer
New terms
Staging provenance is defined in docs/lexicon.md. Avoid using requested method to mean effective method.
Decisions
No ADR changes. Provenance stays inside the existing sleep payload and uses the existing export, restore and repository seams.
Change size
This stack section only: diff from #756. The 4,000-line size check passes.
Tests
Boat, owned default VM from
edge-ready-2, Flutter 3.41.6:The gate was selected once against
90a879b21and ran atf5bbee727, before the final review fixes. All 409 files completed; no OOM kills. Its four version-only failures are fixed by exact version expectations and a version-125-specific projection of three tables' version columns to 124. Committed goldens, payload hashes and metric comparisons are unchanged. The final core and zone checks cover those fixes.One gate sleep-classifier failure reproduces on the exact base. The other seven gate failures/timeouts are in the BLE-link file and do not reproduce in fresh whole-file controls on either base or head. They are reported as non-reproducing gate-only failures, not as inherited. Five additional classifier failures match the exact base by file, test name and assertion. The classifier implementation and those test files are byte-identical to #756.
Analysis: 13 files, no issues. Five mutations produced assertion failures; all five restored controls passed. All 14 final source hashes match Boat. The size gate passes at 592 changed lines. No full-suite or final-head 409-file green claim is made.
On Linux, the existing WHOOP freeze comparison retains its 10-significant-digit policy and excludes stored-byte hashes because of platform libm differences. The new six-minute cardio fixture independently pins the full serialized payload hash captured from #756, with its clock frame fixed to UTC.
Provenance
OpenAI GPT-6.1 Sol, high reasoning, implemented the changes and tests, inspected the reference fork, and answered all ten review questions against the code. The export/read test scaffolding adapts the reference implementation named in the brief. Self-review found and corrected the paired-device provenance case. Self-review also found and corrected a missing flag capture on the imported-substrate path. No independent review round has run in this implementation task; the orchestrator owns external review and publishing. No Co-Authored-By or AI trailers were added.
Summary by Sourcery
Add provenance-aware experimental HMM sleep staging while preserving cardio compatibility and making the staging setting part of derived-data identity.
New Features:
Bug Fixes:
Enhancements:
Documentation:
Tests:
Chores: