Skip to content

sleep: per-night HMM provenance, cardio parity and setting in cache identity (follow-up to #756) - #763

Draft
mikenrafter wants to merge 12 commits into
OpenStrap:mainfrom
mikenrafter:hmm-structural-756
Draft

mikenrafter wants to merge 12 commits into
OpenStrap:mainfrom
mikenrafter:hmm-structural-756

Conversation

@mikenrafter

@mikenrafter mikenrafter commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Door: one-way, because this writes stored payload keys and bumps the producing algorithm version.
Blast radius: data
Stack: 2 of 2, after #756

Upstream review order: #756, open, first; then this draft follow-up.

What it does and why

Each experimental sleep night records which method supplied its labels. It also records how many observed seconds kept the original labels when the hidden Markov model (HMM) could not stage them. Before this change, the HMM source tag could not distinguish a successful HMM pass from a partial fallback, and a whole-night fallback carried no method record.

The setting travels with the prepared worker input and participates in staging cache identity. Toggling it recomputes requested nights with the captured choice. Cardio output remains byte-identical to the #756 fixture. Algorithm version 125 makes retained nights eligible for the changed stored output. The HMM remains in #756's existing Edge compute layout.

Why this door

The optional keys become part of stored and exported day payloads, and version 125 creates new derived rows. A code revert leaves those rows and exported copies intact. For a practical rollback, disable the experimental setting and re-analyze nights whose inputs remain available; preserve monotonically increasing algorithm versions when shipping a follow-up. No database schema migration is needed.

Known limits

Questions for reviewers

Please check that the effective-method names make partial fallback clear, especially when the original labels came from a paired device. Confirm that version 125 is still the next version when this stack merges.

Review check (10 questions, 30 points)

If you can answer these at 80% or better (24 of 30 points) without looking, you have probably reviewed it well enough.

1. What does a whole-night HMM abstention record?
Hint

lib/compute/onehz_pipeline.dart:718

Answer
  1. The requested method is hmm.
  2. The effective method names the original label source, cardio or vendor_staged.
  3. The fallback reason and observed fallback seconds are stored beside it.
2. How does a partially staged night account for missing epochs?
Hint

lib/compute/onehz_pipeline.dart:2769

Answer
  1. A null HMM label retains the original label.
  2. Fallback seconds exclude unobserved gaps.
  3. The effective method names HMM and the original source when both contributed.
3. Why must a device-staged night have distinct fallback provenance?
Hint

test/hmm_staging_provenance_test.dart:69

Answer
  1. Upstream can send vendor_staged labels through the HMM.
  2. Whole-night abstention keeps vendor_staged as the effective method.
  3. Partial abstention uses hmm_vendor_staged; the labels follow the existing pipeline.
4. What happens when an older or hand-edited payload is read?
Hint

lib/compute/staging_provenance.dart:4

Answer
  1. Absent keys remain absent.
  2. Wrong-typed values and non-finite or negative durations are omitted.
  3. Unknown method and reason strings survive, and duration requires a reason.
5. Can a successful rerun retain an old fallback reason?
Hint

lib/compute/onehz_pipeline.dart:705

Answer
  1. Provenance is built afresh for each bundle.
  2. A complete HMM result records hmm as the effective method.
  3. Complete success omits fallback reason and seconds.
6. Where is the setting captured and how does it reach compute?
Hint

lib/compute/derivation_engine.dart:4458

Answer
  1. Preparation reads the existing preference once for this night.
  2. The same value keys the staging and prefix contexts.
  3. PreparedDerivationDay carries it into DayBundleInput, including the separate import path.
7. Which cache entries move when the setting changes?
Hint

test/hmm_staging_flag_cache_test.dart:67

Answer
  1. Each requested night misses once for its new flag value.
  2. Repeating the same request hits, and an unrelated preference keeps the old hit.
  3. The real engine re-derives selected nights while an unrelated stored day stays unchanged.
8. How is provenance preserved across backup and restore?
Hint

test/hmm_staging_provenance_read_test.dart:204

Answer
  1. The keys live inside the existing day_result payload.
  2. Selected-day and full exports preserve all four literal fields.
  3. After deleting local day rows, importing each export restores readable provenance.
9. What proves cardio compatibility?
Hint

test/hmm_staging_provenance_test.dart:16

Answer
  1. The full serialized cardio bundle matches the hash captured at the exact feat(sleep): hybrid HMM emission over HR + RMSSD + ENMO + skin temp + multi-night calibration #756 head.
  2. Explicit false and the absent flag produce identical bytes.
  3. Flag-off sleep sections gain no staging_ or stager_ keys; the hash fixture pins UTC.
10. Why bump the version and how can this be rolled back?
Hint

lib/compute/derivation_engine.dart:1899

Answer
  1. Version 125 gives retained inputs a new producing version for the changed stored output.
  2. Reverting code does not erase stored or exported provenance or versioned rows.
  3. Disable HMM and re-analyze available data to return those nights to cardio; pruned inputs cannot be recreated.
New terms

Staging provenance is defined in docs/lexicon.md. Avoid using requested method to mean effective method.

Scenario Chosen term in use Is this how the owner talks?
Successful night The staging provenance says HMM supplied the labels. Yes
Whole-night abstention The staging provenance records cardio fallback for 360 observed seconds. Yes
Missing epochs The staging provenance says HMM and cardio both contributed. Yes
Paired device The staging provenance identifies the vendor labels retained after abstention. Yes
Old backup The night has no staging provenance, so its method stays unknown. Yes
Decisions

No ADR changes. Provenance stays inside the existing sleep payload and uses the existing export, restore and repository seams.

Change size
Area Files Added Removed
Product code 6 80 8
Tests 7 497 3
Docs 1 4 0
Tools 0 0 0
Generated 0 0 0
Total 14 581 11

This stack section only: diff from #756. The 4,000-line size check passes.

Tests

Boat, owned default VM from edge-ready-2, Flutter 3.41.6:

Check Passed Skipped Failed Timed out
Final core, guards, substrate parity and version checks 81 0 0 0
WHOOP freeze, UTC / Kolkata / New York 6 0 0 0
New tests, America/Denver 19 0 0 0
Frozen selected gate, 409 files 4,460 7 7 5
Exact-base controls for the five gate-failing files 74 0 1 0
Fresh feature-head BLE-link file control 30 0 0 0
Supplemental existing HMM tests, head 15 0 5 0
Same supplemental HMM tests, exact base 15 0 5 0

The gate was selected once against 90a879b21 and ran at f5bbee727, before the final review fixes. All 409 files completed; no OOM kills. Its four version-only failures are fixed by exact version expectations and a version-125-specific projection of three tables' version columns to 124. Committed goldens, payload hashes and metric comparisons are unchanged. The final core and zone checks cover those fixes.

One gate sleep-classifier failure reproduces on the exact base. The other seven gate failures/timeouts are in the BLE-link file and do not reproduce in fresh whole-file controls on either base or head. They are reported as non-reproducing gate-only failures, not as inherited. Five additional classifier failures match the exact base by file, test name and assertion. The classifier implementation and those test files are byte-identical to #756.

Analysis: 13 files, no issues. Five mutations produced assertion failures; all five restored controls passed. All 14 final source hashes match Boat. The size gate passes at 592 changed lines. No full-suite or final-head 409-file green claim is made.

On Linux, the existing WHOOP freeze comparison retains its 10-significant-digit policy and excludes stored-byte hashes because of platform libm differences. The new six-minute cardio fixture independently pins the full serialized payload hash captured from #756, with its clock frame fixed to UTC.

Provenance

OpenAI GPT-6.1 Sol, high reasoning, implemented the changes and tests, inspected the reference fork, and answered all ten review questions against the code. The export/read test scaffolding adapts the reference implementation named in the brief. Self-review found and corrected the paired-device provenance case. Self-review also found and corrected a missing flag capture on the imported-substrate path. No independent review round has run in this implementation task; the orchestrator owns external review and publishing. No Co-Authored-By or AI trailers were added.

Summary by Sourcery

Add provenance-aware experimental HMM sleep staging while preserving cardio compatibility and making the staging setting part of derived-data identity.

New Features:

  • Record per-night sleep staging provenance, including the effective label source and seconds retained during HMM fallback.
  • Extend experimental HMM staging with motion, skin-temperature, multi-night calibration, atypical-night marking, and AASM-aligned wake handling.
  • Expose staging provenance through sleep reads and preserve it across selected-day and full database exports and restores.

Bug Fixes:

  • Ensure HMM enablement is captured consistently through preparation and imported-day paths so cache results and worker inputs use the same setting.
  • Prevent malformed or legacy provenance values from being fabricated or causing read failures.

Enhancements:

  • Include the HMM setting in staging cache identity and carry it through prepared derivation inputs.
  • Preserve byte-identical cardio output when experimental HMM staging is disabled.
  • Persist nightly ENMO medians for subsequent multi-night motion calibration.

Documentation:

  • Define sleep staging provenance terminology and effective-method semantics in the project lexicon.

Tests:

  • Add coverage for provenance classification, fallback duration accounting, cache invalidation, export/restore preservation, hybrid HMM behavior, motion baselines, multi-night calibration, AASM wake handling, and cardio payload parity.

Chores:

  • Advance the producing algorithm version to 125 for the changed stored sleep outputs.

mistral-vibe and others added 12 commits November 18, 2026 04:20
Wake was under-detected because the emission used only HR/RMSSD and
'motion_artifact' was hardcoded false — movement never reached the
classifier. Now:

- DayBundleInput carries sleepEnmo (per-second ENMO, g; 0 = absent
  sentinel = neutral, never 'still'); derivation_engine computes it
  from the sleep substrate's accel vector via accelPresentAt.
- Emission is a noisy-OR over autonomic (HR up / RMSSD down), motion
  (vs the night's own ENMO median), and skin temp (vs night median) —
  any strong cue can carry wake, so brief movement-only awakenings
  are caught. Sleep states are penalised by movement.
- Unusable HR/RR falls back to motion-only evidence instead of uniform
  neutral (which used to decode unmeasured-but-moving nights as wake
  via transition ties).
- Transitions less sticky into wake: light->wake 0.05->0.12,
  deep->wake 0.02->0.06, rem->wake 0.05->0.10.
- 5 new hybrid tests (motion-only arousal, quiet sleep, HR arousal,
  absent-ENMO neutrality, motion-only fallback). Still default-off.

Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
…ng-term baseline

Extends the hybrid HMM with per-user multi-night calibration:
- computeMultiNightBaselines() builds a 7-day rolling and a long-term
  baseline (medians) from the persisted nightly rhr/rmssd histories
- the emission now uses a blended effective baseline (nightly + 7d),
  so one restless night no longer dominates the thresholds
- outlier detection: if tonight's baseline deviates >2 robust z (MAD)
  from the long-term baseline, the night is flagged atypicalNight and
  persisted as stager_atypical_night in sleepJson (lower confidence)
- no history -> falls back to the nightly baseline alone
- 4 new tests in test/sleep_hmm_multinight_test.dart

Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
…in temp

Medical grounding (AASM Scoring Manual v2.x):
- an EEG arousal lasts >=3 s and needs >=10 s stable sleep before it; an
  epoch is stage W only when >50% of the 30 s epoch is awake -> wake runs
  shorter than 15 s in our per-second decode are arousals, not wake.
  _hmmRestage now reverts wake runs <15 s to the surrounding sleep stage.
- long wake runs (>=15 s) stay wake and count towards WASO.
- staging after a wake phase continues sequentially (no restart from 0) —
  the Viterbi path already respects continuity; pinned by a test.

Also: skin temp now self-calibrates across nights too (7d rolling + long-term
medians from skinTempAdcHistory, blended into the emission like HR/RMSSD).
ENMO stays nightly-only: no persisted per-night motion history exists yet.

3 new AASM wake-rule tests.

Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
…e opening window

The first 120 min contain settling-in/turning and are systematically more
active, so a motion median taken from them overstated the person's normal
sleep motion and pushed the wake-vs-arousal threshold too high. New
computeMotionBaseline() takes the ENMO median from the middle 50 % of the
night by elapsed time. HR/RMSSD keep the opening-window baseline (autonomic
settling is intended there). Absent ENMO (0 sentinel) is still excluded.

2 new tests.

Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
- transition rows normalized to sum 1.0 (deep light 0.14->0.11, rem self
  0.85->0.80); REM->deep hard-forbid preserved
- fully missing epochs (no HR/RR AND absent ENMO) return uniform neutral
  log(0.25) again, so gaps cannot bias adjacent epochs towards wake
- absent optional sensors are excluded from the wake noisy-OR (factor 0),
  never the 0.5 that silently boosted wake
- ENMO absent sentinel is now -1.0 (measured zero motion is >= 0), so
  epoch means count real still seconds in the denominator
- skinTempHistory forwarded to computeMultiNightBaselines on the prod path
- stager_atypical_night forwarded into the output sleep map
- 2 new neutrality regression tests

Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
…nStrap#593 pattern)

Lying still and awake (low motion, HR clearly above the personal sleep
baseline, RMSSD barely moved) was scored light because the autonomic cue
is an AND of HR-up and RMSSD-down and motion was absent. Added a capped
HR-only term to the wake noisy-OR: sigmoid(hrNorm, 1.22, 0.07) * 0.3.

Guarded against REM (wake-like HR/HRV, Zemaityte et al.): high center
1.22x + cap 0.3 + REM's own emission and 0.80 self-transition keep
REM-like still epochs non-wake — pinned by a regression test. A second
test pins that still + HR ~1.25x reads as wake.

2 new tests.

Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
…ross nights

Closes the last calibration gap: motion now has a persisted per-night
history like HR/RMSSD/skin temp.

- HMM-restaged nights store their stable-core ENMO median as
  stager_enmo_median in sleepJson, forwarded into the bundle sleep block
  and persisted to metric_series as 'enmo_median_night'
  (only for HMM-staged nights, so the series fills from real staging)
- _BaselineHistoryCache gains the 'enmo_median_night' key; _attachHistory
  feeds it as enmo_history into DayBundleInput
- computeMultiNightBaselines builds enmoMedian7d/enmoMedianLong (absent
  sentinel -1 excluded; measured zero motion is a valid 0) and the
  emission blends nightly core + 7d via effectiveEnmoMedian
- 2 new tests

Co-authored-by: BucciMobile <BucciMobile@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@sourcery-ai

sourcery-ai Bot commented Nov 19, 2026

Copy link
Copy Markdown

Reviewer's Guide

Data structures

Add the captured HMM choice to prepared-day data and add sleep ENMO and ENMO history to the bundle input so the worker receives the inputs needed for staging.

  • lib/compute/derive_prepare.dart L23-302 (6 small edits)
  • lib/compute/onehz_pipeline.dart L340-346 (+3 small edits)

Define the optional stored staging-provenance fields and their tolerant typed read boundary, preserving unknown strings while omitting malformed values.

  • lib/compute/staging_provenance.dart L1-17

API

Expose valid staging-provenance fields in the sleep detail repository response without inventing values for older payloads.

  • lib/data/local_repository_impl.dart L23, L1179

Uses of the API

Capture the staging preference for each prepared night, pass it through the normal and imported worker paths, and build per-second sleep ENMO from the substrate.

  • lib/compute/derivation_engine.dart L6903-6915 (+6 small edits)

Implementation

Make the HMM setting part of staging-cache identity so changing the requested method invalidates the relevant cached candidate.

  • lib/compute/staged_sleep_cache.dart L80, L120
  • test/hmm_staging_flag_cache_test.dart L1-96
  • lib/compute/derivation_engine.dart L4458 (removed lines)

Restage with hybrid HR, RMSSD, motion and temperature evidence; add multi-night calibration, wake-run post-processing, and fallback accounting and provenance in the produced sleep bundle.

  • lib/compute/onehz_pipeline.dart L705-736, L2660-2685, L2742-2773 (+10 small edits)
  • lib/compute/sleep_pipeline_hmm.dart L64-157, L194-281, L286-314 (+8 small edits)

Persist HMM-night ENMO medians into metric history and attach that history to later bundle inputs for motion self-calibration.

  • lib/compute/derivation_engine.dart L2686, L7786-7789, L8414-8415

Bump the producing algorithm version so retained nights can be re-derived with the new provenance output.

  • lib/compute/derivation_engine.dart L1896-1899
  • test/ed2a_substrate_transfer_test.dart L244
  • test/ed2b_algo_version_114_test.dart L9

Documentation

Define staging provenance terminology and distinguish requested method, effective method, and observed fallback seconds.

  • docs/lexicon.md L10-13

Tests

Pin cardio byte parity, provenance semantics for whole-night and partial fallback, paired-device cases, stale-key clearing, and tolerant parsing.

  • test/hmm_staging_provenance_test.dart L1-117

Cover repository reads and selected-day and full-export restore of staging provenance, including legacy and malformed payloads.

  • test/hmm_staging_provenance_read_test.dart L1-242

Exercise hybrid emission, AASM wake handling, stable-core motion baselines, and multi-night calibration behavior.

  • test/sleep_hmm_aasm_wake_test.dart L1-90
  • test/sleep_hmm_hybrid_test.dart L1-117
  • test/sleep_hmm_motion_baseline_test.dart L1-30
  • test/sleep_hmm_multinight_test.dart L1-69

Provide the deterministic HMM-night input fixture used to verify worker output and provenance.

  • test/support/hmm_night_fixture.dart L1-23

Keep the WHOOP freeze comparison compatible with version 125 by projecting only version columns back to the pinned version.

  • test/whoop_freeze_golden_test.dart L267-281, L388-389

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants