Skip to content

WIP: preserve YCBT sleep fidelity and explore overnight fragment stitching - #54

Draft
DBozhinovski wants to merge 2 commits into
foureight84:mainfrom
DBozhinovski:fix/ycbt-sleep-session-fidelity
Draft

DBozhinovski wants to merge 2 commits into
foureight84:mainfrom
DBozhinovski:fix/ycbt-sleep-session-fidelity

Conversation

@DBozhinovski

Copy link
Copy Markdown
Contributor

Status: WIP / request for design feedback

This is deliberately a draft PR, not a merge-ready recommendation.

The branch contains a tested implementation of one possible solution, but local hardware evidence exposed an ambiguity in what a YCBT sleep-record boundary means. Before narrowing or finalizing the code, I would like feedback on:

  • whether separate YCBT records should be stitched at all;
  • whether this behavior is specific to the R10M or shared by other YCBT devices;
  • what an unreported interval between records means semantically;
  • how any inferred interval should be represented in PulseLoop and Health Connect.

Problem

PulseLoop currently treats each complete YCBT sleep record as an independent sleep session. The Today tile then surfaces the longest session for the waking day.

On a real R10M FCF4 (wearableModelID=r10m, YCBT, firmware 2.32) connected to a Pixel 7, one complete CRC-accepted 1,640-byte sleep-history block contained three records for the same apparent overnight period:

Record Header bounds Classified duration
A 02:07:46–03:08:32 60.7 min
B 05:01:08–05:51:46 50.6 min
C 06:33:32–08:30:17 116.7 min

Each record had valid header bounds and contiguous timestamped deep/light/REM segments. There were no explicit awake segments covering the gaps.

Observed downstream behavior:

  • PulseLoop displayed only record C: 1h56m.
  • The three records contain about 3h48m of classified sleep.
  • Their overall span is 02:07–08:30, about 6h22m.
  • The wearer independently estimated roughly six hours of sleep.

This suggests the ring may be emitting fragments of one overnight session, but the wire data does not prove whether the two gaps mean:

  1. sleep that the ring failed to classify;
  2. awake or off-finger time inside one night;
  3. deliberate boundaries between separate sleep sessions.

Additional correctness findings

The investigation also exposed less ambiguous pipeline problems:

  • YCBT header start/end timestamps and timestamped segments were flattened into first timestamp + summed stage minutes.
  • Truncated/malformed concatenated records could consume the next record or become authoritative.
  • Independent minute flooring could lose substantial stage duration and short explicit awake transitions.
  • Complete-session reconciliation could leave stage blocks outside their parent Room session.
  • A corrected Room session was not guaranteed a strictly higher Health Connect version.
  • Sleep Health Connect IDs depended on main/nap rank, so a correction could change identity and leave stale records.

Experimental implementation in this branch

The branch currently:

  • preserves and validates YCBT header bounds and timestamped segments;
  • safely recovers from malformed/truncated concatenated records;
  • preserves explicit awake stages and fills uncovered time within accepted bounds as UNKNOWN;
  • quantizes the complete timeline cumulatively at Room's minute resolution;
  • keeps persisted blocks inside their parent session and reconciles corrections idempotently;
  • uses stable v2 Health Connect IDs derived from stable Room session identity;
  • guarantees increasing correction versions;
  • includes a one-time app-scoped Health Connect sleep identity migration;
  • experimentally stitches complete overnight YCBT fragments when all of these hold:
    • same waking day;
    • both start in the overnight window (>=19:00 or <12:00 local);
    • non-overlapping, chronological records;
    • gap <=3 hours;
    • combined span <=16 hours;
    • neither record is partial/non-authoritative.

The stitched gaps are represented as UNKNOWN, while explicit AWAKE remains awake. Daytime naps, different waking days, >3-hour gaps, partial records, and overlapping/duplicate records stay separate.

The generic 60-minute nap/session segmentation behavior for other ring families is unchanged.

Tradeoffs

Stitch as one session with UNKNOWN gaps

Pros

  • Produces a single 02:07–08:30 overnight session for the observed R10M data.
  • Preserves all measured stages while being honest that the gaps are unclassified.
  • Avoids the Today/HA result being determined solely by the longest fragment.

Cons

  • Health Connect and downstream HA aggregation generally count UNKNOWN as sleep, so this can turn missing/off-finger/awake time into reported sleep duration.
  • The current three-hour threshold is evidence-based for one capture, not a protocol guarantee.
  • It may merge genuinely separate early-morning sessions.
  • Including unknown time in total duration can dilute deep-sleep ratios and change sleep scores.

Keep records separate

Pros

  • Makes no inference beyond the ring's explicit record boundaries.
  • Preserves split sleep and nap semantics if those boundaries are intentional.

Cons

  • PulseLoop's single main-session surfaces under-report nights like this one.
  • Summing only the three records still gives 3h48m rather than the user's approximately six-hour experience.
  • Health Connect receives multiple sessions and HA may show only the latest/longest depending on its query.

Device-specific versus YCBT-wide behavior

The safe parser/persistence/identity fixes are general. The stitching behavior is proven only on one R10M firmware. Applying it to every YCBT profile may be too broad; gating it behind a model/profile quirk is safer, but requires deciding which devices share this wire semantic.

Health Connect migration

The proposed v2 identity migration deletes only PulseLoop-owned SleepSessionRecords and rebuilds them from Room. Room and other apps' records are not touched. However, deletion and reinsertion are not atomic, and migration ordering matters: it should not export old fragmented Room rows under new IDs immediately before a corrected sync collapses them.

Questions for maintainers / device owners

  1. What does one YCBT sleep record represent? A complete independent session, a classification fragment, or something device/firmware-specific?
  2. Has anyone captured multiple same-night sleep records from another R10M, TK5, SmartHealth-Colmi/Yawell, or other YCBT device?
  3. Should gaps with no explicit stage count as:
    • sleep of unknown stage;
    • time in bed but not sleep;
    • awake;
    • no data that must not affect sleep duration?
  4. Should the product display both classified sleep and overall overnight span/time in bed instead of forcing one scalar?
  5. If stitching is appropriate, what boundary should define one night: a maximum gap, overnight clock window, same waking day, motion/HR evidence, or another wire field?
  6. Is a three-hour maximum gap defensible, or should the threshold differ?
  7. How should true split sleep and early-morning naps remain distinguishable?
  8. Should stitching be:
    • R10M-only;
    • enabled through a YCBT model/profile quirk;
    • the default for every YCBT device;
    • omitted entirely?
  9. Should sleep scoring use the full stitched duration or only classified non-awake minutes?
  10. For Health Connect, should UNKNOWN contribute to the exported sleep session, or should unclassified gaps be omitted while retaining broader Room/time-in-bed bounds?
  11. Should the Health Connect identity migration wait until a successful post-upgrade ring reconciliation before deleting/rebuilding old records?
  12. What additional captures would make this decision robust (other models, firmware versions, known awake/off-finger intervals, naps, travel/DST)?

Possible directions

  • A. No stitching: keep records separate and change UI/HA projection only.
  • B. Aggregate classified duration: keep separate sessions but show their summed classified sleep for the waking day.
  • C. Stitched time-in-bed: preserve one overnight span but expose classified sleep and unknown gaps separately.
  • D. Model-gated stitching: enable the current experiment only for confirmed R10M variants.
  • E. Evidence-assisted stitching: use adjacent HR/wear/movement evidence rather than a gap threshold alone.

My current preference is C + D: keep the general correctness fixes, gate overnight stitching behind an explicit R10M/profile quirk, and distinguish classified sleep from inferred/unknown time rather than silently presenting the entire span as measured sleep.

Verification

  • Full JVM unit suite: 1,249 tests, 0 failures.
  • :app:assembleDebug: passed.
  • git diff --check: clean.
  • The exact overnight-stitch regression fails when stitching is disabled and passes with the experimental implementation.
  • Local Pixel 7/R10M test reproduced the three raw records and the 1h56 dashboard result.
  • The experimental stitched build was installed locally, but final post-fix Room/Health Connect readback is intentionally not claimed yet.

Scope / non-goals

  • This PR does not address the separate Health Connect step-source overlap/doubling issue.
  • No production release or migration has been performed.
  • Please do not merge while the stitching semantics and Health Connect migration ordering remain unresolved.

@foureight84

Copy link
Copy Markdown
Owner

@DBozhinovski apologies for letting this slip off our radar for so long because of the WIP draft status — this is a super thorough investigation, and the real R10M capture + questions you laid out caught several real pipeline bugs.

A lot has happened in the sleep pipeline on main since August (now shipped in v2.12.0+90), so we now have real multi-device hardware evidence answering most of your design questions, plus a few genuine bugs you found here that are still unfixed on main and would be great to land.


1. Answers to your design questions (from hardware evidence on main)


2. What's still unfixed on main from this PR (would love to land!)

Several of your general correctness fixes are still missing on main (v2.12.0+90) and are clean, standalone wins:

  1. Truncated/malformed af fa boundary hardening in YCBTHealthRecords.sleep: main resyncs on af fa between records, but inside the segment loop it doesn't break on an embedded af fa header (so a truncated record with an inflated recordLength can eat 8 bytes of the next record's header and drop it) and still emits completeSession = true when segmentCount < declared or (recordLength - 20) % 8 != 0. Your validDeclaredWidth + nextRecordBoundary + completeSession = useHeaderBounds checks would directly harden that.
  2. Strictly monotonic updatedAt (nextSleepUpdatedAt): EventPersistenceSubscriber.reconcileWakingDay on main still writes raw updatedAt = now, so same-ms reconciles or a backward clock step can fail to bump Health Connect's clientRecordVersion (or skip watermark re-selection). maxOf(now, existing.updatedAt + 1L) is a great zero-risk fix.
  3. Health Connect stable v2 sleep identity (pl-sleep-v2-<uuid(session.id)>) + one-time migration (optionally paired with explicit STAGE_TYPE_AWAKE for between-record gaps in SleepExporter): SleepExporter.kt on main still uses rank-dependent pl-sleep-<day> / -1 IDs (sleepSessionSuffix), so main/nap rank flips can still leave stale duplicates.

Suggested next steps

Since main has diverged heavily from b33c89c over the last three months, rather than rebasing this whole draft, we'd suggest splitting the still-needed pieces into 1–2 small, focused PRs off current main (without decoder-level stitching):

  1. YCBTHealthRecords.sleep truncated-record/af fa-resync & completeSession validation.
  2. Monotonic nextSleepUpdatedAt + Health Connect stable v2 sleep IDs (and optionally STAGE_TYPE_AWAKE for between-record gaps).

Happy to review if you're still interested in picking those up — and if you don't have time right now, just let us know and we can open tracking issues crediting your findings here. Thanks again for the great write-up!

@foureight84

Copy link
Copy Markdown
Owner

@DBozhinovski I went ahead and merged main (fdb95ef) into this branch (normal merge commit, no force-push — your original commit 59fb400 is intact in history) so it's conflict-free and green against current main.

To align with the post-August sleep pipeline on main (#63, #68, #78, #81, #82, #105, #106, #111, #126), the conflict resolution drops decoder-level stitching / UNKNOWN-gap filling and keeps the non-superseded fixes from your PR:

  • YCBTHealthRecords.sleep: truncated/partial-width/af fa-preamble resync, exact 24 h header bound, and completeSession validation (preserving main's zero-length segment handling from Only one sleep session per night is shown when the ring records several #63).
  • RingEventBridge: future-end check on sleep timelines.
  • EventPersistenceSubscriber.reconcileWakingDay: strictly monotonic nextSleepUpdatedAt.
  • Health Connect stable v2 sleep IDs (pl-sleep-v2-...) + one-time migration (HealthConnectExporter, HealthConnectPrefsStore, HealthConnectTypeMappings, SleepExporter).

Verified ./gradlew :app:testDebugUnitTest passes cleanly (0 failures). Left this as a draft so you can review, adjust (e.g., if you also want explicit STAGE_TYPE_AWAKE for between-record gaps in SleepExporter or any migration tweaks), and mark ready for review whenever it looks good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants