diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index a43168809..728688b48 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -29,6 +29,7 @@ jobs: - { name: "Lint", cmd: "CHANGED=$(git diff --name-only --diff-filter=d origin/${BASE_REF#refs/heads/}...HEAD -- '*.ts' '*.tsx' | head -200); if [ -n \"$CHANGED\" ]; then echo \"$CHANGED\" | xargs -d '\\n' pnpm eslint --; else echo 'No TS/TSX files changed'; fi" } - { name: "Typecheck", cmd: "pnpm tsc" } - { name: "Unit Tests", cmd: "pnpm test run --coverage --passWithNoTests" } + - { name: "Browser Tests", cmd: "NODE_ENV=production CI=1 pnpm build && pnpm test:browser" } steps: - name: Checkout @@ -68,6 +69,10 @@ jobs: working-directory: modules/piezo-processor run: uv sync --locked --no-install-project + - name: Install browser + if: matrix.task.name == 'Browser Tests' + run: pnpm exec playwright install --with-deps chromium + - name: Cache Assets (Vitest) id: asset-cache uses: actions/cache@v5 @@ -108,3 +113,10 @@ jobs: files: test-results/junit.xml flags: javascript + - name: Upload browser evidence + if: matrix.task.name == 'Browser Tests' && !cancelled() + uses: actions/upload-artifact@v6 + with: + name: calibration-browser-results + path: test-results/browser + retention-days: 7 diff --git a/docs/biometrics-backtest-jobs.md b/docs/biometrics-backtest-jobs.md new file mode 100644 index 000000000..4bfd7c57d --- /dev/null +++ b/docs/biometrics-backtest-jobs.md @@ -0,0 +1,262 @@ +# Durable biometric backtest jobs + +`BacktestJobStore` persists experiments, immutable evaluation inputs, leases, +progress, cancellation, results, and holdout exposure in `biometrics.db`. +It is the job repository; execution, API/UI integration, and production worker +resource validation are separate parts of the calibration implementation. + +Enqueue pins the retained dataset hash, protocol hash, exact resolved +configurations, engine manifest hash, and selected split. Duplicate idempotency +keys return the original job, while changed requests fail. Hash or engine +mismatches are rejected. Configuration snapshots are limited to 64 KiB. + +Only one unexpired running or cancellation-pending job can hold the worker +slot. Claim increments both the attempt and fencing token. Every heartbeat, +completion, failure, and cancellation acknowledgment verifies owner, token, +state, and expiry under a short SQLite transaction. An expired worker cannot +publish after replacement. Recovery marks abandoned work interrupted and +allows at most three attempts; abandoned cancellations become cancelled. + +Cancellation of queued work is immediate. Running work exposes a cancellation +request through heartbeat, and cannot publish a successful result afterward. +The worker supervisor must terminate unresponsive execution after its grace +period. The repository itself does not execute code or control processes. + +A result and its terminal job state commit atomically. Summaries are bounded +to 16 MiB each and 64 MiB retained total; job and experiment histories each stop at 1000 +entries. These repository bounds do not replace worker CPU, memory, wall-time, +free-space, or live-ingest safeguards. They require target-pod measurement +before enabling production replay. + +Holdout exposure is recorded on enqueue, conservatively including jobs later +cancelled or failed. It follows physical side and overlapping recorded time +ranges across datasets and experiments. Creating another dataset or changing +its asserted subject name does not erase exposure. Repeated access is labeled +`reused_holdout_is_validation`. Idempotent retries of the same enqueue do not +count twice. Future promotion must recheck current exposure, not rely solely +on the count captured in an older report. + +The current evaluation protocol is explicitly descriptive. This repository +does not award validated improvement; that requires the complete evaluator, +reference support, coverage-aware policy, independent nights, and operational +validation. No backtest state activates a profile. + +## Shipped processed-data worker + +`evaluator.ts` executes comparison and stage replay over the frozen selected +split. Overlapping session rows are deduplicated by original ID; overlapping +windows merge and disconnected sessions are classified separately. It calls +the existing retrospective classifier with the exact resolved rules and the +protocol's explicitly assumed calibration quality. Default/current snapshots +with identical hashes execute once. Fixed clock offsets also apply to stage +scoring. Original unshifted stage traces remain available in the report. + +Numeric comparisons use original stored readings. They do not evaluate a +candidate's waveform-processing parameters. Stage comparisons report +interval-duration agreement, not the chart's older midpoint agreement. +Reports remain descriptive/insufficient-evidence until the promotion policy, +reference support, common-support checks and operational qualification are +complete. Merged replay windows are currently limited to one day, and complete +reports up to 2 MiB remain inline. Larger reports use checksummed files beside +the biometrics database, with a small descriptor in SQLite. The complete report +is available through the same result reader, with both byte and canonical-data +hash verification; it is never truncated to fit the inline limit. + +`pnpm build` now bundles the worker into +`.next/standalone/biometrics-worker/worker.cjs`; `pnpm build:biometrics-worker` +builds it independently. Packaging rejects database, hardware, automation, +scheduler, HomeKit and streaming dependencies. Shared schemas live in the pure +`contracts.ts` module so the worker does not import repositories. Enqueue pins +the SHA-256 of the actual worker bundle, and the runner checks it before launch. + +`executeNextBacktest` runs one leased attempt after an injected host admission +check. It does not start itself or enable production replay. The host must wire +measured resource/ingest admission and lifecycle handling. `runBacktestChild` +uses a separate low-priority Node process, a clean environment without pod +credentials, a 128 MiB JavaScript heap limit, bounded stdin/stdout/stderr, a +wall-time watchdog, and cancellation with SIGTERM followed by SIGKILL. This is +isolation for shipped trusted code; it is not the external sandbox required +for user-proposed algorithm packages. The heap limit is not a total RSS cap. +The child records evaluation runtime and peak RSS for operational inspection. + +The runner heartbeats independently of the child's progress. It rechecks resource +admission every five seconds and before publication; denied or unavailable health +evidence aborts the child and records a resource interruption under the same +bounded retry policy. Checks never overlap. Shutdown leaves +owned unfinished jobs interrupted for bounded retry. Cancellation is +acknowledged only after the child has stopped, and old attempts cannot publish. +Result reads verify their hash and winning fencing token and return the current +holdout exposure count, so later reuse remains visible. + +## Large report storage + +`ReportArtifactStore` stages a uniquely named file, fsyncs it, atomically renames +it and fsyncs the directory before a fenced database transaction publishes the +descriptor and successful job state. File paths derive from the server-owned +database directory, job UUID, attempt token and content hash; request filenames +never select a path. An in-memory database needs an explicitly injected artifact +store for large-report tests. Existing inline results remain readable. + +A failed database commit can leave an unpublished file, but cannot expose a +partial result. Staged files count toward storage checks. The collector removes +only owned files that are unreferenced and older than one hour; it preserves +committed descriptors and conservatively retains future-version named references +during downgrade. A separate SQLite publication lock serializes quota checks, +file publication, and collection across processes. File writes hold only this +sidecar lock; the live database transaction remains limited to fenced publication. +A terminated process releases the lock automatically. Identical staged retries +reuse the verified file without counting its bytes twice. + +`BacktestHost` provides a serialized, recoverable five-second polling loop and +orphan collection every thirty minutes. Shutdown aborts and awaits the current +attempt before database closure. Its global singleton requires an explicit +admission callback. Production startup and measured resource admission remain +pending; merely importing the host does not start replay. + +## Admission evidence + +`admission.ts` defines a deployment-policy contract with a qualification ID and +explicit limits for memory reserve, free disk, one-minute load per available CPU, +current event-loop delay, piezo age, temperature and sample age. It has no shipped +threshold defaults yet. Missing or malformed policy yields `unqualified`; missing +or malformed health evidence yields `health_unavailable`. Fresh evidence must +meet every limit. Future timestamps and stale samples also defer replay, and the +decision returns all applicable pressure reasons for later API/UI exposure. + +The Linux collector uses `MemAvailable`, filesystem `bavail`, normalized load, +thermal zone zero and a twenty-millisecond scheduling probe in the Node host. +The probe measures current scheduling delay rather than using the lifetime +performance histogram. Its supplied timestamp callback must read specifically +`getSensorFrameTimes()['piezo-dual']`, never the newest arbitrary sensor frame. +Unknown piezo freshness, missing thermal data, read failures or unsupported +platforms yield unavailable evidence. Collection makes no hardware commands or +database queries. These helpers are not yet connected to production startup; +measured thresholds and full publication-load qualification remain required. + +## Explicit report release + +Trusted deployment configuration may set `SLEEPYPOD_BIOMETRICS_REPLAY_BUDGET` +to JSON containing `jobWallMs`, `waveformWallMs`, and `waveformCpuSeconds`. +These are operational ceilings, not candidate parameters or RPC inputs. The +coordinator validates the complete budget before spawning: the job wall limit +is at most 48 hours, each waveform process at most one hour, and CPU time at +most 3,600 seconds. CPU time must fit within waveform wall time, which must fit +within job wall time. No production values are inferred from those ceilings. + +Only execution tasks receive the extended job deadline. Read, publication, +inspection, and cleanup deadlines remain unchanged, as does the Node scorer's +separate deadline. Python applies the selected CPU limit before importing the +numerical implementation. Without explicit configuration, existing short +deadlines and the 120-second CPU limit remain. Changing this environment setting +requires restarting the coordinator service; it does not start the worker host, +grant resource admission, or establish hardware qualification. The synthetic +waveform benchmark accepts the same setting for qualification runs. + +The coordinator reserves a separate process slot for read-only report, trace, +dataset-descriptor, and raw-catalog requests. At most one reader and one work +process run at once. Replay, scanning, publication, release, and cleanup share +the serialized work slot. Each lane accepts at most seven active-plus-pending +requests, with a combined limit of eight, reserving capacity for the other lane. +Pending requests expire after 30 seconds; shutdown cancels both active children +and rejects pending work. This prevents a long replay from consuming report-read +capacity. Resource qualification must include the additional reader's memory +and CPU, and execution time budgets remain independently enforced. + +Completed results support an exact-hash, idempotent release. The release +transaction replaces payload bytes with an empty stored object and records +`released_at_ms`, preserving the original content hash and completion/attempt +metadata. Readers return a null payload with explicit unavailability. Report +reads and releases share the publication lock, preventing file deletion between +descriptor lookup and byte verification. After commit, an unreferenced artifact +is unlinked; failed file deletion leaves an orphan counted against disk quota +until collection can remove it. A failed database transaction never deletes the +file. Referenced artifacts are retained conservatively, including future-format +named references. There is no automatic age-based expiry of committed reports; +release is an explicit user action through the shared API. + +## Paired stage support + +Each stage-replay night now includes comparisons of current versus default and, +when supplied, candidate versus default and current. Each window preserves the +configuration hashes and reports agreement, macro F1 and coverage on each +configuration's own support and on the exact intersection of their predicted +intervals. Conflicting or unknown reference intervals remain excluded by the +shared duration scorer. Common-support confusion matrices are retained; absent +paired support yields null metrics, never zero error. Fixed clock offsets apply +to both outputs for scoring without modifying the original displayed traces. + +A separate coverage-reduction flag and duration/coverage differences expose +candidates that omit difficult intervals. For example, omitting the incorrect +half of a prediction may raise its own-support agreement from 50% to 100%, but +cannot produce an agreement gain on the remaining common support when its +labels are unchanged. This is descriptive reporting only. It does not yet +implement the frozen promotion policy, per-night uncertainty aggregation, or +validated activation. Numeric waveform candidates still require raw replay. + +This comparison adds evaluator work and report bytes. Earlier target benchmark +numbers apply only to their recorded worker hash; repeat qualification with the +final worker before enabling production admission defaults. + +## Whole-night stage summaries + +`stageNightSummary` groups comparisons by baseline/candidate labels and exact +configuration hashes. It first sums confusion durations across each cohort's +nonoverlapping windows, then computes nightly agreement and per-stage +precision/recall/F1. It does not average window F1 values. Both full-window +output coverage and eligible-reference coverage remain available per night. +Duplicate cohorts/comparisons and overlapping windows are rejected. + +Each comparison reports scored and unscored night counts, nights with positive, +negative or zero agreement differences, and nights with any coverage reduction. +The equal-night mean agreement difference gives every scored whole-night cohort +one vote. Pooled duration confusion and agreement are reported separately and +can differ substantially when night lengths differ. Missing reference support +remains null and is counted as unscored, not converted to a zero difference. +The observed nightly minimum/maximum is explicitly a descriptive range, not a +confidence interval. The optional seeded interval is reported separately below. The frozen promotion +policy remains required before a report can establish validated improvement. + +## Reproducible exploratory uncertainty + +Experiment protocols may now include `nightUncertainty` with version +`paired-night-percentile-v1`, an explicit seed (1–64 characters), resample count +(200–5,000), and minimum scored nights (3–60). These fields participate in the +immutable protocol hash. Older protocols remain readable and report +`not_requested`; no parameters are silently added to their hashes. + +The statistic is the equal-night mean of paired common-support agreement +differences. Whole-night cohorts are sorted by ID, then resampled with +replacement, preserving the baseline/candidate pairing. A SHA-256 counter +stream over JSON `[version, seed, counter]` supplies big-endian 32-bit draws; +rejection sampling removes modulo bias. The two-sided 95% percentile interval +uses linear interpolation at ranks `(resamples - 1) * 0.025` and +`(resamples - 1) * 0.975`. This is a versioned local implementation, not a call +to SciPy. The general resampling procedure and pairing are described in +[the SciPy bootstrap reference](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html). + +Missing-reference nights are counted and omitted from the statistic. Too few +scored nights return `insufficient_nights`, and identical observed differences +return `degenerate`, both with a null interval. The requested minimum is a +reporting threshold, not a claim of adequate statistical power. Intervals assume +exchangeable nights; serial dependence and unobserved nights can make the +estimate misleading. Reports retain these limitations and remain +`insufficient_evidence` for promotion. Objective/coverage/operational policy and +held-out qualification are still required for validated activation. + +## Shared support across all submitted configurations + +Saved stage replays compute one intersection across default, current and any +candidate for each evaluation window, after applying the fixed clock offset. +Every pairwise comparison uses that same mask and identifies its +`commonSupportScope` as `all_configurations`. If any configuration has no output, +the shared mask is empty and common-support agreement is null. The configuration +is never silently omitted from the intersection. Each output's own coverage and +agreement remain separate, preserving visibility into lost output and references. + +The interval sweep clips to the evaluation bounds, preserves gaps and labels, +and rejects overlapping or malformed input intervals. The standalone two-output +comparison helper still supports explicitly labelled `pair` scope, but saved +replay reports always supply the all-configuration mask. Nightly summaries and +uncertainty therefore use the same scored intervals for every comparison in a +run. The output-availability policy and validated promotion gate remain separate +requirements; common support alone does not establish improvement. diff --git a/docs/biometrics-calibration-api.md b/docs/biometrics-calibration-api.md new file mode 100644 index 000000000..c6909e213 --- /dev/null +++ b/docs/biometrics-calibration-api.md @@ -0,0 +1,530 @@ +# Biometric calibration API + +The `biometricTuning` tRPC router and `/api/biometric-tuning/*` OpenAPI routes +share the same services. They use the existing LAN trust model. Supplied +requester names are attribution, not authenticated identities. + +## Inspect and create profiles + +- `getManifest` returns the versioned default, parameter schema and current + manifest hash. +- `getProfileOverview` separates desired configuration, the existing stage + view, sensor-calibration freshness, and observed runtime receipts. No receipt + means unreported, not applied. A legacy stage customization awaiting migration + remains visible. +- `resolveProfile` validates explicit leaf/ordered-array overrides and returns + the resolved snapshot, hash and override provenance without saving anything. + Omit a path to restore inheritance; null does not mean remove. +- `createProfileRevision`, `getProfileRevision`, and `listProfileRevisions` + create and inspect immutable candidates. Creating one does not bind it or + write an activation outbox event. Public origins are manual or agent; + migration/system origins are reserved for server workflows. + +Manual activation is available. Desired profile state must not be presented as +actual runtime adoption; availability means requests can be queued, not that a +processor is currently healthy or has adopted a particular revision. + +## Retain data and submit work + +`listSessions` is cursor-paginated and side-specific. `getSessionCapabilities` +reports actual processed timestamp coverage, gaps, reference metric counts, +and unknown processing provenance. Raw waveforms are marked not inventoried; +the presence of processed readings does not imply signal replay capability. +Reference counts include revisions, so dataset creation requires explicit +import IDs and accepts only one revision per family. + +`createDataset` takes side, asserted sleeper ID, session IDs, reference import +IDs, and an idempotency key. The server pins the configured device timezone. +`getDataset` and `listDatasets` return metadata/manifests, not the retained raw +rows. Snapshot integrity is verified on a dataset read. Reference deletion and +observation retention do not change an existing snapshot. + +`createExperiment` pins a dataset and descriptive protocol: evaluation mode, +explicit assumed stage-calibration quality, and fixed alignment tolerance and +clock offset. `enqueueBacktest` takes the experiment ID, exact candidate +revision ID, expected binding generation, split, and idempotency key. Callers +cannot supply default/current snapshots, evaluator hashes, or executable paths. +The server resolves them. The current stage baseline comes from the actual +retrospective view, not a pending desired vitals profile. + +The current evaluation scopes are original-reading comparison and stage rules +on original readings. Waveform parameter changes are not evaluated by these +scopes. The job descriptor states this explicitly. + +`listBacktests`, `getBacktest`, `cancelBacktest`, and `getBacktestResult` expose +bounded job history, progress, cancellation and verified results. Enqueue +persists work; an admitted worker host must execute it. A queued response is +not a completed result or an activation. Result reads include the current +holdout exposure count. Existing successful retries return the original job +when the same public submission key and payload are supplied. + +`getBacktestSummary` (`GET /biometric-tuning/backtests/{id}/summary`) is the +interactive review path. It verifies the complete report in the coordinator, +then omits stage-trace rows before IPC and HTTP serialization. All scores, +per-night comparisons, support, provenance, and operational measurements remain. +Its `projection` is `scores_without_stage_trace_rows_v1`; `contentHashScope` is +`complete_retained_report`. The hash identifies the retained full artifact, +not the summary's serialized bytes. Summaries exceeding 2 MiB fail explicitly +rather than silently truncating nights or scores. `getBacktestResult` continues +to return the full verified report, and `getBacktestTrace` pages exact trace +rows using that same retained hash. The UI and MCP report reader use the summary +path; release and holdout exposure semantics are unchanged. + +The backtest host loop, measured resource admission, richer promotion reports, +and full client UI/MCP integration remain separate implementation steps. The API does not claim that these are enabled merely because job +submission exists. + +Profile diffs report both changed resolved values and changed override intent, +including explicit-default overrides whose numerical output is unchanged. +`validateProfileRevision` checks schema/engine compatibility only; it explicitly +leaves evaluation evidence unchecked. + +Experiment creation requires an idempotency key. `getExperiment` and +`listExperiments` preserve access to its frozen protocol and audit identity. + +`releaseDataset` is explicit and idempotent. It refuses release while queued, +running, cancellation-pending, or interrupted jobs still need the inputs. +After cancellation or completion, release removes the retained input snapshot, +keeps its manifest/hash and holdout audit metadata, and marks it unavailable. +Existing reports remain separate retained results. New runs cannot use a +released dataset; a fresh snapshot requires a fresh creation key. Storage +accounting continues to include the retained manifest. Active configuration +lives in the configuration database and is unaffected. + +Canonical dataset/report hashing uses the same finite-number encoding as +configuration hashing, with a separate bounded budget for realistic multi-night +rows. Configuration limits remain unchanged. Dataset hashes are checked when +reading retained input data, and metadata reads verify available snapshots; +released metadata remains readable without pretending inputs still exist. + +Activation history is available through `listActivationHistory` (side and +optional `beforeGeneration` cursor). Pages contain at most fifty events and +include fallback/default-update evidence. Profile overview separates its +historical receipts from `runtimeHealth`, which reports current numerical +heartbeat freshness and configuration consistency. General and legacy stage +settings use the same versioned intent and acknowledgment protocol. + +## Manual activation and observation + +- `activateProfile` / `POST /biometric-tuning/activate` selects an exact revision. +- `resetProfile` / `POST /biometric-tuning/reset` selects current shipped defaults + without deleting revisions or audit history. Malformed unmigratable legacy + stage settings are retained in the legacy row and reset audit evidence; valid + unmigrated customizations must first be preserved as a revision. +- `rollbackProfile` / `POST /biometric-tuning/rollback` selects the previous + binding, including its exact revision identity (null means the default binding). + +All three require side, expected binding generation, expected resolved hash, +immediate/next-session boundary, requester, idempotency key, and explicit +`allowUnvalidated: true`. These are deliberate manual changes, recorded as +`manual_unvalidated`. They do not assert that a backtest established improvement. +A stale binding, changed idempotent request or mismatched hash is a conflict. +Creating or evaluating a candidate never calls these mutations implicitly. + +`getActivationStatus` / `GET /biometric-tuning/activation-status` accepts a side +and optional event ID. It returns the persisted audit event, observed delivery +and matching-receipt state, receipt freshness and whether the event still +belongs to the desired generation. Polling is read-only and cannot acknowledge +the stage consumer. The persisted event can lag observed receipts until the +five-second reconciliation pass. An applied old event is history, not proof it +is still selected; check `isCurrent` and the overview's current runtime health. + +The cross-runtime integration test uses temporary fixture databases and the +locked piezo environment. Prepare it with +`cd modules/piezo-processor && uv sync --locked --no-install-project`, then run +`pnpm exec vitest run src/biometrics/profiles/tests/runtime-integration.test.ts` +from the repository root. The JS unit-test CI job installs that same environment. +The test runs the shipped Python adapter with a synthetic frame, verifies the +next-session boundary, TypeScript stage receipt, restart restoration and rollback; +it neither starts a sensor follower nor accesses hardware. + +## Releasing completed reports + +`releaseBacktestResult` (`POST /biometric-tuning/backtests/{id}/result/release`) +requires the job ID and `expectedContentHash`. It applies only to successful +jobs and rejects a mismatched hash. Repeating the same release is idempotent. +The result endpoint now includes `available` and nullable `releasedAtMs`. +After release it returns `available: false`, `report: null` and the original +content hash, completion time, winning fencing token and current holdout count. +A released report is distinct from a job that has not completed (which still +returns null as the whole result). + +Release removes inline payload bytes or detaches an artifact descriptor in a +transaction before deleting the file. It does not erase experiments, jobs, +holdout exposure, profile revisions or activation history. A retried enqueue +with the same idempotency key still resolves to the original successful job; +it cannot recreate released bytes or erase previous holdout use. A new replay +requires a new request key and available retained inputs. + +The additive biometrics migration `0027_release_backtest_reports` preserves +existing reports with a null release timestamp. Older readers cannot validate +a released payload against the original hash, so they report an unavailable or +corrupt result rather than silently displaying different data. UI/MCP clients +must use the explicit availability fields when presenting released evidence. + +## Paginated stage traces + +`getBacktestTrace` (`GET /biometric-tuning/backtests/{id}/trace`) requires +`expectedContentHash` and `configuration` (`default`, `current`, or `candidate`). +It accepts an optional `cohortId`, an `offset` (default zero), and `limit` +(default 200, maximum 500). The hash must match the verified retained result. +Unfinished jobs reject trace requests, changed hashes return a conflict, released +reports return not found, and comparison-only runs report that no stage replay +was produced. Missing cohorts/configurations and cursors beyond the selected +trace are errors rather than fabricated empty results. + +The response includes exact intervals, configuration hashes/revisions, cohort +and source-window identities, the total selected interval count, and +`nextCursor: { offset, contentHash }` or null. Reuse that offset and hash as +`offset` and `expectedContentHash`, keeping the selection unchanged. No +intervals are downsampled or interpolated. Trace timestamps precede the fixed +reference-alignment offset, which is returned separately as `clockOffsetMs`. +The provenance is explicitly `versioned_stage_replay`, not original observed +stages. Filtering/pagination never changes evaluation scores. + +This bounds response detail, not artifact-read cost: the service currently +verifies the entire retained report before selecting a page. Parent-process +artifact parsing and hashing remain part of the target performance qualification. + + +## Retained waveform discovery and publication + +`getReplayAdmission` (`GET /biometric-tuning/replay-admission`) reports whether +server policy is configured, its operator-supplied qualification ID, and current +admission reasons. `SLEEPYPOD_BIOMETRICS_ADMISSION_POLICY` is server-owned JSON +matching `admissionPolicySchema`; missing or malformed policy denies admission. +Supplying an ID does not itself qualify hardware. Each check uses fresh process, +storage, thermal, and live piezo receipt evidence; no caller can supply health +samples or override admission through an RPC parameter. + +`inspectRawSource` (`POST /biometric-tuning/raw-sources/inspect`) accepts only +`afterName`, initially null, and inspects at most one server-owned archive. +Persisted evidence is a mutation even though scanning does not modify the +archive. Reuse `nextCursor` until `complete`; `source_unavailable` advances past +an unreadable source, and `archive_unavailable` reports a missing archive root. +`deferred` preserves the cursor and means no new scan was admitted. This API +accepts neither filesystem paths nor executable code. + +`listRawSources` (`GET /biometric-tuning/raw-sources`) accepts `startMs`, `endMs`, +and optional `afterId`, returning at most fifty descriptors and a cursor. Query +all selected cohort windows including five minutes of pre-roll. Catalog event +bounds indicate possible overlap, not continuous support. Descriptors explicitly +say `requires_reverification`; inventory alone has not retained any waveforms. + +`createWaveformDataset` (`POST /biometric-tuning/waveform-datasets`) accepts a +parent dataset ID, its exact `expectedContentHash`, an idempotency key, and +`cohorts: [{ cohortId, sourceIds }]`. The server derives context from that frozen +parent, re-verifies selected archives, and atomically retains inputs and publishes +the immutable dataset. New publication requires ongoing admission; a `deferred` +response contains no dataset. A retry of already committed publication returns +the original descriptor even when current admission is unavailable or original +archives have disappeared. Changed selections under the same key conflict. +Released descriptors remain released; retries never recreate their bytes. + +Create a `signal_replay` experiment using the published dataset, then submit and +poll a durable backtest. Publication does not start a worker host or activate a +profile. Host qualification and scheduling remain separate operational gates. + + +The retained-dataset UI exposes **Select recorded waveforms** on available +processed datasets. It inspects one archive per explicit request, keeps scan +cursors on deferral, and preserves source selections across pages and nights. +Every parent cohort requires a selection, with at most 64 source IDs per cohort. +Publication pins the parent content hash and reuses its request key after a lost +reply or deferral. Released publication retries are shown as unavailable. + +Published waveform datasets offer **Replay retained waveforms** in the protocol +editor. Sleep-stage replay is optional and requires an explicit calibration +quality assumption; numerical-only replay does not. Protocol creation and queued +submission still depend on the separate worker host for execution. The browser +demo contains no RAW archive fixture or qualified pod telemetry, so its waveform +operations remain explicitly deferred. + +### Numeric objective policies + +Signal-replay protocols may pin an `objectivePolicy` with version +`numeric-objective-v1`. Its metric, minimum supported nights and paired references, +maximum lost-observation fraction, minimum MAE improvement, night and subgroup +regression tolerances, required subgroup names, and operational limits are included +in the protocol hash. Processed-reading protocols reject numeric objectives. +Reports retain the thresholds and per-baseline `objectiveAssessments`. + +Caller-supplied thresholds are not a designated promotion policy. The current +production evaluator has no designated policy and does not yet attach measured +operational or subgroup evidence, so its assessments remain insufficient evidence. +The pure assessor checks an independent first-use holdout, supported whole nights, +uncertainty, coverage, subgroup regressions, and measured operational limits before +it can award improvement under a designated policy. Test-only thresholds exercise +boundaries; they are not release defaults or evidence of clinical validation. + +### Replay host startup + +After migrations and hardware initialization, production startup reads +`SLEEPYPOD_BIOMETRICS_ADMISSION_POLICY`. A missing or invalid policy leaves the +background host stopped. A valid policy starts the existing singleton host; +startup does not assert that hardware has been qualified. Its execution and +cleanup operations still obtain fresh admission, including live piezo receipt +age. CI skips this startup with the other hardware-dependent services. Shutdown +aborts and awaits the host before closing the coordinator. + +Deploy only a policy backed by recorded pod measurements. Configure the matching +`SLEEPYPOD_BIOMETRICS_REPLAY_BUDGET` before process startup; the coordinator captures +that budget when its singleton is created. Changing process environment requires +a service restart to start a previously disabled host or replace that budget. + +New waveform dataset manifests advertise retained-input replay capability. +Execution availability remains a separate, current admission query. Older +immutable manifests and their hashes are not rewritten by this change. + +### Exact numeric trace pages + +`GET /api/biometric-tuning/backtests/{id}/numeric-trace` (tRPC +`getBacktestNumericTrace`, MCP `backtest_biometrics` action `numeric_trace`) +returns up to 500 exact recomputed readings per page. Supply the immutable report +`expectedContentHash`, configuration label, optional cohort, and offset/limit. +The next cursor carries the same hash. Released results and stale hashes fail; +paging does not rerun an experiment or change its scores. + +Pages contain only the report dataset's side, within its scoring windows. They +preserve null readings, quality, flags, routing provenance, and the original +replay timestamp; `clockOffsetMs` is metadata for reference alignment and is not +applied to the returned timestamp. Opposite-side readings and internal `hrRaw` +values are excluded. No downsampling is performed. Historical reports without +explicit side provenance cannot supply this projection. Numeric trace UI and +reference overlays remain separate consumers of this API. + +Waveform reports expose these pages through **Inspect numeric traces**. Select a +night, configuration, and metric; the chart shows only that page's individual +readings. The exact-value table includes missing values, quality, flags, source +channel, and limitations. Changing night/configuration resets the cursor. The +processed-only demo does not fabricate waveform readings for this view. + +### Frozen reference pages + +`getBacktestReferenceTrace` / `GET .../backtests/{id}/reference-trace` and MCP +`backtest_biometrics` action `reference_trace` require a report hash, cohort, +metric, and bounded offset/limit. They read the immutable dataset selected by +that report, verify its hash, and expose the frozen import revision, source, +normalized measurement, compatibility reason, and original reference times. +They never resolve the current revision of an import. Original submitted payloads, +annotations, and subject fields are excluded. + +These are display records, not newly matched pairs or independent evidence. +Point/interval and aggregation semantics remain explicit. HRV-family records +with an incompatible definition remain inspectable with their compatibility +reason. Scoring-window boundaries are preserved. A released report or dataset +makes this detail unavailable; it does not silently fall back to live data. + +In a waveform report, open **Inspect numeric traces**, then **Compare frozen +references**. The overlay pages references independently from recomputed readings +(200 records per page), preserves point and interval shapes, and applies the +experiment's fixed clock offset only to recomputed timestamps. The source table +keeps reference timestamps in UTC. Incompatible definitions remain visible in the +table but are not plotted. Display overlap does not establish a new scored pair; +use the report's frozen alignment and common-support scores for comparisons. + +### Numeric subgroup evidence + +Waveform experiments with a numeric objective additionally compute +`reference-defined-numeric-subgroups-v1` for that objective's metric. Fixed IDs +are `reference_points`, `reference_intervals`, `window_first_half`, and +`window_second_half`. Membership uses native reference timestamps and temporal +kind, never candidate values, errors, or missingness. Each disconnected scoring +window has its own midpoint. A point at the midpoint belongs to the second half; +an interval ending there belongs to the first. Intervals spanning the midpoint +are excluded from both half-window groups without splitting or changing their +aggregation; the report retains their excluded count. + +Each group uses the same all-configuration common timestamp support as the main +numeric comparison. References are weighted within a night, then nights receive +equal weight. Objective evidence requires the configured minimum paired records +in every included night; an unsupported night makes that group's mean unavailable +for promotion rather than silently dropping it. A required group with inadequate +night support blocks improvement. Per-window observation coverage counters still +refer to complete scoring windows, not inferred continuous coverage within a +reference subgroup. The report UI shows subgroup means, night counts and boundary +exclusions. Groups overlap and are not independent statistical experiments. +They do not establish motion, pump, empty-bed, or artifact-specific performance; +those require separately qualified context/fixture evidence. Unrecognized required +group names remain unsupported and cannot award improvement. + +### Stage objective evidence + +`stageObjectivePolicy` freezes a `stage-objective-v1` policy alongside the +experiment protocol. It requires `stage_replay`, or `signal_replay` with +`waveformStages: true`; plain comparison cannot evaluate a stage objective. +Numeric and stage policies are separate, and a joint replay may retain both. + +The stage policy sets minimum independent nights and paired reference duration, +minimum candidate output coverage, maximum loss of baseline output from shared +support, minimum equal-night agreement improvement, tolerated night regressions, +macro-F1 regression, and required-stage F1 regression/support. Common-support +confusion durations are combined within each whole night before scoring. The +uncertainty gate uses the lower bound of the frozen whole-night agreement +bootstrap; epochs are not treated as independent nights. Missing required stages +on any evaluated night remain insufficient evidence, never fabricated zero error. + +`stageObjectiveAssessments` preserve the exact policy and evidence for candidate +versus both default and current configurations. The coordinator replaces any +scorer-supplied operational evidence after the final admission check, using the +same explicit sampling-policy rules as numeric objectives. Only a separately +designated qualified policy can award improvement. The UI presents thresholds, +missing evidence and regression reasons. These are agreement results against the +named stage reference, not claims of clinical stage accuracy or live activation. + +### Evidence verification before activation + +The internal profile evidence verifier checks retained report bytes and exact +candidate revision, current configuration, shipped default, dataset/side, frozen +holdout cohorts, protocol, evaluator build, and waveform dependencies. Reused +holdout data or released artifacts cannot pass. These expectations must be +loaded by the service; they are not client assertions. + +The verifier recomputes objective assessments using supervised operational +measurements and separately designated policies. It requires both baseline +comparisons and every frozen holdout night. A changed stage configuration must +improve stage agreement; a numeric change must improve its metric and pass +downstream stage coverage/regression checks, which may report no material change. +An assessment of one numeric metric cannot validate changes to another metric. +A signal replay can freeze `objectivePolicies`, a list of one to three numeric +policies with unique metric identities. It is mutually exclusive with legacy +`objectivePolicy`; legacy snapshots and hashes remain unchanged. Evaluation, +supervisor finalization, and activation verification require candidate comparisons +against both default and current for every policy. Each changed numeric component +must have its own qualified improvement evidence; a passing heart-rate objective +cannot validate breathing or HRV changes. Stage objectives remain separately +required for downstream effects. Report displays group thresholds and reasons by +metric. These policy lists do not themselves designate a production policy. + +The internal validated activation service loads the retained job, experiment, +dataset, report, profile revision and current runtime baseline. It verifies +the report before acquiring write reservations, then rechecks report/dataset +availability, fencing identity, holdout exposure, runtime baseline and binding +generation before committing. A short biometrics write reservation prevents +concurrent exposure, release or runtime writes until the configuration commit +finishes. Full report parsing and hashing happen before this reservation; the +final artifact checks read only metadata. Only the configuration database is +written: binding, evidence audit and outbox intent commit together. + +The audit keeps the evaluation ID, report/configuration/build hashes, policies +and assessment summary even if biometric artifacts are later released. An exact +idempotent retry returns the original activation event without requiring those +artifacts again. Changing the request under the same key is rejected; manual +and validated requests cannot alias one another. A successful request still +returns `pending_delivery`, not proof of processor adoption. + +Stage replay now captures the entire acknowledged current configuration, including +numeric parameters, instead of substituting defaults for those parameters. Pending +or incompatible personalization blocks validated activation. Qualified production +policy designation remains unfinished. The shared +`activateValidatedProfile` procedure (`POST /biometric-tuning/activate-validated`) +accepts the exact revision, resolved hash, binding generation, evaluation ID, +report hash, processing boundary, requester, and idempotency key. Build hashes and +policy designations are resolved on the server. New requests currently fail closed +because no production policy is designated. Exact retries precede build probing +and artifact lookup. MCP `apply_biometric_profile` exposes the same procedure with +`action: "activate_validated"`; it does not accept `allowUnvalidated`. Explicit +manual activation retains its existing unvalidated semantics. + + +The holdout report UI offers application of its exact candidate and report hash. +`reviewValidatedActivation` (`GET /biometric-tuning/validated-activation-review`) +performs a read-only eligibility check using the same stored evidence and installed +build authority. A review creates no activation event or outbox item, and cannot +reserve evidence: application repeats verification and its transactional guards. +The apply control remains disabled when review fails or is refreshing. Once sent, +an uncertain request retains its exact identity and boundary for retry, including +when detailed report data is subsequently released. Processor adoption is polled +separately. The demo explicitly rejects validated activation of synthetic evidence. + + +Promotion designation also pins the complete experiment protocol hash. Matching +objective thresholds alone is insufficient: changes to clock offset, alignment +tolerance, assumed stage calibration quality, uncertainty settings, replay mode, +or objective lists require qualification of the resulting protocol. This server +value cannot be supplied through activation or review requests. The audit already +retains that exact protocol hash. This identity gate does not establish that missing +historical pump/occupancy context is acceptable; replay-context qualification is +still a separate release requirement. + + +Waveform promotion also requires a server-designated list of accepted replay +limitations. The verifier reads both per-run and per-reading diagnostics across +every held-out cohort and distinct configuration, verifies their retained payload +hash and build/configuration identities, and rejects missing or duplicate outputs. +Missing pump/occupancy/sleeper-mode context, short pre-roll, cold starts and event +clock substitution cannot pass merely because numeric scores improved. Unknown +limitation codes cannot be designated until the contract is reviewed. Activation +audit evidence records both the accepted list and the limitations actually observed. +No production replay-context allowance is installed yet; this gate does not by +itself qualify any approximation. + + +The publisher and activation service share a release-installed qualification +registry, `promotion-qualifications.json`, beside the packaged replay worker. +Its version is `biometric-promotion-qualifications-v1`; each entry pins the full +protocol, manifest hash, evaluator build hash, optional Python build hash, +qualification ID, fixture and pod evidence hashes, and replay-context allowance. +Every objective policy must use the entry's qualification ID. Exactly one matching +entry is required. Missing, malformed, oversized, duplicate or mismatched entries +grant no designation. Reads are capped at 256 KiB; no public endpoint writes this +registry. Evidence hashes identify release-reviewed artifacts; loading a registry +is not itself a qualification experiment. The publisher additionally rejects +context outside the registered allowance before awarding metric-level statuses. +No production registry has been generated or installed yet. + + +The supervisor now derives the overall retained report status with the same +assessment logic used for activation. `improved` requires complete qualified +independent evidence for every changed domain and acceptable downstream stages. +Supported failures produce `regressed`; insufficient support remains +`insufficient_evidence`, incompatible metric evidence is `incompatible`, and +supported outcomes below the improvement threshold are `no_material_change`. +An identical candidate/current configuration is also reported as no material +change. The supervisor replaces worker status claims. The UI displays these as +historical evaluation outcomes; an improved report never substitutes for current +activation review, holdout-exposure checks, or processor acknowledgement. + +## Original vitals provenance and CSV compatibility + +`biometrics.getVitals` retains its array and existing numerical fields. It now +adds nullable `processingRunId` and `provenance` metadata for each original +observation: `interpretation: original_observation`, a status, the recorded +configuration hash, and the recorded HRV metric identifier. The run must belong +to the same side and be a live run. Missing associations are `unavailable`; rows +without a run remain `legacy_unknown`. A missing, malformed, or unavailable HRV +definition stays null. The current shipped manifest is never used to infer an +old row's metric identity. This metadata describes the recorded association; +it is not validation evidence or a current processor-health receipt. + +Only bounded scalar metadata is projected from the processing bundle. The raw +bundle, input identity, and unrelated fields are not returned. A future recorded +HRV metric identifier is preserved rather than relabelled as the current window +IBI variability index. Consumers must keep different/unknown definitions +separate when computing comparable baselines. + +Summary and baseline responses now include exact SQL aggregate provenance counts +across the full selected range, independent of chart pagination: distinct recorded +configurations (manifest plus resolved configuration hash), unknown configuration +rows, HRV definitions and unknown HRV rows. Mixed known definitions, or known +and unknown HRV values together, produce null HRV aggregates. Entirely legacy +ranges retain their descriptive values with `legacy_unverified`; no definition +is inferred. All baseline means and deviations are null when multiple recorded +configurations or recorded/unknown configurations coexist. A summary may still +describe original HR/breathing values across configurations, with explicit counts. + +The chart separates recorded processing runs and HRV definitions before smoothing, +omits bands/statistics across those boundaries, and explains original/unknown +provenance. Its raw log exposes run IDs and HRV definitions. Unknown legacy rows +remain unverified: missing history cannot prove that their processing was uniform. +A run boundary is deliberately conservative even when two restarts used the same +configuration. This is distinct from the manifest/configuration counts in the +whole-range API summary. + +The vitals CSV keeps its original five columns first and appends interpretation, +processing-run ID, provenance status, configuration hash, and HRV metric ID. +Unavailable definitions are written as `unknown`, never inferred as clinical +RMSSD. Missing numerical values remain empty. Metadata cells are CSV-escaped +and spreadsheet-formula prefixes are neutralized. Older consumers can continue +using the original named columns; positional consumers must accept appended +columns. Demo/older responses without metadata export unknown provenance. diff --git a/docs/biometrics-calibration-readiness.md b/docs/biometrics-calibration-readiness.md new file mode 100644 index 000000000..60f309ec4 --- /dev/null +++ b/docs/biometrics-calibration-readiness.md @@ -0,0 +1,72 @@ +# Biometric calibration release evidence + +This is the acceptance evidence map for the calibration implementation, including +prospective monitoring and external algorithm experiments. A passing synthetic +benchmark is resource evidence for its exact build and input; it is not evidence +of improved accuracy or authorization to promote an algorithm. + +## Validation recorded on October 13, 2026 + +- Full CI Vitest run at `cb39444b`: 6,856 passed, 18 skipped; 480 files passed, + one skipped. Skips are not counted as verified behavior. +- Locked piezo environment: 203 piezo/shared numerical tests passed; the separate + common-module suite passed 415 tests with five skipped. +- TypeScript and the production build passed after the MCP workflow addition. +- Browser checks at `216dab46`: five passed, including the real database manual + calibration/receipt/rollback flow and English/Spanish provenance views. +- Deployment archive regression at `b1e05db3`: 26 tests passed. Local Python + environments/caches are excluded while source and locked dependencies remain. +- The shadow benchmark harness at `4ee74c25` passes TypeScript/lint and bundles; + those checks do not establish target-pod execution. + +## Acceptance map + +These links locate implementation contracts and behavioral evidence. Target-pod +acceptance is separate from fixture and browser results. + +| Requirement | Implementation and evidence | +| --- | --- | +| Defaults without user configuration | [Registry tests](../src/biometrics/algorithms/tests/registry.test.ts), [numerical golden fixtures](../modules/piezo-processor/test_default_golden.py) | +| Independent compatible side overrides | [Profile activation tests](../src/biometrics/profiles/tests/activation.test.ts), [runtime routing](../modules/common/test_biometric_routing.py) | +| Sensor calibration distinct from personalization | [Runtime ownership and lifecycle](biometrics-runtime-activation.md), [sensor calibration architecture](hardware/calibration-architecture.md) | +| Preserve existing stage overrides | [Runtime integration](../src/biometrics/profiles/tests/runtime-integration.test.ts), [migration/provenance tests](../src/biometrics/profiles/tests/provenance-migration.test.ts) | +| Reference source, metric, interval, timezone and import provenance | [Reference normalization](../src/biometrics/references/tests/normalize.test.ts), [reference storage](../src/biometrics/references/tests/store.test.ts), [import contracts](biometrics-reference-imports.md) | +| HRV definition across storage, reports, UI, exports and prompts | [API/provenance contracts](biometrics-calibration-api.md), [router tests](../src/server/routers/tests/biometrics.provenance.test.ts), [CSV tests](../src/components/biometrics/tests/vitals-csv.test.ts), [MCP contracts](../src/mcp/tests/server.test.ts) | +| Identical retained inputs, whole-night separation, coverage-aware evaluation | [Frozen datasets](biometrics-datasets.md), [dataset tests](../src/biometrics/backtest/tests/datasets.test.ts), [evaluation assessment](../src/biometrics/profiles/tests/evaluation-assessment.test.ts) | +| Missing raw/context limits claims | [Waveform dataset tests](../src/biometrics/backtest/tests/waveform-dataset.test.ts), [retained replay tests](../modules/piezo-processor/test_retained_replay.py) | +| Shared processing, durable jobs, bounded resources, no canonical/hardware effects | [Backtest lifecycle](biometrics-backtest-jobs.md), [waveform publication integration](../src/biometrics/backtest/tests/waveform-publication.integration.test.ts), [shared live/replay tests](../modules/piezo-processor/test_waveform_replay.py) | +| Shared UI/MCP workflow | [Browser workflow](../tests/browser/workflow.e2e.ts), [failed validation](../tests/browser/calibration.e2e.ts), [MCP session tests](../src/mcp/tests/server.test.ts) | +| Exact revision, concurrency and actual acknowledgment | [Runtime integration](../src/biometrics/profiles/tests/runtime-integration.test.ts), [daemon adapter tests](../modules/piezo-processor/test_runtime_adapter.py) | +| Distinct audited reset and rollback | [Activation tests](../src/biometrics/profiles/tests/activation.test.ts), [runtime protocol](biometrics-runtime-activation.md) | +| Original observations distinct from recomputation | [Historical API semantics](biometrics-calibration-api.md), [provenance browser checks](../tests/browser/provenance.e2e.ts) | +| Coherent updates, resets, cleanup, failures and restarts | [Runtime host tests](../src/biometrics/profiles/tests/runtime-host.test.ts), [backtest lifecycle](biometrics-backtest-jobs.md), [deployment tests](../tests/updateDeploy.test.ts) | +| Isolated generated code and reviewed promotion | [External experiment contracts and real-Docker checks](../scripts/algorithm-experiments/README.md), [shadow admission tests](../src/biometrics/shadow/tests/admission.test.ts), [promotion registry tests](../src/biometrics/profiles/tests/qualification-registry.test.ts) | +| Target-pod operational behavior | [Measurements and limitations](biometrics-replay-benchmark.md); deployment and live measurement status below | + +## Delivery and qualification status + +- The current eight-hour replay and 30-minute synthetic shadow supervisor runs + completed successfully; their receipts are linked in the measurement document. + The separate 180-second live source/isolated sink assessment also completed; + its exit-race regression, build identities and limitations are recorded there. + Synthetic output cannot supply held-out accuracy. +- The measured replay policy is installed and the pod reports admitted. + Reviewed promotion/shadow registries still need their own matching fixture and + held-out evidence; absence must remain visible and deny qualified promotion. +- Build `4c3a6a6e` is deployed, including the live-shadow exit-race fix. Migrations succeeded; the calibration API, live + data path and core/numerical services are healthy, and new vitals carry processing + provenance. [Deployment receipt](performance/biometrics-replay-2026-10-13/deployment-verification.json). + Unpersonalized defaults have no requested activation generation, so this check + does not claim a new profile acknowledgment or change the user's profile. +- [PR #824](https://github.com/sleepypod/core/pull/824) is pushed against `dev`. + CodeRabbit full review was requested, but the provider skipped it because + the PR exceeds its 100-file review limit. No completed CodeRabbit review is claimed. +- Build, lint, typecheck, unit, browser and all Python jobs passed at `8909f305`. + The patch-coverage gate passed at 86.47%, within the repository's existing + eight-point margin below its 88.93% target. No coverage threshold was changed. + Direct coordinator and host tests cover admission, cancellation, retention, + error sanitization and resource ownership. + +Legacy reference availability alone does not establish independent nights, metric +compatibility, reference reliability or sufficient held-out support. No accuracy +qualification is inferred from an inventory of existing references. diff --git a/docs/biometrics-datasets.md b/docs/biometrics-datasets.md new file mode 100644 index 000000000..b52e093ae --- /dev/null +++ b/docs/biometrics-datasets.md @@ -0,0 +1,251 @@ +# Frozen biometric datasets + +`DatasetStore` retains bounded processed-data snapshots in `biometrics.db`. +It is the persistence layer for the calibration backtest workflow; it is not +yet a waveform replay service. Creating a dataset does not modify live vitals, +active settings, or hardware. + +A snapshot contains the selected full recorded sessions, original vitals and +movement rows, exact reference import revisions and normalized measurements, +and available processing-run bundles. Unknown legacy processing provenance +stays null. Measured calibration bytes are not inferred from today's profile. +The manifest explicitly records that limitation and disables signal replay. +Stage replay must state its calibration-quality assumption when the original +quality context is unavailable. + +Each snapshot owns its input bytes. Deleting or correcting a reference import, +or pruning original readings, leaves a frozen dataset unchanged. Dataset reads +verify the hash of both manifest and snapshot. Reusing an idempotency key +returns the original dataset; a different selection under that key fails. +Only one revision from a reference family may appear in a dataset. Both side +and asserted sleeper mapping must match its references. + +The initial processed-data limits are 60 selected sessions, 20 reference +imports, 20,000 retained rows, 8 MiB per snapshot including its manifest, and +64 MiB across these snapshots. These are conservative service bounds, not a +claim about target-pod replay performance. Raw artifacts need separate retention +and free-space safeguards before signal replay can be enabled. + +Whole-night grouping uses the selected timezone's local noon as the night +boundary. Overnight sessions and naps touching the same night stay together; +a session spanning noon connects adjacent night cohorts. Cohorts are sorted +chronologically before assigning tuning, validation, and holdout. Fewer than +three cohorts yields descriptive tuning only. More cohorts establish a split, +not validated improvement or adequate statistical power. ICU/timezone runtime +versions and the actual assignment are retained; later timezone updates cannot +silently change an existing dataset. + +Coverage includes actual timestamp counts, duplicates, and leading/internal/ +trailing gaps under an explicit nominal cadence. This estimates recording +availability, not measured signal support duration. Overlapping selected +sessions retain original row IDs; evaluators must deduplicate shared rows and +score one independent night cohort rather than treating sessions as independent +nights. Retrospective classification remains retrospective. + +## Shared waveform replay driver + +`modules/common/biometrics/replay.py` provides the internal streaming driver for +retained waveform experiments. Callers supply immutable per-logical-side +configurations, integer Unix-millisecond record times, recorded sleeper mode +and occupancy snapshots, pump-context availability, and an experiment output +sink. The driver does not read current bindings, occupancy files, or databases. +It processes both physical channels in left-before-right order using the same +sample repair, pump/presence gates, stateful estimator, and single-sleeper +arbitration as the live adapter. + +Records must be ordered within the retained half-open time window. Each channel +is capped at 5,000 int32 samples per record; processor buffers retain only their +fixed windows. Pre-roll is processed, but outputs are scored only inside the +requested half-open scoring window. The output records preserve computed +logical side, physical source side, configuration hash, event timestamp, +quality, and flags, including when a held candidate is flushed after a mode +change. Context and outputs are immutable; outputs stream to the supplied sink. + +Cold-start state and replacing live arrival time with recorded event time are +explicit limitations. Missing mode, occupancy, or pump context adds separate +limitations; unknown mode uses a disclosed two-side fallback. Less than five +minutes of pre-roll is also flagged. These labels do not claim exact historical +parity. The shared driver does not itself discover or retain RAW files, capture +historical context, or authorize a public waveform job. Those integration and +resource checks are required before dataset capabilities may advertise raw +signal replay. + +`modules/common/biometrics/raw_artifacts.py` supplies retained-byte storage for +that integration. It accepts a selected `.RAW.gz` filename under a server-owned +archive directory, rejects path traversal, symlinks and non-regular files, and +copies into a private temporary file. It detects source changes, verifies gzip +expansion within a byte limit, hashes the exact compressed bytes, fsyncs the +copy, and atomically publishes a content-addressed artifact. Repeated retention +of identical bytes reuses the verified artifact. Deleting the rolling archive +thereafter does not remove the retained input. + +A process lock serializes publication/quota checks and verified reads. Crash +staging files are reclaimed only while holding that lock. Published artifacts +are immutable and private (0600 files, 0700 directory); readers check their +size and SHA-256 before exposing a bounded decompression stream. A changed or +missing artifact fails rather than substituting an archive file with the same +original name. Gzip validation is not CBOR/frame or timestamp validation: the +record inventory must still establish those properties before advertising +waveform coverage. + +The injectable retention limits default to 256 MiB compressed per artifact, +512 MiB expanded, 1 GiB and 1,024 published artifacts total, and a 512 MiB free-space +reserve. One additional compressed staging copy can exist while publication +is serialized, including during an idempotent retry at the published quota. +These are provisional storage limits, not a measured production replay policy. +Dataset references, release/garbage collection, record inventory, and job +admission must be integrated before this storage helper is exposed publicly. + +`raw_records.py` adds a bounded firmware-CBOR decoder and retained-file inventory. +It supports the definite and indefinite maps/arrays/strings used by captured +firmware, with explicit record-byte, string-byte, node-count and nesting limits. +Container lengths are checked before allocation; duplicate/non-text map keys, +semantic tags, malformed UTF-8 and trailing values are rejected. The retained +iterator distinguishes clean end-of-file from a truncated outer RAW record and +counts empty sequence placeholders toward its record limit. + +The inventory reports exact retained artifact identity, event-time bounds, +relevant record counts, usable dual-channel frames, lost-marker counts, +truncated sample tails, timestamp reversals and oversized channels. Its +`ordered_waveform_available` flag means the retained records meet this driver's +ordering and frame bounds; it does not establish continuous recording coverage, +complete pump context, or availability of occupancy/calibration snapshots. +Equal event timestamps are reported and preserved. Other sensor records are +counted without being misrepresented as waveform support. Numerical sensor +values retain CBOR float semantics; nonfinite or invalid event timestamps are +reported as unavailable for ordered replay. + +`raw_sources.py` inspects server-owned cold archives without retaining them. +Discovery is bounded and excludes symlinks; inspection streams checksum, +expanded-size validation and record inventory. Its result explicitly states +`retained: false`. Selection passes the inspected descriptor as `expected` to +`RawArtifactStore.retain`; changed compressed bytes or sizes fail before +publication. A source name or successful inspection alone is never evidence +that a dataset has frozen input bytes. Persisted inventory caching and dataset +references remain responsible for tracking inspected versus retained state. + +Migration `0028_retained_raw_artifacts` adds separate tables for inspected +sources, content-addressed retained artifacts, and ordered dataset references. +`RawSourceCatalog` validates the Python inspection contract, including matching +hashes, event bounds, record counts and ordered-waveform eligibility. Repeating +an inspection refreshes its timestamp; different evidence for identical bytes +and inventory version fails. Replacing a filename preserves the old identity +and marks it unavailable. A late failure for an old identity cannot mark newer +bytes unavailable. + +Catalog queries return at most 50 entries per page for potential event-time +overlap. Unexpired inspection evidence is labelled `requires_reverification`, +not available or retained: selected copies still require the expected-descriptor +check. Source scans do not create datasets or retained-artifact rows. Shared +artifact references have unique dataset positions and cannot dangle; release +and file garbage collection must coordinate those references before deletion. +The catalog and schema do not yet enable public waveform jobs. + +Dataset release now atomically tombstones its RAW references with the dataset +snapshot. Queued, running, cancellation-pending and interrupted jobs prevent +release. An artifact becomes eligible for collection only when its last live +reference is released; releasing another dataset never changes that timestamp. +This eligibility timestamp does not itself mean the file has been deleted. + +`RawArtifactStore.collect_released` holds the same process lock as verified +readers and publication, then takes a SQLite immediate transaction while it +checks eligibility and unlinks files. The default one-hour grace period is +injectable. Directory traversal is bounded before any deletion; the collector +preserves unknown files/metadata, rejects symlinks and mismatched identities, +and keeps database tombstones. It scans present files, so previously collected +rows cannot starve later eligible files. Unregistered crash orphans are not yet +collected automatically. + +The internal `retain(..., expected=descriptor, publish=callback)` hook allows a +trusted publisher to commit references before releasing the file lock, including +when reusing existing bytes. Publishers must lock files before beginning their +SQLite transaction, commit before returning, and never reacquire the file lock +inside the callback. Publication failure leaves complete bytes for recovery. +The public waveform dataset creation path and scheduled collector still need +to be wired through this protocol before waveform replay is enabled. + +`retained_replay.py` connects immutable artifact descriptors to the shared +waveform engine. Its versioned `retained-waveform-input-v1` contract contains a +retained/pre-roll/scoring window, ordered artifact descriptors with inspected +event bounds, and frozen context intervals. It rejects duplicate or overlapping +artifact windows instead of guessing which boundary frames to discard. Missing +context intervals resolve to unknown; it never reads the current sleeper mode, +occupancy or calibration to fill historical gaps. + +The adapter verifies retained bytes and consumes every selected file to EOF, +including events outside the scoring window. It checks actual event bounds and +ordering against the frozen input, then passes selected frames to the same +numerical driver as direct replay. Initial bounds are 64 artifacts, 1 GiB +compressed input, one million RAW records across all artifacts (including +empty placeholders), 20,000 context intervals and 20,000 output readings. +These are explicit implementation bounds, not a qualified pod resource policy. +The caller must also enforce a process deadline; cooperative cancellation does +not interrupt a file checksum operation already in progress. + +Output sink rows are provisional until the adapter returns successfully. A +late archive integrity failure must fail the job and discard its staged result. +The returned run metadata includes consumed/decoded counts, exact artifact and +configuration hashes, processing counters and limitations. The public durable +worker still needs to capture this input contract, retain its references and +publish the completed result through its fenced job lifecycle. + +The fixed Python entry point is `python -B -m common.biometrics.replay_worker +--retained-directory `. Its input is one bounded JSON +`retained-waveform-work-v1` envelope containing `input` and both logical-side +`configs`. It opens existing retained storage in read-only mode: missing storage +or a missing lock fails, and it neither creates directories nor changes their +permissions. RAW retention/collection methods reject calls on that reader. + +The worker emits bounded JSON-lines `reading` events and exactly one final +`result` on success. Readings explicitly identify HRV as +`window_ibi_variability_v1`; nonfinite output fails serialization. Input rejects +duplicate JSON fields and nonfinite constants. Errors emit one fixed diagnostic +without exporting inputs, paths or stack traces. CPU limits are 120 seconds +(121 hard), file descriptors are capped at 64, core dumps are disabled, numerical +thread counts are one, and process priority is reduced. These provisional caps +are not a qualified memory or pod admission policy and do not sandbox uploaded +code. + +`runWaveformChild` provides the coordinator-side transport with a clean child +environment, 8 MiB input, 16 MiB output, 64 KiB event and 16 KiB diagnostic +limits, a 20,000-reading bound, and timeout/cancellation with TERM/KILL escalation. +It verifies event order, schema, final row counts and per-side configuration +hashes, returning readings only after a final result and zero exit. A missing or +failed worker never returns its provisional rows. Executable, module and retained +storage locations are internal server configuration and must not be accepted +from public requests. The transport is tested against the actual Python worker; +the coordinator still needs to invoke the frozen dataset publisher and durable +waveform jobs before public waveform capability can be enabled. + +Waveform dataset publication now has an internal complete path. The TypeScript +preparer verifies a processed parent dataset, detaches its historical context, +and creates a new `waveform-dataset-v1` snapshot with one frozen input per +whole-night cohort. It preserves original observations and references, requires +matching scoring windows, normalizes cohort order, and hashes the complete new +snapshot. Shared artifacts must have identical descriptors and inspected event +bounds across cohorts. The parent dataset remains independently readable. + +The fixed `common.biometrics.publication_worker` accepts a server-prepared row +and inspected source selections, with database/archive/retained paths supplied +only by server configuration. It opens an existing database, enables foreign +keys, and uses `retain_batch` to hold the file lifecycle lock through all copies +and reference publication. Copies and checksum/decompression verification finish +before the SQLite immediate transaction. That transaction inserts the dataset, +artifact metadata, and all ordered references together, enforcing the aggregate +snapshot quota. A failed transaction rolls back every database write. Complete +unreferenced files can remain after failure and still require orphan recovery. + +Identical retries return the original dataset, including a released tombstone; +they do not resurrect it or require the rolling archives. A new dataset may +reuse exact retained bytes after rechecking compressed identity and expanded +size, even when the original archive directory has expired. Reusing an artifact +clears its collection eligibility only in the transaction adding its new live +references. Existing reference history and artifact creation time remain intact. + +The shared reader understands both processed and waveform snapshot versions and +verifies their full content hashes. A committed integration test covers actual +archive inspection → TypeScript preparation → Python publication → shared +reader verification → Python numerical replay → release → collection, while +checking that canonical observations and the parent dataset remain unchanged. +Public `signalReplay` remains false until coordinator/API source selection and +fenced waveform-job/report integration are wired to this publisher. diff --git a/docs/biometrics-reference-imports.md b/docs/biometrics-reference-imports.md new file mode 100644 index 000000000..5413a8c94 --- /dev/null +++ b/docs/biometrics-reference-imports.md @@ -0,0 +1,175 @@ +# Biometric reference import contract + +Reference imports are comparison data. They never write the pod's `vitals`, +change a profile, or control hardware. Agreement with a named reference source +is not clinical accuracy. APIs use the existing LAN trust model. + +## Preview and commit + +Call `biometricTuning.previewImport` with an explicit physical `side`, a +`subjectId` identifying the sleeper, a named `source`, and bounded `rows`. +Neither sleeper nor side is inferred from a filename. The preview returns +accepted/rejected/duplicate/conflict counts, bounded rejection examples, +metric identities, time interpretation, coverage endpoints and a content hash. +It writes nothing. + +```json +{ + "side": "left", + "subjectId": "sleeper-A", + "source": "wearable-export", + "timePolicy": { "timezone": "America/Los_Angeles", "disambiguation": "reject" }, + "rows": [ + { + "recordKey": "sample-123", + "metric": "heart_rate", + "value": 62, + "unit": "bpm", + "timestamp": "2026-10-10T23:00:00-07:00" + } + ] +} +``` + +Commit with `biometricTuning.commitImport`, passing the same payload as `import` +and the returned hash as `previewHash`. Changes to data, mapping, units, source +or time interpretation require another preview. When some rows were rejected, +commit additionally requires `acceptRejectedRows: true`. An import with no +accepted rows cannot be committed. Repeating the same content is idempotent. + +For a corrected full export, include `supersedesImportId` naming the latest +revision in the family. Corrections preserve source, sleeper and side; they +create a new immutable revision rather than editing old measurements. A stale +parent is rejected. Selecting an import ID always selects that exact revision. + +`listImports` pages at 50 imports using both `beforeCreatedAtMs` and `beforeId`. +`getImportMeasurements` pages at up to 500 rows using `afterId`. Follow +`nextCursor` to read the complete import. Preview summaries never contain the +full export. `deleteImport` explicitly removes its measurement rows and any +linked legacy chart rows, leaving a compact unavailable-import tombstone; +retrying the deletion is a no-op. A deleted import ID cannot supply readings. + +REST equivalents are under `/api/biometric-tuning/references`, with POST +`/preview`, POST `/commit`, GET `/`, GET `/{id}`, GET `/{id}/measurements` and +DELETE `/{id}`. All use the same service as tRPC. + +## Records + +Required fields: `recordKey`, `metric`, `value`, `unit`, and a time representation. +`recordKey` is the source's stable record identity. Repeated identical keys in +one batch are deduplicated; different measurements with the same key are all +rejected. Optional row `subjectId` must agree with the batch mapping. + +Use exactly one time representation: + +- Point: `timestampMs` (integer Unix milliseconds) or `timestamp` (ISO text). +- Interval: `startMs` plus `endMs`, or ISO `start` plus `end`. + +Intervals are half-open `[start, end)` and may span at most 24 hours. Unix +milliseconds must fall between 2001 and 2100; Unix seconds in millisecond fields +are rejected. ISO text includes seconds, optional milliseconds, and preferably +an explicit `Z` or `±HH:MM` offset. Invalid calendar dates are rejected. + +Naive local ISO times require a `timePolicy` with an IANA `timezone` or explicit +`utcOffsetMinutes`. Repeated DST times are rejected unless `disambiguation` is +explicitly `earlier` or `later`, or a valid offset is supplied. Nonexistent local +times are rejected rather than shifted. An explicit offset must agree with a +supplied timezone. An absent source timezone stays absent in provenance; it is +never inferred from the pod's current timezone. + +| Metric identifier | Units accepted | Meaning | +| --- | --- | --- | +| `heart_rate` | `bpm`, `beats/min`, `Hz` | Reference heart rate; Hz converts to bpm | +| `breathing_rate` | `breaths/min`, `br/min`, `Hz` | Reference breathing rate | +| `window_ibi_variability_v1` | `ms`, `s` | Pod window-level IBI variability index, not clinical RMSSD | +| `hrv_rmssd` | `ms`, `s` | Reference RMSSD, kept distinct from the pod index | +| `hrv_sdnn` | `ms`, `s` | Reference SDNN, kept distinct from RMSSD and the pod index | +| `hrv_unknown` | `ms`, `s` | Unspecified HRV definition; never directly scored | +| `sleep_stages` | `stage` | Interval label: `wake`, `light`, `deep`, `rem`, `asleep`, `in_bed`, `unknown` | + +Numeric values must be finite, nonnegative and within JavaScript's safe numeric +range after conversion. These are data representation constraints, not medical +normal ranges. Original submitted values and units are retained. + +Optional fields are `aggregation` (`instant`, `mean`, `median`, `window_index`, +`stage`), `windowMs`, `quality` (0–1), and `annotation` (up to 1000 characters). +The pod index requires `aggregation: "window_index"` and `windowMs: 300000` for +direct comparison. Matching HRV definitions with different or unknown windows +are not directly comparable. RMSSD/SDNN must never be scored as the pod index. + +Unknown, generic asleep and in-bed stages remain separate labels and are +unscored. Conflicting overlapping stage labels are preserved, but both entire +intervals are marked unscorable under the persisted `exclude_conflicts` policy. +Back-to-back intervals do not overlap. An annotation is untrusted data, never +an instruction, executable code or a command. + +## Text adapters and limits + +The shared `parseReferenceText` adapter accepts a JSON array of records or CSV +with the field names above as the header. CSV supports quoted commas, escaped +quotes and multiline annotations. Duplicate/unknown headers, malformed quotes +and mismatched column counts are rejected. CSV timestamps must retain their +explicit field units; there is no seconds/milliseconds guessing. + +Each upload is limited to 2 MiB, 2000 records, 12 JSON nesting levels and a +bounded node count. Rejection examples stop at 25, while aggregate counts are +complete. No server path, URL, shell command, archive or automatic cloud upload +is accepted. Hosts may parse a local export and send bounded normalized batches. +Do not log complete health exports. + +## Alignment semantics + +`reference-alignment-v1` uses chronological nearest-unused point matching with +a fixed tolerance (default 30 seconds). Each pod observation and reference is +used in at most one pair. Interval comparisons aggregate actual observations +inside the interval using the declared mean or median. Sparse references are +never forward-filled. Equivalent duplicate references do not count twice. + +Clock offset is explicit and fixed for the comparison; a training-derived +offset must remain frozen for validation/holdout. Reports include MAE, median +and 95th-percentile absolute error, signed bias, matched/unmatched counts, +reference and output coverage, exclusions and limitations. Point samples +contribute counts, not inferred duration. Interval duration counts only known +observation support and does not fill gaps. Incompatible HRV has no error score. +This alignment primitive does not by itself establish validated improvement. + +## Legacy Apple Watch clients + +The existing `biometrics.importReferenceStages` and `importReferenceVitals` +contracts still accept Unix seconds and return `{ written }`. They retain +first-write-wins heart-rate behavior and whole-request stage-overlap rejection. +Newly accepted rows and normalized provenance commit atomically; if provenance +cannot be saved, the legacy insert rolls back too. + +Old clients provide no subject identity or source timezone. Such imports are +visibly mapped to `legacy_unknown:left` or `legacy_unknown:right`, with that +limitation in the original record annotation. Older unlinked rows remain +legacy/unknown instead of receiving invented provenance. New clients should +use explicit sleeper mapping and the preview/commit workflow. + +### Stored-reading comparison + +`biometricTuning.compareReadings` (`POST /api/biometric-tuning/compare`) accepts an +exact `importId`, `side`, `subjectId`, metric, and a half-open `startMs`/`endMs` +interval of at most 24 hours. Optional `pointToleranceMs` (default 30 seconds) +and `clockOffsetMs` (default zero) are fixed, reported settings; this endpoint +does not fit an offset. It reads all stored observations in that interval, +without chart downsampling, and returns bounded summary scores, coverage, +source revision, and processing-run counts. A null processing-run ID means +`legacy_unknown`. `requestedMetric` identifies the export's metric; `metric` +identifies the pod metric actually available. Clinical HRV cannot be scored +against the pod's window IBI variability index. + +This is a read-only comparison of original readings. Its input hash identifies +the read snapshot but does not retain bytes or constitute a saved evaluation. +Only reference intervals fully inside the requested range are used, preserving +the original aggregation window. Stored readings do not record their support +duration, so comparisons report point counts and unknown duration (zero), +without filling gaps. A changed reference revision must be selected explicitly. + +The shared stage scorer uses intersecting interval duration, reports a confusion +matrix (rows reference, columns predicted), per-stage precision/recall/F1, +macro F1, coverage, and stage-duration/transition differences on common support. +Absent classes have null scores. Transitions across unobserved gaps are not +inferred. Conflicting intervals from separate sources remain unscored. This +scorer is a primitive for replay reports; it is not a new live stage classifier. diff --git a/docs/biometrics-replay-benchmark.md b/docs/biometrics-replay-benchmark.md new file mode 100644 index 000000000..d6684c9c8 --- /dev/null +++ b/docs/biometrics-replay-benchmark.md @@ -0,0 +1,666 @@ +# Initial synthetic replay measurement + +Measured on the requested pod (`192.168.1.88`, SSH port 8822) on October 12, +2026. This is an initial worker-cost measurement, not the final admission-policy +or end-to-end live-system qualification. + +| Property | Observation | +| --- | --- | +| Runtime | ARM64, four CPUs, Node 24.16.0 | +| Physical memory | 2,014,796 KiB | +| Available memory before / after | 1,440,260 / 1,425,340 KiB | +| `/persistent` available before | 3,741,108 KiB | +| One-minute load before / after | 0.54 / 0.36 | +| Input | 27 separate 12-hour nights, 720 minute readings each; 19,440 readings total | +| Input bytes | 2,374,215 | +| Configurations | Identical default/current; one unique numerical evaluation after deduplication | +| Child settings | nice 19, 128 MiB Node heap limit, 30-second external timeout | +| Evaluator elapsed time | 2,519.355 ms | +| Child peak RSS | 132,436 KiB (about 129 MiB) | +| Complete report size | 2,450,542 bytes | +| Core / piezo service status | Both active before and after | + +The shipped worker SHA-256 was +`16bab5135fcf0ba4d4a1624660f8565ba0b073cf39e7d13760df17347cd24d6a`. +Input was generated from the repository's processed-backtest fixture, expanded +to daily-separated nights with synthetic `60 + sin(minute / 10) * 8` BPM, +HRV index 30 and breathing rate 14. No personal recordings were transferred. +The worker and synthetic input/output existed only in a unique temporary pod +directory, which was removed after measurement. No live database, service +configuration or hardware controls were changed. + +The report exceeded the old 2 MiB inline limit. That concrete failure motivated +the large-report artifact path rather than reducing dataset size or dropping +traces. The worker was invoked directly for measurement; the old repository +would have rejected publication of this report. + +Still required before choosing final admission limits: three distinct candidate +configurations, repeated and worst-case inputs, wall-clock startup/serialization, +artifact I/O, CPU utilization, SQLite contention, ingest age, temperatures and +control/API responsiveness while replay runs. Service `active` status alone does +not prove those properties. This measurement does not cover waveform replay or +untrusted algorithm execution, and does not justify a universal speed claim. + +## Three-configuration follow-up + +On October 12, the same pod and worker hash completed three consecutive runs +with distinct default/current/candidate configuration hashes. Input remained +27 synthetic twelve-hour nights (19,440 readings). The deep-stage HR-ratio +thresholds were 0.92, 0.90, and 0.94 respectively. Other input and protocol +settings were unchanged, including assumed calibration quality zero. These +are distinct evaluations, but this fixture does not exercise every stage branch +or establish a worst-case runtime. + +| Run | Wall time, including child startup/output | Evaluator time | Peak worker RSS | JSONL output bytes | +| --- | --- | --- | --- | --- | +| 1 | 4,295.804 ms | 3,397.305 ms | 156,036 KiB | 3,675,631 | +| 2 | 4,163.752 ms | 3,252.410 ms | 141,956 KiB | 3,675,638 | +| 3 | 4,247.537 ms | 3,330.915 ms | 145,188 KiB | 3,675,631 | + +A Python harness sampled local read-only performance, sensor-source, and device +status HTTP endpoints approximately every half second (requests are sequential, +so request time extends the interval). It recorded only latency, frame age, +resource counters and thermal-zone-zero temperature. It used four seconds of +baseline and three seconds afterward. There were seven baseline samples, +twenty-three replay samples and four afterward; all requests succeeded. + +| Measurement | Baseline | During replay | After | +| --- | --- | --- | --- | +| Minimum available memory | 1,358,520 KiB | 1,215,752 KiB | 1,328,848 KiB | +| Median aggregate CPU busy | 23.78% | 56.65% | 22.22% | +| Maximum aggregate CPU busy | 28.76% | 76.68% | 29.92% | +| Maximum zone-zero temperature | 33.503 °C | 37.356 °C | 30.934 °C | +| Maximum performance GET latency | 34.92 ms | 328.04 ms | 26.11 ms | +| Maximum sensor-source GET latency | 103.29 ms | 45.32 ms | 27.32 ms | +| Maximum device-status GET latency | 51.58 ms | 108.24 ms | 74.00 ms | +| Maximum reported live-frame age | 457 ms | 514 ms | 418 ms | + +CPU busy is the change in `/proc/stat` non-idle counters across adjacent samples +in the same phase, aggregated across all four CPUs. It includes the running +services and measurement harness. The server's lifetime event-loop p95 remained +21.14 ms; that lifetime histogram cannot establish the replay interval's p95. +The sensor endpoint reports the newest frame of any sensor type, so this is not +proof of piezo-specific freshness. Temperature zones one through three were +unreadable; only zone zero was sampled. Status GET latency is not proof of +hardware-write or closed-loop control responsiveness. + +The harness read the worker's complete output file and extracted operational +counters; it did not run the production Node result parser, artifact publisher +or database transactions. Thus parent-process event-loop cost, persistent +artifact I/O, SQLite contention, larger/reference-heavy fixtures, waveform replay +and sustained-load qualification remain outstanding before final admission limits +are selected. Node ran at nice 19 with a 128 MiB heap and a 45-second external +timeout. Both services remained active afterward. Temporary synthetic input, +worker, harness and output were removed from the pod; no live configuration, +observations or hardware settings were changed. + +## Production runner and reference-heavy qualification + +A follow-up on October 12 used the production `DatasetStore`, `BacktestJobStore`, +child driver, result publication, and integrity-checked result reader against a +new synthetic SQLite database on the pod's persistent storage. The application +service and its databases were left untouched. The reusable harness is +`scripts/benchmarks/biometrics-runner.ts`; bundle it with esbuild for Node, leaving +`better-sqlite3` external, and pass a **new** directory, copied migration +folder, packaged worker path, and `mixed` or `dense` fixture name. Directory +creation is exclusive so an existing database cannot be selected accidentally. + +The tested worker SHA-256 was +`45547553cd456cba5051fe6e60e8562b5510ae32bbf8b92cfeb62c59cdeb59d1`. +The coordinator used Node's 384 MiB heap limit; its evaluator child retained +nice 19 and the 128 MiB heap limit. These are measurement settings, not yet an +approved deployment policy. + +The mixed fixture retained 20 twelve-hour nights, 14,400 vital readings, +3,600 movement rows, and 1,600 heart-rate/stage reference records. It includes +periodic missing HR readings, motion, and varying HR/HRV/breathing. Chronological +splitting evaluated 12 tuning nights. The three configurations have distinct +deep-stage HR-ratio thresholds; assumed calibration quality is explicitly one. +This synthetic fixture measures execution and does not validate biometrics. + +| Measurement | Run 1 | Run 2 | Run 3 | +| --- | --- | --- | --- | +| Coordinator execution through publication | 12,007.73 ms | 11,543.85 ms | 11,673.61 ms | +| Evaluator runtime | 5,202.12 ms | 4,962.23 ms | 5,048.52 ms | +| Evaluator peak RSS | 168,392 KiB | 167,180 KiB | 174,336 KiB | +| Integrity-checked result read | 2,286.37 ms | 2,256.85 ms | 2,267.35 ms | +| Coordinator event-loop maximum | 2,386.56 ms | 2,300.58 ms | 2,365.59 ms | +| Coordinator event-loop p95 | 22.64 ms | 22.95 ms | 20.92 ms | +| Coordinator process peak RSS (cumulative) | 257,564 KiB | 261,296 KiB | 262,292 KiB | + +All three attempts succeeded, produced 1,813,618-byte reports, and left the +synthetic canonical vitals unchanged. Dataset creation took 4,322.13 ms. These +reports remain below the 2 MiB inline threshold, so this workload does not +qualify the artifact path. Minimum available system memory during the replay +phase was 1,140,300 KiB and maximum zone-zero temperature was 39.457 °C. + +**Admission is not qualified by these results.** The coordinator's multi-second +synchronous work would block the application if wired into its main event loop. +Dataset construction and result reads have the same issue. Move these operations +into a bounded coordinator worker before enabling production execution, then +repeat the benchmark. A favorable p95 conceals the long individual stalls. + +The harness's own HTTP requests and piezo receipt callbacks run on that same +coordinator event loop. Their delays therefore cannot be attributed solely to +the application server or sensor ingestion. The separate standard-library probe +`scripts/benchmarks/biometrics-probe.py` measures loopback API latency from another +process. Neither probe retains API bodies or raw sensor samples. The Node +probe retains only sensor timing and resource measurements. This remains +synthetic, non-production-database testing: it does not establish live SQLite +contention, hardware-write responsiveness, waveform replay, or untrusted-code +isolation. + +### Large artifacts and independent API probe + +The dense fixture used the same worker with 19,000 synthetic vitals, no movement +rows, and 800 reference records across 20 nights. Twelve tuning nights produced +reports of 2,301,040, 2,301,041, and 2,301,034 bytes. All three stored result +descriptors were verified as `biometric-report-artifact-v1`, all result reads +passed integrity checks, and canonical synthetic vitals were unchanged. + +| Measurement | Run 1 | Run 2 | Run 3 | +| --- | --- | --- | --- | +| Coordinator execution through publication | 13,236.50 ms | 12,901.88 ms | 13,064.22 ms | +| Evaluator runtime | 5,501.95 ms | 5,455.62 ms | 5,499.04 ms | +| Integrity-checked artifact result read | 2,803.76 ms | 2,933.55 ms | 2,973.02 ms | +| Coordinator event-loop maximum | 2,961.18 ms | 2,780.82 ms | 2,885.68 ms | +| Evaluator peak RSS | 166,692 KiB | 161,240 KiB | 162,472 KiB | + +Dataset creation took 4,477.81 ms. Coordinator peak RSS reached 262,456 KiB; +minimum available system memory during replay was 1,105,904 KiB; zone-zero +maximum temperature was 40.274 °C. Piezo source-timestamp age seen by the +coordinator peaked at 4,728 ms during replay versus 2,584 ms in its baseline. +Because this callback shares the blocked coordinator loop, that measurement +still combines actual frame age with callback delay. + +The independent Python probe made 12 baseline samples, 251 samples spanning +fixture setup and all three attempts, and 9 afterward. Every GET returned 200. + +| Maximum HTTP latency | Baseline | Entire benchmark | After | +| --- | --- | --- | --- | +| Performance | 36.88 ms | 57.52 ms | 22.48 ms | +| Sensor source | 93.53 ms | 72.37 ms | 22.72 ms | +| Device status | 66.74 ms | 80.61 ms | 52.32 ms | + +The production service remained active after the benchmark. These observations +support keeping heavy coordination separate from the application event loop; +they do not approve running it on that loop. The harness's configuration labels +are synthetic comparison roles (thresholds 0.90, 0.92, 0.94), not a promotion +comparison against an asserted shipped-default profile. Repeat qualification +after coordinator isolation and again for the final deployed worker. The +measurements do not yet select universal admission thresholds or qualify live +SQLite contention and closed-loop control responsiveness. + +## Isolated coordinator follow-up + +The coordinator implementation at `0e5c87c0` was measured on the same pod on +October 12 using the dense fixture above. The optional final `COORDINATOR` +argument to `scripts/benchmarks/biometrics-runner.ts` routes dataset creation, +experiment creation, execution, and report reads through the shipped coordinator. +The caller uses the same 384 MiB heap cap as the preceding benchmark; each +coordinator has a 384 MiB cap and nice 19, and its evaluator retains 128 MiB/nice +19. Each operation gets a fresh process. The worker hash remains +`45547553cd456cba5051fe6e60e8562b5510ae32bbf8b92cfeb62c59cdeb59d1`. + +An initial isolated run succeeded three times. The repeat below adds a timer +turn after report serialization, so its event-loop maximum includes final IPC +deserialization and JSON serialization as well as execution/publication. This +is a calling-process measurement, not the production application's event-loop +histogram. Both runs used exclusively created synthetic databases and retained +no personal recordings or API response bodies. + +| Measurement | Run 1 | Run 2 | Run 3 | +| --- | --- | --- | --- | +| Execution through publication | 14,955.19 ms | 15,390.62 ms | 15,141.36 ms | +| Integrity-checked full report read | 4,178.71 ms | 4,418.67 ms | 4,237.17 ms | +| Calling-process event-loop maximum | 144.83 ms | 187.83 ms | 174.59 ms | +| Calling-process event-loop p95 | 21.22 ms | 21.04 ms | 21.12 ms | +| Calling-process cumulative peak RSS | 153,428 KiB | 156,940 KiB | 156,940 KiB | +| Evaluator peak RSS | 164,460 KiB | 171,136 KiB | 162,032 KiB | + +All three attempts succeeded; full reports remained approximately 2.30 MB and +canonical synthetic vitals were unchanged. Dataset construction took 5,693.98 ms +with a calling-process event-loop maximum of 56.30 ms. During replay, minimum +available system memory was 1,046,688 KiB, maximum zone-zero temperature was +41.909 °C, and maximum piezo source timestamp age was 2,756 ms. This does not +measure the coordinator's peak RSS; the memory values above must not be summed +as a complete process-tree maximum. + +The independent probe recorded 12 baseline samples, 291 spanning the complete +benchmark, and nine afterward. Every request returned HTTP 200. + +| Maximum HTTP latency | Baseline | Entire benchmark | After | +| --- | --- | --- | --- | +| Performance | 30.48 ms | 160.12 ms | 38.15 ms | +| Sensor source | 27.47 ms | 109.92 ms | 24.23 ms | +| Device status | 65.74 ms | 159.46 ms | 76.02 ms | + +Isolation removed the multi-second calling-loop stalls observed previously, +but full-report IPC/serialization still transfers large trace arrays. In one +retained synthetic report, `nights` occupied about 2.21 MB; the first evaluated +window's `blocks` alone occupied 56,786 bytes. A bounded summary for interactive +review, with exact traces fetched through the existing paginated endpoint, is +an appropriate next optimization. Keep the complete immutable report and its +hash; a transport projection must not silently redefine the retained artifact. + +These observations still do not qualify live-database write contention, +closed-loop hardware responsiveness, waveform processing, or arbitrary-code +isolation. No production resource policy or automatic background execution was +enabled by this benchmark. Temporary benchmark databases and binaries were +removed after measurements were copied locally; the production service was +left running without a deployment or restart. + +## Retained waveform pipeline on the pod + +The October 12 follow-up used the coordinator and Node evaluator built from +`fe25e71a`, plus the Python process-limit compatibility fix accompanying this +measurement. The pod's Python 3.10.4 installation omits the optional `resource` +extension. Workers now use `/usr/bin/prlimit` on Linux when that extension is +missing, failing startup if the utility fails. A separate child on the pod +confirmed CPU soft/hard limits of 120/121 seconds, core limits of zero, and +open-file limits of 64/64 in `/proc/self/limits`. + +The new `scripts/benchmarks/biometrics-waveform-fixture.py` creates deterministic +500 Hz dual-channel sine-plus-noise CBOR archives in an exclusively created +directory. The runner `biometrics-waveform-runner.ts` creates a separate database, +retains those archives through the production publisher, enqueues three unique +configuration hashes, executes the leased waveform pipeline, and reads its +summary. The first five minutes are retained pre-roll. Configuration labels +are synthetic roles; this fixture does not measure biological accuracy. + +The 30-minute fixture contained 5,976,451 compressed bytes. Dataset creation +took 1.309 seconds and retention took 15.809 seconds. Execution reached its +180-second coordinator limit before publishing a result. Canonical synthetic +observations were unchanged. **This workload is not qualified for automatic +execution.** Full-night execution needs a different scheduling/budget design; +simply enabling the existing host would not make these jobs complete. + +The independent probe sampled the benchmark process and its descendants through +`/proc`, in addition to the existing loopback HTTP measurements. RSS is a sampled +sum, not an exact peak or unique physical-memory accounting. The failed +30-minute run produced 12 baseline, 448 benchmark, and nine post-run samples. +All sampled HTTP responses returned 200. + +| Measurement | During the complete 30-minute-fixture benchmark | +| --- | --- | +| Maximum sampled process-tree RSS | 313,332 KiB | +| Minimum available system memory | 1,181,144 KiB | +| Maximum zone-zero temperature | 40.041 °C | +| Maximum performance GET latency | 49.01 ms | +| Maximum sensor-source GET latency | 79.46 ms | +| Maximum device-status GET latency | 115.39 ms | +| Maximum age of piezo WebSocket receipt | 1,446 ms | + +The receipt age is measured at the benchmark client; it is not the age of the +original sensor timestamp. The calling process's largest sampled scheduling +delay was 179.88 ms, so favorable HTTP timings alone do not establish its event +loop budget. Measurements use synthetic databases and do not qualify live +SQLite write contention or hardware-command latency. No production admission +policy is selected by these results. + +A 10-minute follow-up (five minutes pre-roll and five minutes scored) completed +the entire pipeline with the same three configurations. Its 1,992,541 compressed +bytes took 15.599 seconds to retain. Leased replay and publication took 83.402 +seconds, and integrity-checked summary retrieval took 1.289 seconds. The summary +was 28,798 bytes; synthetic canonical observations again remained unchanged. +The replay caller's event-loop p95/max were 22.00/63.90 ms, including a timer +turn after the coordinator reply. This is a successful short fixture, not a +qualified whole-night workload. + +The independent probe recorded 12 baseline, 247 benchmark, and nine post-run +samples, all with HTTP 200 responses. During the benchmark, process-tree RSS +reached 314,852 KiB, available memory stayed at or above 1,184,736 KiB, and +zone-zero temperature reached 39.924 °C. Maximum performance/sensor-source/device +GET times were 155.93/96.78/75.31 ms. Maximum piezo receipt age was 1,626 ms. + +The successful run pinned compound build +`b91e8552bc6d973922bd38808d04490683e3e05d8c67c9af5ad37d40a64bfe45` +and Python build +`fad13cfff0b361fbec950e0b6ddd892fe3d97f42f3ef9c5031af2e700c579aca`. +Its complete report hash was +`8446b1a8fd4c0f1530c25ed186ac5231c8020c60e4dffc42b1ec248aa2b46291`. +Python used the pod's installed NumPy 2.2.6 and SciPy 1.15.3. The staged +benchmark used fixed production worker bundles, not the temporary diagnostic +coordinator used to investigate the missing Python extension. + +After both runs, no benchmark descendants remained. Synthetic telemetry and +logs were copied locally, and the staging directory was removed. Both installed +services remained active on the original `homekit-0b53800` release. No deployment, +restart, live-database modification, or hardware command was part of this test. + +### Where waveform execution time goes + +A subsequent synthetic 30-minute profile on the same pod separated startup +from processing. The Yocto Python installation also lacks `_lsprof`, so the +diagnostic used `time.monotonic()` and `time.process_time()` around the shipped +functions. These timings include small wrapper overhead and are not a complete +function profiler. It retained the same noisy dual-channel fixture in a new +directory and discarded derived readings after counting them. + +| Operation | Wall time | CPU time | +| --- | --- | --- | +| Import replay and numerical dependencies | 12.726 s | Not measured | +| Retain the synthetic archive | 0.369 s | Not measured | +| Complete one-configuration retained replay | 45.184 s | 44.751 s | +| Inner CBOR decoding (included above) | 0.596 s | 0.583 s | +| Per-side ingestion (included above) | 41.623 s | 41.238 s | +| Numerical estimates (included in ingestion) | 30.365 s | 30.082 s | + +The run consumed 1,860 records including 1,800 piezo frames and produced 50 +scored readings plus ten pre-roll readings. These overlapping timings must not +be summed. Both startup and numerical processing matter; archive decoding was +not the dominant cost. A full-night run still needs direct measurement rather +than extrapolating this short profile into a production runtime guarantee. + +### Interrupted eight-hour qualification (2026-10-13 UTC) + +The three-configuration, eight-hour synthetic run on `192.168.1.88:8822` +was cancelled during its third configuration after systemd explicitly stopped +`sleepypod` at 01:09:44 UTC. Core and piezo services restarted at 01:10:17 UTC; +`NRestarts=0` and the journal showed the stop request and SIGTERM. This task did +not issue that stop. The installed release remained `homekit-0b53800`. +The admission monitor withdrew permission when HTTP health disappeared. This +run is **not a completed qualification**, regardless of the operation timer's +`succeeded` field (which only records that the execution RPC returned). + +Before cancellation, the benchmark recorded 293 concurrent descriptor reads, +zero read failures, read p95 1520 ms / maximum 2077 ms, 5962 HTTP probes with six +failures at shutdown, HTTP p95 42 ms / maximum 937 ms, at least 1,045,213,184 bytes +available memory, maximum temperature 44.594 C, maximum piezo receipt age 1768 ms, +and peak benchmark process-tree RSS 396,940 KiB. Canonical observations remained +unchanged. These observations support cancellation behavior but do not establish +a production resource policy or a successful full-night replay. + +Synthetic measurements and probe metadata were copied locally for analysis. +After confirming all benchmark descendants had exited, its exact staging directory +was removed; both installed services were active. Repeat full qualification in an +uninterrupted window before enabling the production replay host. + +Retained RAW cleanup runs through the coordinator's serialized work lane when +its host performs startup/periodic collection. It uses a fixed Python module, +server-derived database/storage paths, a two-minute deadline, bounded diagnostics, +and a count-only response. Admission is checked before execution and every five +seconds; host shutdown cancels the child. Released artifacts and unpublished +orphans retain their one-hour grace periods and cross-process publication locks. +An absent retained directory is a no-op. Enabling the production host still +requires deployment qualification; adding cleanup does not enable replay. + +### Per-run process measurements + +New retained waveform runs include `resources` with version +`waveform-process-resources-v1`. Monotonic wall time and process CPU time cover +input reading, build verification, replay, and the final build check through +result generation. They exclude interpreter startup before `run`, final result +serialization, and teardown. Peak RSS is the kernel process-lifetime high water +mark: Linux `/proc/self/status` `VmHWM` converted from KiB to bytes (including +Yocto without Python's `resource` extension), or Darwin `ru_maxrss` in bytes. +Unavailable telemetry is null; current RSS is never substituted for a peak. + +Waveform reports summarize the sequential processes as total measured wall/CPU +time and the largest individual process memory peak. Shared configuration hashes +are executed once and are counted once. Missing legacy measurements make the +corresponding aggregate unavailable. Compact report projections retain these +measurements, and the review UI labels their units and scope. They exclude Node +coordination/scoring and do not measure live ingestion health, so they cannot +alone satisfy a whole-job operational promotion policy. Execution measurements +are part of the retained report hash; replay determinism is assessed on derived +readings rather than equality of timing counters between executions. + +Successful jobs also retain `jobOperations` (`backtest-job-operations-v1`), +written by the coordinator after its final admission check rather than supplied +by the scoring child or experiment. Its monotonic duration runs from initial +admission through publication preparation. The supervisor sends ingestion age +from the same fresh health sample used to admit work over the internal process +channel. Only count, maximum observed age, largest monotonic observation gap, +and qualification identity are retained; raw sensor frames are not included. + +`allChecksMeasured` means every admission check supplied evidence under one +qualification identity, with at least initial and final checks. It does not mean +continuous monitoring: the nominal check interval is five seconds, delays are +reported, and intervening health is unobserved. Boolean-only admission callbacks +used by older tools/tests remain supported but produce unavailable health +evidence. A failed or invalid health callback cannot fall back to a permissive +boolean callback. Admission withdrawal still interrupts work without publishing +a result. The waveform report UI shows these observations separately from Python +process costs. Whole-job memory evidence and designated promotion policy remain +separate requirements. + +The supervisor now also samples summed RSS for the owned replay process group +at a nominal one-second interval, plus admission boundaries. Linux procfs scans +are bounded to 4,096 process IDs and 64 group members with at most 16 concurrent +reads. Only group IDs and RSS are read; arguments, environments and sensor data +are excluded. Missing procfs, unsupported platforms, malformed data, or exceeded +bounds yield unavailable samples. Exited processes may disappear during a scan. + +The final admission receipt carries this accumulator into `jobOperations.memory` +(`process-group-rss-observations-v1`): attempts, valid sample count, largest +observed summed RSS and largest observation gap. The coordinator and its worker +children share the owned group; the main application process is outside it. +The sampler stops when the coordinator closes, with no detached polling loop. +This is not an exact kernel peak: short-lived processes or peaks between scans +may be missed, scans are not atomic, and shared pages may be counted more than +once. Reports retain these scope limitations rather than substituting this value +for the per-process lifetime high-water marks. + +### Completed eight-hour synthetic qualification (2026-10-13 UTC) + +The uninterrupted run on `192.168.1.88` completed all three distinct +configurations over the 28,800-second dual-channel 500 Hz synthetic fixture +(95,621,299 compressed bytes). Staged production code was `e1ab3942`, with the +completion-validating benchmark runner from `2ff9bf58`. Completion required a +successful job, retained waveform report, matching dataset/Python identities, +three unique expected configuration hashes, and positive scored outputs for each. +Canonical synthetic observations remained unchanged. + +- Replay: 2,625,664 ms (43.8 minutes); retention: 22,064 ms; summary: 2,849 ms. +- 366 concurrent dataset reads, zero failures; p95 1,516 ms, maximum 1,867 ms. +- Internal probe: 7,456 API requests, zero failures; p95 41.5 ms, maximum 311 ms. +- External probe: 16,905 API requests, zero failures; p95 38.6 ms, maximum 1,429 ms. +- Peak observed benchmark process-tree RSS: 413,163,520 bytes (394 MiB). +- Minimum sampled available memory: 1,132,576,768 bytes; maximum temperature 43.777°C. +- Maximum sampled ingestion age: 1,250 ms. In 68 samples a newer frame arrived + after the collector captured its start timestamp, producing a negative age; + those are collection-timing artifacts, not measured negative latency. +- Replay parent event-loop p95 23.1 ms, maximum 96.7 ms. + +Retained report hash: +`bc1ac53d9168a5f2e6fd521212c69f6ae53939911e7ba17e4f1ccb0fb8e7cdcc`. +Compound build: +`375b2ae0e0e81545b234f1ee3357a2b0cd4cde222cd413591c71c88766b75607`. +Python build: +`23670017f01a37b0281fdea016440261af6f9e6519bff5f294b9056d6f5caf29`. + +Synthetic measurements, probe output, and compact summary were retained locally +under `/tmp/biometrics-waveform-qualification-results/fullnight-success/`. +The owned pod staging directory was removed only after successful termination +and a descendant-process check. Core (`sleepypod.service`) and piezo services +remained active and the health API returned 200 afterward. These results qualify +the stated staged build and fixture; later telemetry and objective changes need +additional target verification. Synthetic signal recovery does not establish +agreement with real reference exports or clinical accuracy. + +A subsequent target smoke test exercised the new process-group sampler against +an isolated Node child holding a touched 32 MiB buffer. Four of four observations +were available; peak observed RSS was 83,582,976 bytes, maximum observation gap +999.8 ms. The child exited and the temporary sampler directory was removed; +both application and piezo services remained active. This verifies Linux +collection and cleanup, not a repeat full-night qualification of newer code. + +### Frozen sampled operational objectives + +Numeric objective policies may explicitly opt into +`operational.observations.version = sampled-process-group-v1`, with maximum +memory and ingestion observation gaps. Without that opt-in, sampled RSS cannot +satisfy the existing peak-memory field. After the final admission check, the +coordinator replaces the scorer's operational evidence with its own measured +job duration, group RSS, and ingestion observations. Missing/partial samples +remain unavailable; excessive observation gaps or a qualification identity +mismatch prevent improvement. Threshold values and sampling semantics remain +part of the frozen experiment protocol. This supplies operational evidence; +a designated qualified policy, sufficient independent reference support, and +subgroup checks are still required for validated promotion. + +### Current-build 30-minute smoke (2026-10-13 UTC) + +An isolated run of `4453c0a5` on the same aarch64 pod (Node 24.16.0, locked +Python 3.10.4 environment) completed three distinct configurations over an +1,800-second dual-channel 500 Hz synthetic fixture, 5,976,451 compressed bytes. +It used the current packaged coordinator/evaluator and common Python sources, +including numerical flock, shared application admission and process-group cleanup +confirmation. Production services continued running; this was not a deployment. + +- Replay 202,760 ms; retention 16,172 ms; summary 1,542 ms. +- 27 concurrent dataset reads, zero failures; p95 1,860 ms. +- 638 internal API requests and 1,545 independent probe requests, zero failures; + respective p95 latencies 65.7 ms and 41.6 ms. +- Peak sampled benchmark process-tree RSS 433,369,088 bytes (413 MiB). +- Minimum available memory 992,690,176 bytes; minimum free disk 3,450,945,536 bytes. +- Maximum temperature 46.696°C; maximum measured ingest age 992 ms. Nine negative + ages reflect collector/frame timing, as in the earlier full-night run. +- Replay parent event-loop p95 28.0 ms, maximum 186.4 ms. + +Completion verifies a retained report, exact dataset/Python/configuration identities, +and positive scored outputs for each configuration. Synthetic canonical rows +remained unchanged. After termination, no process referenced the owned staging +directory; it was removed. Core and piezo services remained active and the health +API returned 200. No production databases, bindings or service configuration changed. + +The [compact measurement summary](performance/biometrics-replay-2026-10-13/smoke-4453c0a5.json) +records exact result/build hashes, explicit test budgets and SHA256 identities of +the inputs and telemetry. Full synthetic telemetry is retained locally at +`/tmp/biometrics-qualification-4453c0a5/`. These are sampled observations, not exact +kernel peaks or continuous guarantees. They verify this short replay workload; +full-night current-build, representative-reference and prospective shadow +qualification remain outstanding. The measured event-loop maximum also prevents +interpreting the admission samples as a guarantee that all delays stayed below +the benchmark's 100 ms admission threshold. No promotion registry or production +budget defaults were derived from this smoke alone. + +### Synthetic shadow supervisor measurement harness + +`scripts/benchmarks/biometrics-shadow-runner.ts` exercises the shipped Node +supervisor and kernel-limited Python shadow child with generated 500 Hz dual +channels. It retains bounded resource receipts, partial failure observations, +output identity, exact Python/runner identities and parent event-loop measurements. +It uses no live waveform subscription, database, credentials or hardware API. + +Bundle with esbuild (`--bundle --platform=node --target=node22 --format=cjs`), +then run on the target Linux pod with arguments `OUTPUT_JSON PYTHON_PATH +MODULES_PATH [SECONDS]`. The default is 1,800 synthetic seconds, processed as +fast as the worker permits; accepted durations are 150–28,800 seconds. Use the +locked installed Python environment and the source modules matching the build. +Wrap it in `biometrics-probe.py` for independent live-service telemetry. Run only +after other replay/shadow measurements finish: the child uses their same kernel +execution lock. The harness's explicit ceilings are measurement stop conditions, +not installed qualification or recommended deployment limits. + +This measures synthetic processing and supervision only. It does not establish +live capture/sink performance, representative artifact robustness, held-out +accuracy or permission to install a shadow qualification registry. + +### Current worker eight-hour run (2026-10-13) + +The current worker/coordinator bytes completed three configurations over 28,800 +synthetic seconds at 500 Hz on the same four-core ARM64, 2 GiB pod. The staged +source was `db52f1da`; subsequent API/UI/prompt changes leave the measured worker +and coordinator hashes unchanged. Exact hashes and artifact identities are in +[the compact evidence](performance/biometrics-replay-2026-10-13/fullnight-db52f1da.json). + +- Replay: 2,756.7 seconds (45.94 minutes); retention: 21.74 seconds; summary: + 2.99 seconds. All operations completed and fixture canonical rows were unchanged. +- 363 concurrent report reads: no failures, p95 1,943.7 ms, maximum 2,183.3 ms. +- 7,390 internal HTTP observations: no failures, p95 57.1 ms, maximum 300.8 ms. + The independent probe recorded 17,568 observations, also without failures, + p95 39.6 ms and maximum 500.1 ms. +- Peak sampled process-tree RSS: 443.1 MiB; minimum available memory: 902.6 MiB; + minimum filesystem space: 3.02 GiB; maximum temperature: 49.965 °C. +- Aggregate CPU busy averaged 68.1%, peaking at 99.0% over a sampling interval. + Maximum one-minute load per CPU was 1.0575. These include live services and + the probe, not just the replay child. +- Maximum observed piezo arrival age: 1,560 ms. There were 71 negative age + differences because the collector records its start before asynchronous reads + and can observe a newer frame afterward; those are not negative ingest latency. +- Parent event-loop p95 during replay: 27.28 ms, maximum 154.01 ms. A 100 ms + admission check is therefore not a continuous event-loop latency guarantee. + +The original process handle exited successfully and no owned worker descendants +remained before the separate shadow measurement started. Core and piezo services +were active afterward. The read-only HTTP probes do not prove actuator-write +latency or clinical performance. Raw measurements remain under +`/tmp/biometrics-qualification-db52f1da/` on the measurement workstation; the +committed evidence records their SHA-256 hashes. No live database or profile was +changed by this benchmark. + +### Shadow supervisor measurement (2026-10-13) + +After the full-night replay released its worker slot, the staged shadow harness +processed 1,800 synthetic seconds at 500 Hz through the shipped Linux process +wrapper and Node supervisor. It completed in 61.34 seconds, producing 30 outputs +and reporting 29.17 seconds of numerical CPU time. The child process-group RSS +peaked at a sampled 113.15 MiB; the independent probe observed 256.5 MiB for the +whole benchmark tree. The completion receipt retains cold-start and insufficient +preroll limitations. + +The probe recorded 456 read-only HTTP observations with no failures (p95 45.41 ms, +maximum 130.39 ms). Minimum available memory was 1,019.5 MiB and maximum temperature +47.513 °C. Parent event-loop p95 was 26.10 ms and maximum 68.62 ms. RSS observation +gaps reached 333.42 ms; this is sampled memory evidence, not an exact kernel peak. + +[The compact receipt](performance/biometrics-replay-2026-10-13/shadow-4ee74c25.json) +retains build identities, completion/cleanup observations, limits and raw-artifact +hashes. The original SSH handle exited zero, no owned descendants remained, and +both live services were active before the owned staging directory was removed. +This run does not exercise the live source/database sink, establish held-out +accuracy or authorize a shadow qualification registry. + +### Target-specific replay deployment policy + +The measured pod uses the [recorded policy](performance/biometrics-replay-2026-10-13/deployment-policy.json). +Its admission floor is 768 MiB available memory and 2 GiB free disk; it refuses +load per CPU above 1.1, temperature above 55 °C, piezo age above five seconds, +stale health samples above two seconds, or a scheduling probe above 100 ms. +These are conservative operating cutoffs around the observed workload, not +universal hardware defaults or guarantees that every transient stays below them. +A temporary refusal is expected when current health exceeds a cutoff. + +The whole-job deadline is two hours; each numerical waveform child has a +30-minute wall deadline and 1,200 CPU-second limit, matching the successful +full-night measurement. Larger jobs can still exceed these ceilings and fail +explicitly. Existing worker serialization, process limits, quotas and health +checks remain enforced. This policy authorizes bounded replay on this measured +pod only; it provides no accuracy designation or shadow qualification. + +Deployment stores the two environment values in +`/etc/sleepypod/biometrics-replay.env`, loaded by +`/etc/systemd/system/sleepypod.service.d/biometrics-replay.conf`. A service restart +is required after changing them. Remove that drop-in and restart to return to +unconfigured admission. No user profile or hardware temperature setting changes +when installing this policy. + +### Live source and isolated sink measurement (2026-10-13) + +The [live operational receipt](performance/biometrics-replay-2026-10-13/live-shadow-operational.json) +records a successful 180-second observation through the loopback sensor stream, +read-only active observations, shared numerical worker and fenced isolated sink. +It retained six comparison points (5,396 bytes), with zero canonical rows in the +isolated database. The worker used 4.95 CPU seconds, sampled peak group RSS of +108.61 MiB and a maximum RSS observation gap of 497.24 ms. The independent probe +made 1,587 read-only HTTP requests with no failures (p95 40.21 ms, maximum +123.67 ms); minimum available memory was 1,082.67 MiB and maximum temperature +44.828 °C. No raw frames or numerical readings were exported in this receipt. + +Two earlier attempts failed: one refused health admission; another reported +unavailable RSS after its worker completed. The latter exposed a process-exit +race fixed in `4c3a6a6e`: disappearance after a validated terminal response now +requires bounded, successful pipe closure. Missing RSS during processing, bad +exit status and a worker that never exits still fail. Regression tests reproduce +all three outcomes. The successful repeat used the fixed supervisor without +relaxing resource limits. Its SSH process exited zero, owned processes were +absent, and all three owned staging directories were removed after saving +metadata receipts locally. + +This trusted measurement harness uses read-only production handles and a fresh +isolated database. It does not install a qualification registry, alter profile +bindings or exercise production registry admission. Cold start, insufficient +preroll, missing occupancy context and missing pump context remain recorded; +this operational check cannot establish accuracy or authorize promotion. diff --git a/docs/biometrics-runtime-activation.md b/docs/biometrics-runtime-activation.md new file mode 100644 index 000000000..77d11fcbb --- /dev/null +++ b/docs/biometrics-runtime-activation.md @@ -0,0 +1,142 @@ +# Biometric runtime activation protocol + +`requestManualActivation` is the configuration transaction for an explicitly +unvalidated manual change. It checks side, exact revision/hash, expected binding +generation and idempotency key before committing the binding, audit event and +immutable outbox payload together. A missing legacy migration prevents a new +binding from discarding existing stage personalization. Reset selects the +current shipped default; rollback selects the previous binding and remains a +distinct audited action. Neither operation deletes revision history. + +An explicit reset can recover malformed legacy stage data that cannot pass the +stage-classifier schema. The original legacy row remains untouched, and the +reset event preserves its value as `evidence.preservedLegacyClassifier` with +`recovery: "invalid_legacy_reset"`. This is audit data, never an executable +profile revision. The binding, evidence and default outbox payload commit +atomically; a publication failure leaves the original state intact. Valid +legacy customizations still require migration before any new binding, including +reset. Recovery does not claim processor adoption. + +For a bound revision whose engine is unavailable, the web recovery view shows +its preserved revision ID and the current default recovery hash. It cannot +produce a valid before/after diff of the unavailable engine and explicitly says +so. The user can request an acknowledged, generation-checked reset to defaults; +the old revision remains the previous binding. Rollback to that revision still +requires its engine to be compatible and never silently substitutes defaults. + +`projectRuntimeConfig` validates the manifest/configuration identity and writes +an immutable side/generation projection into the biometrics database. Duplicate +identical delivery succeeds; a different payload for an existing generation +fails. `deliverRuntimeOutbox` acknowledges delivery only after projection. +Transient failures retry with capped exponential delay; invalid envelopes fail +visibly. These are separate database transactions, so a crash after projection +is handled by idempotent redelivery rather than assumed cross-database atomicity. + +`repairCurrentRuntimeProjections` restores the latest authoritative delivered +payload after biometric history has been cleared. It preserves configuration +ownership and does not manufacture runtime receipts. + +`reconcileActivation` considers only receipts matching the desired generation +and exact bundle hash. Heart-rate, breathing-rate, HRV-index and sleep-stage +components must all report applied with an effective timestamp before the +activation is marked applied. Old-generation or wrong-hash receipts cannot +acknowledge a newer binding. A final generation check fences concurrent intent +changes while reconciling. + +The Node startup hook starts one `globalThis` coordinator after migrations. +Every five seconds it migrates legacy stage personalization, repairs delivered +projections, delivers pending intents and reconciles receipts. Shutdown stops +its timer. Public manual activation, reset and rollback submit exact intent; +status polling remains read-only. Availability does not assert processor health +or adoption. The cross-runtime fixture verifies the public request → Python +boundary → stage receipt → restart → rollback handoff. +There are no hardware-command paths in these services. + +The Python `common.biometrics.runtime` reader independently validates the latest +projected payload, its envelope, shipped manifest and resolved hash. Reads are +bounded before materializing SQLite text. Missing tables, corrupt newest +generations and incompatible engines surface errors; the reader neither creates +schema nor silently selects an older generation. Snapshots retain immutable +configuration bytes and return fresh decoded values. + +Its two-phase boundary selector allows immediate changes at the caller's next +sample boundary and next-session changes only when occupancy is explicitly +false. Unknown occupancy, including an uninitialized restart, does not count as +an empty bed. Newer requests supersede pending ones, stale generations fail, and +a late commit cannot acknowledge a superseded selection. The caller must install +the selected configuration before committing selection and writing receipts. +The live piezo daemon now uses these primitives through `runtime_adapter.py`. +It polls projections every five seconds, installs configurations before ingesting +an accepted dual-channel frame, and writes receipts only after both channel +buffers have accepted that frame. Heart-rate, breathing-rate and HRV-index +receipts are separate from the stage-view consumer's acknowledgment. Heartbeats are +bounded to once per thirty seconds unless the applied configuration changes. + +In single-sleeper mode both physical channels use the home side's profile. +Each computed candidate carries its immutable processing context through +quality arbitration and write retries, including its original logical side and +physical source channel. The observation, quality row and processing bundle +commit in one SQLite transaction. Retained contexts recreate their processing +bundle after history reset. This does not imply raw input retention: provenance +explicitly labels raw identity unavailable and timestamps as processing-clock +based. Measured sensor calibration is not consumed by this numerical path and +its calibration hash remains null. + +Numerical changes reset sample buffers, tracking and presence state; stage-only +changes preserve numerical state. Routing changes reset state to prevent mixing +sleepers. A restart restores only a compatible projection previously acknowledged +by all three numerical components, with fresh estimator warmup. Pending +next-session requests still require known empty occupancy. Invalid projections +retain the actual compatible configuration and report degraded module health; +an incompatible restored configuration falls back to defaults. Missing schema +prevents custom adoption and preserves legacy writes. The coordinator records +fallback intent separately from the consumer receipts proving its adoption. + +The stage reader selects the newest projection for which all three numerical +components acknowledged the exact bundle hash and an effective boundary. Partial +adoption of a newer generation keeps the prior view. Reading the classifier for +use in the client or server records the stage-consumer receipt; merely inspecting +profile overview or enqueueing a backtest does not. No stage receipt is generated +by the background coordinator. Until a consumer requests the new view, the +activation can truthfully remain pending. + +`getClassifier` exposes the applied classifier and provenance separately from a +pending desired classifier. The editor retains pending edits and polls for +adoption. `getSleepStages` includes the selected view with classified results; +its interpretation is explicitly `current_classifier_view`, since the rules +rescore original readings rather than replace historical observations. Backtest +current-stage baselines use this same applied view. + +Legacy `setClassifier` writes now use the profile service and durable outbox. +They retain HR, breathing and HRV overrides, and null clears only stage overrides. +The response retains classifier/isDefault fields and adds pending activation +identity. Existing legacy columns remain migration baselines, never a second +writable authority. These changes are manual/unvalidated. Demo responses label +simulated adoption explicitly. + +`recoverRuntimeIntent` handles obsolete defaults, missing intent and unavailable +pinned engines. It publishes a new generation and audit event atomically with +its outbox payload. Fallback preserves the desired pinned revision, immutable +revision history and the previous rollback target. Repeated recovery passes are +idempotent. If that same pinned engine becomes compatible again, restoration +waits for the next session. Repair of a lost custom boundary also defaults to the +next session. The fallback explanation survives acknowledgment and remains +visible in the stage view and desired-profile warning. + +A conflicting biometric projection is retained for diagnosis and its delivery +marked failed. The next recovery pass creates a new generation; it does not +rewrite the conflicting immutable snapshot. Missing projections after history +reset are simply redelivered with their original identity. + +Profile overview includes numerical processor health separately from historical +adoption. All three components must report the same generation/hash and a +heartbeat no more than 120 seconds old to be current. Missing components, +incomplete adoption, failures, future timestamps and mixed generations are +reported explicitly. The retrospective stage consumer is on demand and is not +subject to that continuous heartbeat test. This reports health; it does not +claim an old activation was never applied or automatically switch algorithms. + +`biometricTuning.listActivationHistory` / `GET /biometric-tuning/activations` +returns side-scoped audit events, fifty per page using a generation cursor. +Results include exact hashes, revision identities, evidence and fallback reason; +internal request hashes and idempotency keys are omitted. diff --git a/docs/performance/biometrics-replay-2026-10-13/deployment-policy.json b/docs/performance/biometrics-replay-2026-10-13/deployment-policy.json new file mode 100644 index 000000000..c19bec11f --- /dev/null +++ b/docs/performance/biometrics-replay-2026-10-13/deployment-policy.json @@ -0,0 +1,17 @@ +{ + "admission": { + "qualificationId": "pod-192-168-1-88-replay-20261013-bda8d03d", + "minimumAvailableMemoryBytes": 805306368, + "minimumFreeDiskBytes": 2147483648, + "maximumLoadPerCpu": 1.1, + "maximumEventLoopDelayMs": 100, + "maximumPiezoAgeMs": 5000, + "maximumTemperatureC": 55, + "maximumSampleAgeMs": 2000 + }, + "budget": { + "jobWallMs": 7200000, + "waveformWallMs": 1800000, + "waveformCpuSeconds": 1200 + } +} diff --git a/docs/performance/biometrics-replay-2026-10-13/deployment-verification.json b/docs/performance/biometrics-replay-2026-10-13/deployment-verification.json new file mode 100644 index 000000000..61be2d857 --- /dev/null +++ b/docs/performance/biometrics-replay-2026-10-13/deployment-verification.json @@ -0,0 +1,100 @@ +{ + "verifiedAt": "2026-10-13T14:14:45.388115+00:00", + "build": { + "branch": "feat/biometrics-calibration", + "commitHash": "4c3a6a6e", + "commitTitle": "fix(biometrics): handle shadow RSS disappearance during clean exit", + "buildDate": "2026-10-13T14:12:14.467Z", + "version": null + }, + "admission": { + "httpStatus": 200, + "result": { + "configured": true, + "qualificationId": "pod-192-168-1-88-replay-20261013-bda8d03d", + "admitted": true, + "reasons": [] + } + }, + "manifest": { + "httpStatus": 200, + "result": { + "manifestHash": "96c2f01a329b4346b4a3f2801ba973650d7235a5c4bd00bd0ded96f1a5768b9b", + "activationAvailable": true + } + }, + "dataPath": { + "httpStatus": 200, + "result": { + "nodes": [ + { + "id": "sensor-piezo", + "status": "ok" + }, + { + "id": "sensor-cap", + "status": "ok" + }, + { + "id": "sensor-temp", + "status": "ok" + }, + { + "id": "frames", + "status": "ok" + }, + { + "id": "dac", + "status": "ok" + }, + { + "id": "piezo-processor", + "status": "ok" + }, + { + "id": "sleep-detector", + "status": "ok" + }, + { + "id": "environment-monitor", + "status": "ok" + }, + { + "id": "dac-monitor", + "status": "ok" + }, + { + "id": "database", + "status": "ok" + }, + { + "id": "scheduler", + "status": "ok" + }, + { + "id": "out-vitals", + "status": "ok" + }, + { + "id": "out-sleep", + "status": "ok" + }, + { + "id": "out-temp", + "status": "ok" + }, + { + "id": "out-live", + "status": "idle" + } + ], + "verdict": "ok" + } + }, + "services": { + "sleepypod.service": "active", + "sleepypod-piezo-processor.service": "active", + "sleepypod-sleep-detector.service": "active", + "sleepypod-environment-monitor.service": "active" + } +} diff --git a/docs/performance/biometrics-replay-2026-10-13/fullnight-db52f1da.json b/docs/performance/biometrics-replay-2026-10-13/fullnight-db52f1da.json new file mode 100644 index 000000000..13851cfe2 --- /dev/null +++ b/docs/performance/biometrics-replay-2026-10-13/fullnight-db52f1da.json @@ -0,0 +1,93 @@ +{ + "scope": "Eight-hour synthetic replay resource measurement; not shadow, clinical or activation qualification", + "sourceCommit": "db52f1da", + "architecture": "aarch64", + "nodeVersion": "24.16.0", + "pythonVersion": "3.10.4", + "completed": true, + "fixtureSeconds": 28800, + "compressedBytes": 95621299, + "canonicalObservationsUnchanged": true, + "readFailures": 0, + "replayBudget": { + "jobWallMs": 7200000, + "waveformWallMs": 1800000, + "waveformCpuSeconds": 1200 + }, + "operations": [ + { + "phase": "snapshot", + "succeeded": true, + "elapsedMs": 1512.0559429999994, + "eventLoopP95Ms": 23.134207, + "eventLoopMaxMs": 39.550975, + "parentPeakRssKiB": 130780 + }, + { + "phase": "retain", + "succeeded": true, + "elapsedMs": 21739.24188, + "eventLoopP95Ms": 22.036479, + "eventLoopMaxMs": 51.937279, + "parentPeakRssKiB": 130780 + }, + { + "phase": "waveform-replay", + "succeeded": true, + "elapsedMs": 2756681.940805, + "eventLoopP95Ms": 27.279359, + "eventLoopMaxMs": 154.009599, + "parentPeakRssKiB": 153608 + }, + { + "phase": "summary", + "succeeded": true, + "elapsedMs": 2994.1249830001034, + "eventLoopP95Ms": 22.020095, + "eventLoopMaxMs": 34.865151, + "parentPeakRssKiB": 153752 + }, + { + "summaryBytes": 59919, + "resultHash": "bda8d03d158b04bb3669ff613e0936ed46822c0083c4d37f9f30e5338bc95f97", + "compoundBuildHash": "2b6514629ca87fc82753d6b0940b1c94d754f834585ef115387eaf536ef58df1", + "pythonBuildHash": "212c5643dd24cae1cbf9c346b0729d7ea144c453cd56370ec810223c508a5bc3" + } + ], + "concurrentReadMs": { + "count": 363, + "p95": 1943.6675560000003, + "max": 2183.3170790000004 + }, + "internalApiMs": { + "count": 7390, + "p95": 57.09343400015496, + "max": 300.81892600000015 + }, + "internalApiFailures": 0, + "externalApiMs": { + "count": 17568, + "p95": 39.62092199799372, + "max": 500.0740819959901 + }, + "externalApiFailures": 0, + "minimumAvailableMemoryBytes": 946417664, + "minimumFreeDiskBytes": 3243700224, + "maximumTemperatureC": 49.965, + "maximumIngestionAgeMs": 1560, + "negativeSampleAgeCount": 71, + "peakObservedBenchmarkTreeRssBytes": 464650240, + "maximumBenchmarkProcessCount": 4, + "artifactSha256": { + "benchmark.cjs": "bc1dc66c597b020480830548c2de7368a7bdbe2e68a440779c7de79b8254145f", + "payload.tgz": "9cf50a0a8488a1ef8f79ccc41bdd7d086c9dd68bcbba61156d11d8f768813a36", + "measurements.json": "696bdd6462504844c90936afcfe1c8cac7429d141f16893270292cdcc59c29bc", + "probe-28800.json": "e9865a8019e8e3360b022daab587ff5397d317bb9e17c362231ebc3288747af3", + "fixture-28800.json": "165071ec9b67e9dd2143575e8c2758b99090d47c1fb6412291b778d4be59dce0" + }, + "workerSha256": "c635a135ad0cfc2a327292968d08ab043ca8b523dd4c0149be6b793c8a192a5e", + "coordinatorSha256": "c31f3e6dc614cc688d06d25158318cea9c48f3f9ca3d157b576f1269e7672c05", + "maximumLoadPerCpu": 1.0575, + "meanAggregateCpuBusyPercent": 68.06810450124667, + "maximumAggregateCpuBusyPercent": 99.00497512437812 +} diff --git a/docs/performance/biometrics-replay-2026-10-13/live-shadow-operational.json b/docs/performance/biometrics-replay-2026-10-13/live-shadow-operational.json new file mode 100644 index 000000000..91e9c6862 --- /dev/null +++ b/docs/performance/biometrics-replay-2026-10-13/live-shadow-operational.json @@ -0,0 +1,81 @@ +{ + "version": "live-shadow-operational-measurement-v1", + "date": "2026-10-13", + "runtimeFix": "shadow RSS disappearance during clean exit", + "scope": "loopback live input and read-only active observations with isolated sink; not production registry admission or accuracy qualification", + "windowSeconds": 180, + "state": "completed", + "reason": "observation_window_complete", + "isolatedSink": { + "pointCount": 6, + "byteCount": 5396, + "canonicalRows": 0 + }, + "build": { + "runnerHash": "6a2966046edffea389604b28b89a7d9b4417dd3dcda41e64f7adab2103b11d03", + "pythonBuildHash": "b16390a3796398903b80db34a1e758d6e8ee3fb30dcfc4034e4b30bbdb13d0e9", + "manifestHash": "96c2f01a329b4346b4a3f2801ba973650d7235a5c4bd00bd0ded96f1a5768b9b" + }, + "worker": { + "inputBytes": 1013420, + "outputBytes": 118657, + "peakObservedRssBytes": 113881088, + "rssSamples": 1192, + "maximumObservationGapMs": 497.2393699999993, + "budgets": { + "cpuSeconds": 120, + "addressSpaceBytes": 805306368, + "maximumRssBytes": 402653184, + "wallMs": 270000, + "acknowledgmentMs": 30000, + "idleMs": 10000, + "rssIntervalMs": 100, + "rssTimeoutMs": 1000 + }, + "memoryScope": "sum_rss_of_shadow_process_group", + "exactKernelPeak": false, + "cpuMs": 4953.220225 + }, + "limitations": [ + "cold_start_without_serialized_state", + "less_than_five_minutes_preroll", + "missing_occupancy_context", + "missing_pump_context" + ], + "probe": { + "samples": 529, + "httpRequests": 1587, + "httpFailures": 0, + "httpP95Ms": 40.213834996393416, + "httpMaxMs": 123.67384999379283, + "peakProcessTreeRssBytes": 295608320, + "minimumAvailableMemoryBytes": 1135259648, + "maximumTemperatureC": 44.828, + "meanHostCpuPercent": 56.69213951033886, + "maximumHostCpuPercent": 96.59574468085107 + }, + "cleanup": { + "launchExitCode": 0, + "ownedProcessesAbsentAfterExit": true, + "remoteStageRemoved": true + }, + "qualificationInstalled": false, + "profileBindingsChanged": false, + "priorAttempts": [ + { + "state": "failed", + "reason": "Measurement health admission refused", + "cleanupConfirmed": true + }, + { + "state": "failed", + "reason": "Shadow RSS observation unavailable", + "cleanupConfirmed": true + } + ], + "runtimeFixCommit": "4c3a6a6e", + "rawArtifactSha256": { + "biometrics-live-shadow-third-summary.json": "7e5dca3d1c4a6b3e0360d256a203054ac694b42481eee3ea3b7a333dedc858a9", + "biometrics-live-shadow-third-probe.json": "1b86d9a5c1f2fa1f045b19cd310f7dafb09673f1017c7b43e376884888f01d10" + } +} diff --git a/docs/performance/biometrics-replay-2026-10-13/shadow-4ee74c25.json b/docs/performance/biometrics-replay-2026-10-13/shadow-4ee74c25.json new file mode 100644 index 000000000..f1c02fe11 --- /dev/null +++ b/docs/performance/biometrics-replay-2026-10-13/shadow-4ee74c25.json @@ -0,0 +1,117 @@ +{ + "version": "synthetic-shadow-supervisor-measurement-v1", + "scope": "accelerated synthetic frames through shipped supervisor and kernel-limited child; no live capture, database sink, held-out accuracy or installation qualification", + "runnerHash": "aaf032040970edf5758d0dbb358afe7044ce0274ee82492b5fd6a62925ef930d", + "pythonBuildHash": "b16390a3796398903b80db34a1e758d6e8ee3fb30dcfc4034e4b30bbdb13d0e9", + "node": { + "version": "v24.16.0", + "platform": "linux", + "arch": "arm64" + }, + "manifestHash": "96c2f01a329b4346b4a3f2801ba973650d7235a5c4bd00bd0ded96f1a5768b9b", + "configHash": "933fb4329b8c24091565b43388576d641f2806e0fda52bad44f2a2968b997aba", + "seconds": 1800, + "sampleRateHz": 500, + "channels": 2, + "wallMs": 61340.47504500001, + "outputCount": 30, + "outputHash": "e75ead5601fc3c5da660c9778dea0f6a2c999900a4d345754a0ed07f0b4970cf", + "receipt": { + "health": { + "state": "completed", + "reason": "window_complete", + "records": 1800, + "outputs": 30, + "cpu_ms": 29170.628042, + "input_hash": "a523ceeec2382b220c9f95874c22d74922c21db93dcb313d10824b60e8c27ad2", + "counts": { + "records": 1800, + "accepted_frames": 1800, + "empty_channel_frames": 0, + "pump_gated_frames": 0, + "preroll_outputs": 0, + "scored_outputs": 30 + }, + "limitations": [ + "cold_start_without_serialized_state", + "less_than_five_minutes_preroll" + ] + }, + "inputBytes": 9997366, + "outputBytes": 998314, + "peakObservedRssBytes": 118648832, + "rssSamples": 347, + "maximumObservationGapMs": 333.41997099999935, + "budgets": { + "cpuSeconds": 1200, + "addressSpaceBytes": 805306368, + "maximumRssBytes": 402653184, + "wallMs": 1800000, + "acknowledgmentMs": 30000, + "idleMs": 30000, + "rssIntervalMs": 100, + "rssTimeoutMs": 1000 + }, + "memoryScope": "sum_rss_of_shadow_process_group", + "exactKernelPeak": false + }, + "partial": { + "kind": "partial_supervisor_observation", + "health": { + "state": "completed", + "reason": "window_complete", + "records": 1800, + "outputs": 30, + "cpu_ms": 29170.628042, + "input_hash": "a523ceeec2382b220c9f95874c22d74922c21db93dcb313d10824b60e8c27ad2", + "counts": { + "records": 1800, + "accepted_frames": 1800, + "empty_channel_frames": 0, + "pump_gated_frames": 0, + "preroll_outputs": 0, + "scored_outputs": 30 + }, + "limitations": [ + "cold_start_without_serialized_state", + "less_than_five_minutes_preroll" + ] + }, + "inputBytes": 9997366, + "outputBytes": 998314, + "diagnosticBytes": 0, + "peakObservedRssBytes": 118648832, + "rssSamples": 347, + "maximumObservationGapMs": 333.41997099999935, + "budgets": { + "cpuSeconds": 1200, + "addressSpaceBytes": 805306368, + "maximumRssBytes": 402653184, + "wallMs": 1800000, + "acknowledgmentMs": 30000, + "idleMs": 30000, + "rssIntervalMs": 100, + "rssTimeoutMs": 1000 + }, + "cleanupConfirmed": true, + "memoryScope": "sum_rss_of_shadow_process_group", + "exactKernelPeak": false + }, + "failure": null, + "parentEventLoopP95Ms": 26.099711, + "parentEventLoopMaxMs": 68.616191, + "parentPeakRssKiB": 150836, + "probe": { + "requests": 456, + "failures": 0, + "p95Ms": 45.41382499883184, + "maximumMs": 130.38527100434294, + "maximumTemperatureC": 47.513, + "minimumAvailableMemoryBytes": 1069072384, + "peakBenchmarkTreeRssBytes": 268959744 + }, + "artifactSha256": { + "shadow-measurements.json": "e57bf3bd10374b31059f60e39f8645d2be158ab88d2eb9a1abf1baa9b9a786bd", + "shadow-probe.json": "13fd0f3723b6ebc5650cc52868d8c02f5f4bee6cb0629303b53998f09ee883cd" + } +} diff --git a/docs/performance/biometrics-replay-2026-10-13/smoke-4453c0a5.json b/docs/performance/biometrics-replay-2026-10-13/smoke-4453c0a5.json new file mode 100644 index 000000000..c5c3d9078 --- /dev/null +++ b/docs/performance/biometrics-replay-2026-10-13/smoke-4453c0a5.json @@ -0,0 +1,88 @@ +{ + "scope": "30-minute synthetic worker resource smoke; not full-night, shadow, clinical or activation qualification", + "sourceCommit": "4453c0a5", + "architecture": "aarch64", + "nodeVersion": "24.16.0", + "pythonVersion": "3.10.4", + "completed": true, + "fixtureSeconds": 1800, + "compressedBytes": 5976451, + "canonicalObservationsUnchanged": true, + "readFailures": 0, + "replayBudget": { + "jobWallMs": 900000, + "waveformWallMs": 240000, + "waveformCpuSeconds": 180 + }, + "operations": [ + { + "phase": "snapshot", + "succeeded": true, + "elapsedMs": 1560.2145219999984, + "eventLoopP95Ms": 21.987327, + "eventLoopMaxMs": 29.081599, + "parentPeakRssKiB": 129868 + }, + { + "phase": "retain", + "succeeded": true, + "elapsedMs": 16172.227184, + "eventLoopP95Ms": 22.233087, + "eventLoopMaxMs": 42.532863, + "parentPeakRssKiB": 129868 + }, + { + "phase": "waveform-replay", + "succeeded": true, + "elapsedMs": 202759.751582, + "eventLoopP95Ms": 28.049407, + "eventLoopMaxMs": 186.384383, + "parentPeakRssKiB": 129868 + }, + { + "phase": "summary", + "succeeded": true, + "elapsedMs": 1541.8592840000056, + "eventLoopP95Ms": 21.790719, + "eventLoopMaxMs": 24.936447, + "parentPeakRssKiB": 129868 + }, + { + "summaryBytes": 58915, + "resultHash": "ed2505dfb5cc91be94ebeea9f26bddc5ee214c95e9ca730eebb459dbff9c989d", + "compoundBuildHash": "3f9ee531c01c501f543278dfb5641eb57eea9cf2906d7ce2f94f01d6ab1a2e34", + "pythonBuildHash": "0eb411804b1b4ba608a1a8db11406dcf983f0ff9bea0f2d8c5eb027b6a187254" + } + ], + "concurrentReadMs": { + "count": 27, + "p95": 1859.888511000012, + "max": 1860.4550230000168 + }, + "internalApiMs": { + "count": 638, + "p95": 65.67037400000845, + "max": 305.6198179999999 + }, + "internalApiFailures": 0, + "externalApiMs": { + "count": 1545, + "p95": 41.55511000135448, + "max": 129.56501200096682 + }, + "externalApiFailures": 0, + "minimumAvailableMemoryBytes": 992690176, + "minimumFreeDiskBytes": 3450945536, + "maximumTemperatureC": 46.696, + "maximumIngestionAgeMs": 992, + "negativeSampleAgeCount": 9, + "peakObservedBenchmarkTreeRssBytes": 433369088, + "maximumBenchmarkProcessCount": 4, + "artifactSha256": { + "benchmark.cjs": "bcfd8e5024a606aa1ede8f33481a884d553cc4344eeeb4fcd7c610a943e69984", + "payload.tgz": "82ace2b631e042d8c0993c21d5923580df614357bd00fc598692588b6e485543", + "measurements.json": "d0eb400d98dd78c12921b2bc2f42c1471c3d6a4e529c4b01be863a3f64c95f7e", + "probe.json": "134ded3108c2f678cfa8aafec1bee782ab502820b632214c2ae280c9e581f330", + "fixture-1800.json": "861e719fb0c8bad1ddf89c625e7a19388e32dc35984ff90504c00b3c3e06e4a6" + } +} diff --git a/package.json b/package.json index b2c5c24f6..ee79c37d4 100644 --- a/package.json +++ b/package.json @@ -29,7 +29,8 @@ "tsc": "tsc --noEmit", "prepare": "git rev-parse --git-dir >/dev/null 2>&1 && git config core.hooksPath .githooks || true", "build:biometrics-worker": "node scripts/build/biometrics-worker.mjs", - "biometrics:experiment": "tsx scripts/algorithm-experiments/cli.ts" + "biometrics:experiment": "tsx scripts/algorithm-experiments/cli.ts", + "test:browser": "node tests/browser/run.mjs" }, "lint-staged": { "*.{ts,tsx}": "eslint --fix", diff --git a/src/components/Settings/SettingsScreen.tsx b/src/components/Settings/SettingsScreen.tsx index 920d46c99..8fce69d2e 100644 --- a/src/components/Settings/SettingsScreen.tsx +++ b/src/components/Settings/SettingsScreen.tsx @@ -22,6 +22,7 @@ import { TapGestureConfig } from './TapGestureConfig' import { MqttSettingsForm } from './MqttSettingsForm' import { HomeKitConfig } from './HomeKitConfig' import { ArchivePushSettingsForm } from './ArchivePushSettingsForm' +import { BiometricCalibration } from '@/src/components/biometrics/Calibration/BiometricCalibration' import { SleepStageRules } from './SleepStageRules' import { AppearanceSettings, TempControlPicker, TempDisplayControl, ThemeControl, TimeFormatControl, UnitsControl } from './AppearanceSettings' import { SectionColumns } from './SettingsLayout' @@ -125,6 +126,8 @@ function SectionBody({ section, side, onSideChange }: { return {data => } case 'gestures': return + case 'biometric-calibration': + return case 'sleep-stages': return case 'appearance': diff --git a/src/components/Settings/SleepStageRules.tsx b/src/components/Settings/SleepStageRules.tsx index 9343a2900..9998401f7 100644 --- a/src/components/Settings/SleepStageRules.tsx +++ b/src/components/Settings/SleepStageRules.tsx @@ -40,12 +40,14 @@ const sameRuleset = (a: StageRuleset, b: StageRuleset) => JSON.stringify(a) === export function SleepStageRules({ side, onSideChange }: { side: Side, onSideChange: (side: Side) => void }) { const utils = trpc.useUtils() const { leftName, rightName } = useSideNames() - const query = trpc.sleepStages.getClassifier.useQuery({ side }) + const query = trpc.sleepStages.getClassifier.useQuery({ side }, { + refetchInterval: query => query.state.data?.pending ? 5000 : false, + }) const save = trpc.sleepStages.setClassifier.useMutation({ onSuccess: () => utils.sleepStages.getClassifier.invalidate({ side }), }) - const saved = query.data?.classifier.ruleset + const saved = query.data?.pending?.classifier.ruleset ?? query.data?.classifier.ruleset const [draft, setDraft] = useState(null) const [editing, setEditing] = useState(null) const [nights, setNights] = useState(1) @@ -60,7 +62,7 @@ export function SleepStageRules({ side, onSideChange }: { side: Side, onSideChan } const validation = draft ? stageRulesetSchema.safeParse(draft) : null - const preview = useStagePreview(side, validation?.success ? validation.data : null, saved ?? null, nights) + const preview = useStagePreview(side, validation?.success ? validation.data : null, query.data?.classifier.ruleset ?? null, nights) if (query.error) { return ( @@ -90,6 +92,14 @@ export function SleepStageRules({ side, onSideChange }: { side: Side, onSideChan return (
+ {query.data.pending && ( +

+ {`Saved change ${query.data.pending.generation}; waiting for processor adoption. Displayed nights still use the acknowledged rules.`} +

+ )} + {(query.data.pending?.warning ?? query.data.provenance?.warning) && ( + {query.data.pending?.warning ?? query.data.provenance?.warning} + )} {/* Toolbar: profile, state, preset, reset */}
save.mutate({ side, classifier: null })} > Reset to default diff --git a/src/components/Settings/sections.ts b/src/components/Settings/sections.ts index 69ec4b032..756bb53b4 100644 --- a/src/components/Settings/sections.ts +++ b/src/components/Settings/sections.ts @@ -1,8 +1,9 @@ -import { Cog, Download, Globe, Hand, HardDriveUpload, House, Moon, Palette, RadioTower, User, type LucideIcon } from 'lucide-react' +import { Activity, Cog, Download, Globe, Hand, HardDriveUpload, House, Moon, Palette, RadioTower, User, type LucideIcon } from 'lucide-react' export const SECTIONS = [ { id: 'appearance', label: 'Appearance', icon: Palette, description: 'How this device shows the app' }, { id: 'backup', label: 'Backup', icon: HardDriveUpload, description: 'Nightly archive to your own storage' }, + { id: 'biometric-calibration', label: 'Biometric calibration', icon: Activity, description: 'Profiles, processing state and evaluation history' }, { id: 'device', label: 'Device', icon: Cog, description: 'Pod-wide hardware and maintenance' }, { id: 'gestures', label: 'Gestures', icon: Hand, description: 'What taps on the cover do' }, { id: 'homekit', label: 'HomeKit', icon: House, description: 'Apple Home bridge' }, diff --git a/src/components/Settings/tests/SleepStageRules.test.tsx b/src/components/Settings/tests/SleepStageRules.test.tsx index a371e1f16..ab8d37dd9 100644 --- a/src/components/Settings/tests/SleepStageRules.test.tsx +++ b/src/components/Settings/tests/SleepStageRules.test.tsx @@ -19,6 +19,7 @@ const trpcMock = vi.hoisted(() => { const invalidate = vi.fn() const state = { classifier: null as unknown as StageClassifier, + pending: null as { classifier: StageClassifier, generation: number, warning: string | null } | null, movementOnly: null as unknown as StageClassifier['ruleset'], isDefault: true, changes: [] as unknown[], @@ -45,6 +46,7 @@ const trpcMock = vi.hoisted(() => { ? undefined : { classifier: state.classifier, + pending: state.pending, isDefault: state.isDefault, changes: state.changes, presets: [ @@ -147,6 +149,7 @@ beforeEach(() => { trpcMock.invalidate.mockClear() Object.assign(trpcMock.state, { classifier: DEFAULT_STAGE_CLASSIFIER, + pending: null, isDefault: true, changes: [], preview: undefined, @@ -181,6 +184,16 @@ describe('SleepStageRules', () => { expect(screen.getByRole('button', { name: /Reset to default/ }).hasAttribute('disabled')).toBe(true) }) + it('edits pending rules while showing adoption status and allowing reset', () => { + trpcMock.state.pending = { classifier: { engine: 'rules', ruleset: MOVEMENT_ONLY_STAGE_RULESET }, generation: 7, warning: null } + renderRules() + expect(screen.getByRole('status').textContent).toContain('waiting for processor adoption') + expect(screen.getByRole('button', { name: 'Save' }).hasAttribute('disabled')).toBe(true) + expect(screen.queryByLabelText('Rule 2 stage')).toBeNull() + fireEvent.click(screen.getByRole('button', { name: /Reset to default/ })) + expect(trpcMock.setMutate).toHaveBeenCalledWith({ side: 'left', classifier: null }) + }) + it('opens a rule for editing from a dotted value and saves the edited threshold', () => { renderRules() open('Rule 2 condition 1 value') diff --git a/src/components/Settings/tests/sections.test.ts b/src/components/Settings/tests/sections.test.ts index d591532ae..305c90a0c 100644 --- a/src/components/Settings/tests/sections.test.ts +++ b/src/components/Settings/tests/sections.test.ts @@ -26,7 +26,7 @@ describe('resolveSection', () => { expect(resolveSection('bogus', 'also-bogus')).toBeNull() }) - it('lists all ten sections A–Z', () => { - expect(SECTIONS.map(s => s.id)).toEqual(['appearance', 'backup', 'device', 'gestures', 'homekit', 'mqtt', 'network', 'sides', 'sleep-stages', 'updates']) + it('lists all eleven sections A–Z', () => { + expect(SECTIONS.map(s => s.id)).toEqual(['appearance', 'backup', 'biometric-calibration', 'device', 'gestures', 'homekit', 'mqtt', 'network', 'sides', 'sleep-stages', 'updates']) }) }) diff --git a/src/components/biometrics/BiometricsPage.tsx b/src/components/biometrics/BiometricsPage.tsx index bffb0ddca..aa8b455df 100644 --- a/src/components/biometrics/BiometricsPage.tsx +++ b/src/components/biometrics/BiometricsPage.tsx @@ -7,6 +7,8 @@ */ 'use client' +import { Trans, useLingui } from '@lingui/react/macro' +import type { VitalProvenance } from '@/src/biometrics/profiles/vital-provenance' import { useTimeFormat } from '@/src/providers/TimeFormatProvider' import { useRef, useState, type ReactNode } from 'react' @@ -35,7 +37,7 @@ import { type Side = 'left' | 'right' -interface VitalRow { side: string, timestamp: Date, heartRate: number | null, hrv: number | null, breathingRate: number | null } +interface VitalRow { side: string, timestamp: Date, heartRate: number | null, hrv: number | null, breathingRate: number | null, processingRunId?: string | null, provenance?: VitalProvenance } const RANGES: Array<{ value: BiometricsRange, label: string }> = [ { value: 'night', label: 'Night' }, @@ -80,7 +82,8 @@ function BiometricsBody({ sectionSwitch, now }: { sectionSwitch?: ReactNode, now const referenceQ = trpc.biometrics.getReferenceVitals.useQuery({ side, startDate, endDate, limit: 20000 }, { refetchInterval: 5 * 60_000 }) const rows = (vitalsQ.data ?? []) as VitalRow[] - const points: VitalPoint[] = rows.map(r => ({ t: new Date(r.timestamp).getTime(), hr: r.heartRate, hrv: r.hrv, br: r.breathingRate })) + const points: VitalPoint[] = rows.map(r => ({ t: new Date(r.timestamp).getTime(), hr: r.heartRate, hrv: r.hrv, br: r.breathingRate, + provenanceKey: r.provenance?.processingRunId ?? r.processingRunId ?? 'legacy_unknown', hrvMetricId: r.provenance?.hrvMetricId ?? null })) const sessions = splitSessions(points) const bands = { hr: baselineBand(points, 'hr'), hrv: baselineBand(points, 'hrv'), br: baselineBand(points, 'br') } const reference = ((referenceQ.data ?? []) as Array<{ timestamp: Date | string, heartRate: number }>) @@ -226,6 +229,28 @@ function BiometricsBody({ sectionSwitch, now }: { sectionSwitch?: ReactNode, now
+
+

These are original recorded readings. Charts separate known processing runs and HRV definitions. Legacy readings with unknown provenance remain unverified.

+

The current pod algorithm reports a window IBI variability index, not clinical RMSSD. Older readings may have an unknown definition.

+ {summaryQ.data?.provenance + ? ( +

+ + Recorded configurations: + {' '} + {summaryQ.data.provenance.configurationCount} + ; readings with unknown configuration: + {' '} + {summaryQ.data.provenance.unknownConfigurationRows} + . + +

+ ) + : null} + {summaryQ.data?.provenance?.hrvAggregation === 'incompatible_definitions' + ?

HRV definitions differ or are partly unknown. A combined HRV average is unavailable.

+ : null} +
@@ -497,14 +522,17 @@ function SessionsCard({ sessions, rangeLabel, now, ongoing, stale, lastVitalAt, } function RawLog({ open, onClose, rows, loading, title }: { open: boolean, onClose: () => void, rows: VitalRow[], loading: boolean, title: string }) { + const { t } = useLingui() const columns: Array> = [ { key: 'timestamp', header: 'Time', render: r => {new Date(r.timestamp).toLocaleString()}, sortValue: r => new Date(r.timestamp).getTime() }, { key: 'heartRate', header: 'HR', align: 'right', render: r => fmtNum(r.heartRate), sortValue: r => r.heartRate ?? -1 }, { key: 'hrv', header: 'HRV', align: 'right', render: r => fmtNum(r.hrv), sortValue: r => r.hrv ?? -1 }, { key: 'breathingRate', header: 'BR', align: 'right', render: r => fmtNum(r.breathingRate, 1), sortValue: r => r.breathingRate ?? -1 }, + { key: 'processingRun', header: t`Processing run`, render: r => r.provenance?.processingRunId ?? r.processingRunId ?? t`Unknown` }, + { key: 'hrvDefinition', header: t`HRV definition`, render: r => r.provenance?.hrvMetricId ?? t`Unknown` }, ] return ( - + () + const [selected, setSelected] = useState() + const jobs = trpc.biometricTuning.listBacktests.useQuery({ side, beforeCreatedAtMs: cursor?.createdAtMs, beforeId: cursor?.id }, { refetchInterval: 5000 }) + return ( + + Backtest jobs} subtitle={Execution state is separate from the strength of biometric evidence.} /> + {jobs.error + ? {jobs.error.message} + : !jobs.data + ? + : ( + <> + {jobs.data.items.length === 0 ?

No backtest jobs for this side.

: null} +
    + {jobs.data.items.map(job => ( +
  • +

    {labels[job.state]}

    +

    {job.id}

    +

    {t`Attempt ${job.attempt}`}

    + + {selected === job.id ? : null} +
    • + ))} +
+ {cursor ? : null} + {jobs.data.nextCursor ? : null} + + )} + + ) +} +export function BacktestJobDetails({ id, side }: { id: string, side?: 'left' | 'right' }) { + const { t } = useLingui() + const labels = useJobLabels() + const [showReport, setShowReport] = useState(false) + const job = trpc.biometricTuning.getBacktest.useQuery({ id }, { refetchInterval: query => query.state.data && ['succeeded', 'failed', 'cancelled'].includes(query.state.data.state) ? false : 5000 }) + const cancel = trpc.biometricTuning.cancelBacktest.useMutation() + const utils = trpc.useUtils() + async function requestCancellation() { + const updated = await cancel.mutateAsync({ id }) + utils.biometricTuning.getBacktest.setData({ id }, updated) + await utils.biometricTuning.listBacktests.invalidate() + } + if (job.error) return {job.error.message} + if (!job.data) return + const data = job.data + const candidate = data.configurations.find(config => config.label === 'candidate') + const splits = { tuning: t`Tuning`, validation: t`Validation`, holdout: t`Holdout` } + const configurations: Record = { default: t`System defaults`, current: t`Current stage view`, candidate: t`Candidate` } + return ( +
+

{labels[data.state]}

+

+ {splits[data.split]} + {' — '} + {data.evaluationScope === 'retained_waveform_replay' ? Retained waveform replay : data.evaluationScope === 'stage_rules_on_original_readings' ? Stage rules on original readings : Original stored-reading comparison} +

+

{t`Progress: ${data.progress.completedNights} / ${data.progress.totalNights} nights`}

+ {data.progress.totalNights > 0 ? :

Night progress is not available yet.

} +

{data.progress.phase}

+ {data.error ? {data.error} : null} + {data.state === 'queued' ?

Waiting for a worker and resource admission. No execution is implied by a queued request.

: null} + {data.state === 'succeeded' ?

Execution completed. Review the report before drawing conclusions about agreement or improvement.

: null} + {['queued', 'running', 'interrupted'].includes(data.state) ? : null} + {cancel.error ? {cancel.error.message} : null} + {data.state === 'succeeded' ? : null} + {showReport && data.state === 'succeeded' ? : null} +
+ Pinned job inputs +
+
Dataset
+
{data.datasetId}
+
Dataset hash
+
{data.datasetHash}
+
Evaluator build hash
+
{data.evaluatorBuildHash}
+
+
    + {data.configurations.map(config => ( +
  • + {configurations[config.label] ?? config.label} + {': '} + {config.hash} + {' — '} + {config.revisionId ?? '—'} +
    • + ))} +
+
+
+ ) +} diff --git a/src/components/biometrics/Calibration/BacktestReport.tsx b/src/components/biometrics/Calibration/BacktestReport.tsx new file mode 100644 index 000000000..1a1304311 --- /dev/null +++ b/src/components/biometrics/Calibration/BacktestReport.tsx @@ -0,0 +1,166 @@ +'use client' + +import { useState } from 'react' +import { Trans, useLingui } from '@lingui/react/macro' +import { trpc } from '@/src/utils/trpc' +import { Button, InlineError, Skeleton } from '@/src/components/ds' +import { reportViewSchema, numericReportViewSchema, stageReportViewSchema } from './report-view' +import { NumericReport } from './NumericReport' +import { StageReport } from './StageReport' +import { NumericTrace } from './NumericTrace' +import { ObjectiveThresholds } from './ObjectiveThresholds' +import { ReplayResources } from './ReplayResources' +import { ValidatedProfileActivation } from './ValidatedProfileActivation' +import { StageTrace } from './StageTrace' + +export function BacktestReport({ id, candidate }: { id: string, candidate?: { side: 'left' | 'right', revisionId: string, hash: string } }) { + const result = trpc.biometricTuning.getBacktestSummary.useQuery({ id }, { staleTime: 60000 }) + const [reportHash, setReportHash] = useState() + if (result.data?.contentHash && result.data.contentHash !== reportHash) setReportHash(result.data.contentHash) + return ( +
+ {candidate && reportHash ? : null} + +
+ ) +} + +function BacktestReportContent({ id }: { id: string }) { + const { t, i18n } = useLingui() + const [confirmRelease, setConfirmRelease] = useState(false) + const [showTrace, setShowTrace] = useState(false) + const [showNumericTrace, setShowNumericTrace] = useState(false) + const result = trpc.biometricTuning.getBacktestSummary.useQuery({ id }, { staleTime: 60000 }) + const release = trpc.biometricTuning.releaseBacktestResult.useMutation() + const utils = trpc.useUtils() + if (result.error) return {result.error.message} + if (result.data === undefined) return + if (result.data === null) return

No committed report is available yet.

+ const descriptor = result.data + if (!descriptor.available) return ( +
+

This report was released. Its audit hash remains, but detailed results are unavailable.

+

{descriptor.contentHash}

+
+ ) + const parsed = reportViewSchema.safeParse(descriptor.report) + if (!parsed.success) return This report version or structure is not supported by this view. + const report = parsed.data + const statusLabels = { + insufficient_evidence: t`Insufficient evidence for validated improvement`, + improved: t`Improved agreement under the qualified evaluation policy`, + no_material_change: t`No material improvement under the evaluation policy`, + regressed: t`Regression under the evaluation policy`, + incompatible: t`Incompatible evaluation evidence`, + } + const waveform = report.protocol.mode === 'signal_replay' + const hasStages = report.protocol.mode === 'stage_replay' || (waveform && report.protocol.waveformStages === true) + const numeric = numericReportViewSchema.safeParse(descriptor.report) + const stages = stageReportViewSchema.safeParse(descriptor.report) + const labels: Record = { default: t`System defaults`, current: waveform ? t`Current baseline` : t`Current stage view`, candidate: t`Candidate` } + const splits = { tuning: t`Tuning`, validation: t`Validation`, holdout: t`Holdout` } + const uncertaintyLabels = { not_requested: t`Not requested`, insufficient_nights: t`Insufficient paired nights`, degenerate: t`No estimable variation`, estimated: t`Exploratory interval estimated` } + const percent = (value: number | null) => value === null ? t`Not scored` : new Intl.NumberFormat(i18n.locale || 'en', { style: 'percent', maximumFractionDigits: 2 }).format(value) + async function releaseReport() { + await release.mutateAsync({ id, expectedContentHash: descriptor.contentHash }) + setConfirmRelease(false) + await utils.biometricTuning.getBacktestSummary.invalidate({ id }) + } + return ( +
+

{statusLabels[report.status]}

+ {report.qualificationWarning ? {report.qualificationWarning} : null} + {report.status === 'improved' ?

This is the retained evaluation result, not current activation approval or clinical accuracy. Application checks current evidence and settings again.

: null} +

{splits[report.split]}

+

{t`Current holdout exposure count: ${descriptor.currentHoldoutUseCount}`}

+ {report.evidence === 'reused_holdout_is_validation' || descriptor.currentHoldoutUseCount > 1 ?

Repeated holdout exposure weakens independent evidence. Treat reused holdout results as validation, not untouched holdout evidence.

: null} + {hasStages ?

{t`Assumed stage calibration quality: ${report.calibration.quality}`}

: null} + {waveform ?

Numerical readings were recomputed from retained waveforms. Original stored readings remain unchanged.

: report.protocol.mode === 'comparison' ?

Original stored readings were compared; numerical parameter changes were not replayed.

: null} + {(report.protocol.objectivePolicies ?? (report.protocol.objectivePolicy ? [report.protocol.objectivePolicy] : [])).map(policy => ( + row.policy?.metric === policy.metric || (!row.policy && !report.protocol.objectivePolicies))} /> + ))} + {report.protocol.stageObjectivePolicy ? : null} + {report.limitations.map(limitation =>

{limitation}

)} + {report.stageNightSummary.map(summary => ( +
+

+ {labels[summary.candidate] ?? summary.candidate} + {' / '} + {labels[summary.baseline] ?? summary.baseline} +

+

{t`Scored nights: ${summary.scoredNights}; unscored: ${summary.unscoredNights}; coverage reduced: ${summary.coverageReducedNights}`}

+

{t`Higher agreement: ${summary.improvedAgreementNights}; lower: ${summary.regressedAgreementNights}; unchanged: ${summary.unchangedAgreementNights}`}

+

{t`Equal-night mean agreement difference: ${percent(summary.equalNightMeanAgreementDifference)}`}

+ + + + + + + + + + + + + + + + + + + + + +
Pooled common-support stage scores
ConfigurationAgreementMacro F1
{labels[summary.baseline] ?? summary.baseline}{percent(summary.pooledCommonSupport.baseline.agreement)}{percent(summary.pooledCommonSupport.baseline.macroF1)}
{labels[summary.candidate] ?? summary.candidate}{percent(summary.pooledCommonSupport.candidate.agreement)}{percent(summary.pooledCommonSupport.candidate.macroF1)}
+

Equal-night differences and pooled duration scores use different weighting. Coverage loss must be considered alongside score changes.

+

{uncertaintyLabels[summary.uncertainty.status]}

+ {summary.uncertainty.interval ?

{t`Exploratory 95% interval: ${percent(summary.uncertainty.interval.lower)} to ${percent(summary.uncertainty.interval.upper)}`}

: null} + {summary.uncertainty.limitations.map(limitation =>

{limitation}

)} +
+ ))} + {hasStages ? (stages.success ? : Stage details are unavailable or incompatible.) : null} + {waveform ? : null} + {numeric.success ? : Numerical details are unavailable or incompatible.} + {waveform && numeric.success + ? ( +
+ + {showNumericTrace ? : null} +
+ ) + : null} + {hasStages && numeric.success + ? ( +
+ + {showTrace ? : null} +
+ ) + : null} +
+ Report identity +
+
Report hash
+
{descriptor.contentHash}
+
Dataset hash
+
{report.datasetHash}
+
Protocol hash
+
{report.protocolHash}
+
Evaluator build hash
+
{report.evaluatorBuildHash}
+
+
+ {confirmRelease + ? ( +
+

Release detailed results permanently? The report hash, job history, and holdout exposure audit remain.

+ + +
+ ) + : } + {release.error ? {release.error.message} : null} +
+ ) +} diff --git a/src/components/biometrics/Calibration/BacktestSubmission.tsx b/src/components/biometrics/Calibration/BacktestSubmission.tsx new file mode 100644 index 000000000..f8ade9907 --- /dev/null +++ b/src/components/biometrics/Calibration/BacktestSubmission.tsx @@ -0,0 +1,102 @@ +'use client' + +import { useRef, useState } from 'react' +import { Trans, useLingui } from '@lingui/react/macro' +import { BacktestJobDetails } from './BacktestJobs' +import { trpc } from '@/src/utils/trpc' +import { Button, InlineError, Skeleton } from '@/src/components/ds' + +export function BacktestSubmission({ side, experimentId, datasetId }: { side: 'left' | 'right', experimentId: string, datasetId: string }) { + const { t } = useLingui() + const [cursor, setCursor] = useState<{ createdAtMs: number, id: string }>() + const [candidate, setCandidate] = useState<{ id: string, hash: string, generation: number }>() + const [split, setSplit] = useState<'tuning' | 'validation' | 'holdout'>('tuning') + const [holdoutAck, setHoldoutAck] = useState() + const [attempted, setAttempted] = useState() + const [saved, setSaved] = useState() + const request = useRef<{ fingerprint: string, key: string } | undefined>(undefined) + const profiles = trpc.biometricTuning.listProfileRevisions.useQuery({ side, beforeCreatedAtMs: cursor?.createdAtMs, beforeId: cursor?.id }) + const overview = trpc.biometricTuning.getProfileOverview.useQuery({ side }, { refetchInterval: 5000 }) + const dataset = trpc.biometricTuning.getDataset.useQuery({ id: datasetId }) + const enqueue = trpc.biometricTuning.enqueueBacktest.useMutation() + const utils = trpc.useUtils() + const error = profiles.error ?? overview.error ?? dataset.error + if (error) return {error.message} + if (!profiles.data || !overview.data || !dataset.data) return + if (!dataset.data.available) return

The retained inputs are unavailable for new evaluations.

+ const generation = overview.data.desired.binding.generation + const input = candidate ? { experimentId, candidateRevisionId: candidate.id, expectedBindingGeneration: candidate.generation, split } : undefined + const fingerprint = JSON.stringify(input) + const availableSplits = new Set(dataset.data.manifest.splitProtocol.assignments.map(cohort => cohort.split)) + const valid = candidate && (candidate.generation === generation || attempted === fingerprint) && availableSplits.has(split) && (split !== 'holdout' || holdoutAck === fingerprint) + async function submit() { + if (!input || !valid) return + if (request.current?.fingerprint !== fingerprint) request.current = { fingerprint, key: Array.from(crypto.getRandomValues(new Uint8Array(24)), byte => byte.toString(16).padStart(2, '0')).join('') } + setAttempted(fingerprint) + await enqueue.mutateAsync({ ...input, idempotencyKey: request.current.key }) + setSaved(fingerprint) + void utils.biometricTuning.listBacktests.invalidate({ side }) + void utils.biometricTuning.listExperiments.invalidate({ side }) + } + return ( +
+ {dataset.data.manifest.version === 'waveform-dataset-manifest-v1' ?

The server pins default, current baseline, and candidate configurations, plus the other side’s configuration. Replay uses retained signals and never replaces original observations.

:

The server captures default, current stage-view, and candidate configurations at submission. Stored readings cannot validate numerical parameter changes.

} +

{dataset.data.contentHash}

+
+ {profiles.data.items.length === 0 ?

No saved candidate revisions.

: null} +
    + {profiles.data.items.map(profile => ( +
  • + +
    • + ))} +
+ {cursor ? : null} + {profiles.data.nextCursor ? : null} + {candidate + ? ( +

+ {candidate.id} + {' — '} + {candidate.hash} +

+ ) + : null} + {candidate && candidate.generation !== generation ? : null} + + {split === 'holdout' + ? ( + + ) + : null} + +
+ {enqueue.error ? {enqueue.error.message} : null} + {enqueue.data + ? ( +
+

{t`Backtest request saved: ${enqueue.data.id}`}

+

A saved request does not mean execution has started or succeeded. Inspect the job for its current state.

+

{enqueue.data.evaluatorBuildHash}

+ +
+ ) + : null} +
+ ) +} diff --git a/src/components/biometrics/Calibration/BiometricCalibration.tsx b/src/components/biometrics/Calibration/BiometricCalibration.tsx new file mode 100644 index 000000000..c32e7a9ad --- /dev/null +++ b/src/components/biometrics/Calibration/BiometricCalibration.tsx @@ -0,0 +1,63 @@ +'use client' + +import { useState } from 'react' +import { Trans, useLingui } from '@lingui/react/macro' +import { Button, Card } from '@/src/components/ds' +import { SideTabs } from '@/src/components/Settings/SettingsLayout' +import { SleepStageRules } from '@/src/components/Settings/SleepStageRules' +import { CalibrationOverview } from './CalibrationOverview' +import { CalibrationReferences } from './CalibrationReferences' +import { ReferenceLibrary } from './ReferenceLibrary' +import { ProfileLibrary } from './ProfileLibrary' +import { ProfileRecovery } from './ProfileRecovery' +import { CalibrationBacktests } from './CalibrationBacktests' +import { CalibrationHistory } from './CalibrationHistory' +import { ShadowRuns } from './ShadowRuns' + +type Side = 'left' | 'right' +export function BiometricCalibration({ side, onSideChange }: { side: Side, onSideChange: (side: Side) => void }) { + const { t } = useLingui() + const [view, setView] = useState<'overview' | 'references' | 'backtests' | 'profiles' | 'history' | 'observations'>('overview') + return ( +
+ + + + + {view === 'overview' + ? ( +
+ + +
+ ) + : view === 'references' + ? ( +
+ + +
+ ) + : view === 'backtests' + ? + : view === 'observations' + ? + : view === 'history' + ? + : ( + <> + +

Direct sleep-stage changes are unvalidated manual settings. Saving requests a profile change; processor adoption is reported separately in Overview.

+ + + )} +
+ ) +} diff --git a/src/components/biometrics/Calibration/CalibrationBacktests.tsx b/src/components/biometrics/Calibration/CalibrationBacktests.tsx new file mode 100644 index 000000000..87ad6b528 --- /dev/null +++ b/src/components/biometrics/Calibration/CalibrationBacktests.tsx @@ -0,0 +1,17 @@ +'use client' + +import { BacktestJobs } from './BacktestJobs' +import { ExperimentLibrary } from './ExperimentLibrary' +import { DatasetLibrary } from './DatasetLibrary' +import { DatasetBuilder } from './DatasetBuilder' + +export function CalibrationBacktests({ side }: { side: 'left' | 'right' }) { + return ( +
+ + + + +
+ ) +} diff --git a/src/components/biometrics/Calibration/CalibrationHistory.tsx b/src/components/biometrics/Calibration/CalibrationHistory.tsx new file mode 100644 index 000000000..0d3212224 --- /dev/null +++ b/src/components/biometrics/Calibration/CalibrationHistory.tsx @@ -0,0 +1,67 @@ +'use client' + +import { useState } from 'react' +import { Trans, useLingui } from '@lingui/react/macro' +import { trpc } from '@/src/utils/trpc' +import { Button, Card, CardHeader, InlineError, Skeleton } from '@/src/components/ds' +import { useTimeFormat } from '@/src/providers/TimeFormatProvider' + +export function CalibrationHistory({ side }: { side: 'left' | 'right' }) { + const { t, i18n } = useLingui() + const timeFormat = useTimeFormat() + const [beforeGeneration, setBeforeGeneration] = useState() + const history = trpc.biometricTuning.listActivationHistory.useQuery({ side, beforeGeneration }, { refetchInterval: beforeGeneration === undefined ? 5000 : false }) + const settings = trpc.settings.getAll.useQuery({}, { staleTime: 60000 }) + const timezone = settings.data?.device.timezone ?? 'UTC' + const dates = new Intl.DateTimeFormat(i18n.locale || 'en', { dateStyle: 'medium', timeStyle: 'short', timeZone: timezone, hour12: timeFormat !== '24h' }) + const actions: Record = { activate: t`Apply profile`, reset: t`Use system defaults`, rollback: t`Restore previous profile`, migration: t`Migrate settings`, fallback: t`Compatibility fallback` } + const states: Record = { saved: t`Saved`, pending_delivery: t`Waiting for delivery`, pending_boundary: t`Waiting for a processing boundary`, applied: t`Applied`, failed: t`Failed` } + return ( + + Activation history} subtitle={Past acknowledgements do not establish which profile is running now.} /> +

+ {t`Times shown in ${timezone}`} +

+ {history.error + ? {history.error.message} + : !history.data + ? + : history.data.items.length === 0 + ?

No activation history for this side.

+ : ( +
    + {history.data.items.map(event => ( +
  1. +
    + {actions[event.action]} + {states[event.state]} +
    + +

    + {t`Generation ${event.generation}`} +

    + {event.reason ?

    {event.reason}

    : null} +
    + Exact profile transition +

    + {event.previousHash} + {' '} + → + {' '} + {event.nextHash} +

    +

    + {t`Requested by ${event.requester}`} +

    +
    +
    2. + ))} +
+ )} +
+ {beforeGeneration !== undefined ? : null} + {history.data?.nextGeneration != null ? : null} +
+ + ) +} diff --git a/src/components/biometrics/Calibration/CalibrationOverview.tsx b/src/components/biometrics/Calibration/CalibrationOverview.tsx new file mode 100644 index 000000000..26d68f645 --- /dev/null +++ b/src/components/biometrics/Calibration/CalibrationOverview.tsx @@ -0,0 +1,66 @@ +'use client' + +import { Trans, useLingui } from '@lingui/react/macro' +import { trpc } from '@/src/utils/trpc' +import { Card, CardHeader, InlineError, SettingRow, Skeleton } from '@/src/components/ds' +import { SectionColumns } from '@/src/components/Settings/SettingsLayout' + +export function CalibrationOverview({ side }: { side: 'left' | 'right' }) { + const { t } = useLingui() + const overview = trpc.biometricTuning.getProfileOverview.useQuery({ side }, { refetchInterval: 5000 }) + const activation = trpc.biometricTuning.getActivationStatus.useQuery({ side }, { refetchInterval: 5000 }) + if (overview.error) return {overview.error.message} + if (!overview.data) return + const data = overview.data + const states: Record = { unreported: t`Not reported`, current: t`Current`, stale: t`Stale`, failed: t`Failed`, + mixed_generations: t`Different versions reported`, clock_skew: t`Clock mismatch`, pending_delivery: t`Waiting for delivery`, + pending_boundary: t`Waiting for a processing boundary`, applied: t`Applied` } + const components: Record = { heart_rate: t`Heart rate`, breathing_rate: t`Breathing rate`, hrv_index: t`HRV index` } + return ( + + + Profile and processor} subtitle={Saved settings and processor reports are separate.} /> + Saved profile}>{data.desired.isDefault ? System defaults : Personalized profile} + Requested change}>{activation.data ? states[activation.data.observedState] : t`Not reported`} + Processor health}>{states[data.runtimeHealth.state]} + {activation.error ? {activation.error.message} : null} + {data.desired.warning ?

{data.desired.warning}

: null} + {data.runtimeHealth.components.map(component => ( + + {states[component.state]} + + ))} +
+ Configuration identity +

{data.desired.hash}

+

+ {t`Generation ${data.desired.binding.generation}`} +

+
+ + + What these metrics mean} /> +

The HRV index measures window IBI variability. It is not clinical beat-to-beat RMSSD, and incompatible reference metrics cannot validate it.

+

Historical sleep stages are recomputed with the current classifier. Original stored vitals remain separate from replay results.

+ {data.stageView.provenance.warning ?

{data.stageView.provenance.warning}

: null} + + + )} + right={( + + Measured sensor calibration} subtitle={Sensor measurements are separate from algorithm personalization.} /> + {data.sensorCalibration.length === 0 + ?

No sensor calibration has been reported for this side.

+ : data.sensorCalibration.map(sensor => ( + + {sensor.freshness === 'current' ? t`Current` : sensor.freshness === 'expired' ? t`Expired` : t`Freshness unknown`} + + ))} + {data.limitations.map(limitation =>

{limitation}

)} + + )} + /> + ) +} diff --git a/src/components/biometrics/Calibration/CalibrationReferences.tsx b/src/components/biometrics/Calibration/CalibrationReferences.tsx new file mode 100644 index 000000000..4b01d3a15 --- /dev/null +++ b/src/components/biometrics/Calibration/CalibrationReferences.tsx @@ -0,0 +1,109 @@ +'use client' + +import { useState } from 'react' +import { Trans, useLingui } from '@lingui/react/macro' +import { trpc } from '@/src/utils/trpc' +import { Button, Card, CardHeader, InlineError } from '@/src/components/ds' +import { ReferenceTimeSpan } from './ReferenceTimeSpan' +import { parseReferenceText } from '@/src/biometrics/references/parse' +import type { PreviewSummary, ReferenceImportInput } from '@/src/biometrics/references/contracts' + +export function CalibrationReferences({ side, correction }: { side: 'left' | 'right', correction?: { id: string, subjectId: string, source: string } }) { + const { t } = useLingui() + const [subjectId, setSubjectId] = useState(correction?.subjectId ?? '') + const [source, setSource] = useState(correction?.source ?? '') + const [format, setFormat] = useState<'csv' | 'json'>('csv') + const [text, setText] = useState('') + const [preview, setPreview] = useState<{ input: ReferenceImportInput, summary: PreviewSummary }>() + const [acceptRejected, setAcceptRejected] = useState(false) + const [error, setError] = useState() + const [saved, setSaved] = useState() + const utils = trpc.useUtils() + const previewImport = trpc.biometricTuning.previewImport.useMutation() + const commitImport = trpc.biometricTuning.commitImport.useMutation() + const busy = previewImport.isPending || commitImport.isPending + function invalidate() { + setPreview(undefined) + setAcceptRejected(false) + setError(undefined) + setSaved(undefined) + } + async function inspect() { + invalidate() + try { + const input = { side, subjectId, source, rows: parseReferenceText(format, text) } + const summary = await previewImport.mutateAsync(input) + setPreview({ input, summary }) + } + catch (cause) { setError(cause instanceof Error ? cause.message : t`Reference preview failed.`) } + } + async function save() { + if (!preview) return + setError(undefined) + try { + const result = await commitImport.mutateAsync({ import: preview.input, previewHash: preview.summary.contentHash, acceptRejectedRows: acceptRejected, ...(correction ? { supersedesImportId: correction.id } : {}) }) + setSaved(result.id) + void utils.biometricTuning.listImports.invalidate() + setPreview(undefined) + } + catch (cause) { setError(cause instanceof Error ? cause.message : t`Reference import failed.`) } + } + return ( + + Import reference measurements} subtitle={Preview normalized records before saving them on the pod.} /> +

Use offset-bearing ISO timestamps or Unix milliseconds. Each record needs recordKey, metric, value, unit, and timestamp (or start and end). JSON uses an array; CSV uses these field names as headers. Imports are limited to 2 MiB and 2000 records.

+ {correction ?

{t`Correcting import ${correction.id}. Supply the complete replacement export; the original revision and frozen datasets remain unchanged.`}

: null} + 
{'recordKey,metric,value,unit,timestamp\n1,heart_rate,60,bpm,2026-10-10T23:00:00Z'}
+
+ + + +