Skip to content

fix(release-verify): stop requiring a single present mode across the crosscheck window - #424

Merged
Exoridus merged 4 commits into
nextfrom
fix/present-crosscheck-mixed-mode-oracle
Sep 24, 2026
Merged

Exoridus merged 4 commits into
nextfrom
fix/present-crosscheck-mixed-mode-oracle

Conversation

@Exoridus

@Exoridus Exoridus commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • diagnostics.present-crosscheck required every PresentMon-observed present for the attributed process to share one classified mode across the whole 8-second capture (modes.len() == 1). Real hardware measurement falsified that: DWM legitimately interleaves Composed: Flip and Hardware Composed: Independent Flip for the same process, scattered across the entire window (1127 vs. 157 presents observed in one run).
  • A first fix judged the product's single most-recently-observed present against PresentMon's chronologically last present for the process. Also falsified on hardware: with 253 mode flips in 8s, that comparison was ambiguous or wrong essentially every time -- a single momentary reading is not a meaningful aggregate over an 8-second window.
  • A per-event QPC-identity approach was explored next (thread the product's own decoded PresentStartTime QPC tick out to environment.snapshot.present.presentQpc, match it exactly against PresentMon's independently-captured QPC for the same present) and also abandoned after real hardware testing: two independent ETW consumers of the same present event do not report bit-identical timestamps (confirmed against both PresentMon's --v2_metrics CPUStartQPC and --v1_metrics QPCTime, deltas from single-digit ticks up to several thousand, never exactly 0). Exact per-event identity across two independently-built PresentMon consumers is not achievable on this hardware; that work was reverted before this PR (no presentQpc API surface remains).
  • The final contract compares aggregate present/mode-flip activity over a shared capture window instead of any single-event identity: presents and mode flips are sampled on the product before and after the PresentMon capture, and PresentMon's independent CSV is judged against the resulting deltas. Hard, qualitative rules only: both observers must see presents for the attributed PID; all externally observed modes must be classified; the product's final mode must be one of the externally observed modes; a single stable external mode requires the product to match it with zero flip growth; multiple external modes require the product's modeFlipCount to also have grown (no magnitude requirement).
  • No numeric present/mode-flip-count tolerance gate. An initial version compared the product's and PresentMon's counts with a 15% relative tolerance. Real hardware testing falsified that too: present-count agreement ranged from ~72% under a busy desktop (heavy system-wide DXGI present traffic from other windows, two independent 24s runs) to ~94% on a quiet desktop, and did not converge with a longer capture window -- ruling out capture-boundary skew as the explanation. Mode-flip agreement sat around ~86% independent of desktop load. Since the two observers do not agree to any stable percentage, present/mode-flip counts are recorded as evidence only, never gated numerically.
  • Evidence records both observers' present/mode-flip counts (before/after for the product), PresentMon's transition count and full mode distribution.
  • Bumps diagnostics.present-crosscheck to scenario revision 2 (unreleased; not yet merged, so not a 3rd revision).

Known follow-up (not fixed in this PR)

Under busy-desktop conditions, the product's own present-diagnostics undercounted PresentMon's independently-attributed presents for the same PID by ~20-28%, worsening (not improving) with a longer capture window -- ruling out boundary skew. A quiet-desktop rerun recovered to ~94% agreement. This is consistent with the product's ETW consumer (PresentMonTraceConsumer's system-wide mAllPresents circular buffer, PresentData/PresentMonTraceConsumer.cpp) losing in-progress presents under system-wide present traffic and never surfacing that loss (mLostPresentEvents is decoded by the vendored library but never read by PresentMonTraceBackend::Drain()). This is a genuine product diagnostics-accuracy question independent of this PR's oracle design and needs its own investigation (how much loss under realistic load, whether presentCount/modeFlipCount should be marked partial/unreliable when it occurs, whether PID filtering can move earlier in the consumer pipeline, whether this deserves its own release scenario rather than a tolerance in the independent crosscheck). Tracked separately, not addressed here.

Test plan

  • cargo test in tools/exo-verify (present-crosscheck: 11 tests covering summary parsing/transition counting, stable-window and mixed-window judging, unmatched mode, no-growth, and that present-count magnitude is not numerically gated)
  • cargo fmt --check, cargo clippy --all-targets -- -D warnings
  • pwsh scripts/verify.ps1 -Full (pre-push hook), passed
  • Real hardware smoke against a packaged 0.10.0 release candidate, elevated, production control endpoint (diagnostics.present-crosscheck only): quiet desktop PASS (product present delta 2058 vs. PresentMon 2189, ~94%; product mode-flip delta 783 vs. PresentMon transitions 908, ~86%; mode set {composed, independentFlip}; product's reported mode composed observed externally). Busy-desktop runs (two, 8s and 24s) diverged further from PresentMon's counts, motivating the "Known follow-up" above and confirming counts must not be gated numerically.
  • Branch rebased onto origin/next (eeab35da, fix(release): record what holds the old tree when an update swap fails #423) so the diff is scoped to tools/exo-verify/src/scenarios/present.rs only.

@Exoridus
Exoridus marked this pull request as ready for review September 24, 2026 08:25
Exoridus added 4 commits September 24, 2026 16:50
…crosscheck window

PresentMon legitimately observes DWM interleaving composed and
independent-flip presents for the same process across an 8-second
capture (measured: 1127 vs. 157 presents, scattered throughout) --
that is real desktop composition behavior, not measurement noise.

The product's present.mode is the mode of the single most recently
decoded present at snapshot time, not an aggregate. Judge the
crosscheck by the same semantics: PresentMon must have observed the
product's reported mode at all, and its chronologically last present
for that process must agree with it. Mixing alone no longer produces
INFRA_ERROR; an unclassified mode still does.

Bumps diagnostics.present-crosscheck to revision 2.
…mode-flip parity, not per-event QPC identity
…mode-flip parity, not per-event QPC identity
…esent-crosscheck, keep counts as evidence only
@Exoridus
Exoridus force-pushed the fix/present-crosscheck-mixed-mode-oracle branch from 23fd446 to d5ea27a Compare September 24, 2026 15:06
@Exoridus
Exoridus merged commit 83f2914 into next Sep 24, 2026
13 of 15 checks passed
@Exoridus
Exoridus deleted the fix/present-crosscheck-mixed-mode-oracle branch September 24, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant