Skip to content

Harden the ratchet, tests and CI ahead of the monorepo migration (step P) - #48

Merged
tap merged 4 commits into
mainfrom
claude/step-p-harness-hardening
Sep 27, 2026
Merged

tap merged 4 commits into
mainfrom
claude/step-p-harness-hardening

Conversation

@tap

@tap tap commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

What this changes

This PR is SampleRateTap's half of step P, the pre-work for the monorepo migration. The plan is docs/MONOREPO_PLAN.md on claude/sample-rate-expansion-strategies-ezqzu6, and the companion PR is tap/RatioTap#18. The PR repairs breakage that nothing was catching. It also makes the ratchet, the tests, CI and the notebooks produce the evidence the migration's gates will compare.

P.1: repairs (98cbe9b)

  • Pico 2 examples build again. Both examples were ported to the current snake_case API and given the dsptap include path. Both firmwares build against Pico SDK 2.1.1.
  • The book's trace tool builds again. scripts/book_figures_trace.cpp compiles against both the current tree and the pre-fix commit 045de5d. book_figures.py passes the dsptap include path.
  • .git-blame-ignore-revs is fixed. Its only entry named a commit that exists in no history. It now lists b84020e and e2f5a48.
  • Cross-validation figures updated. README and book now quote RatioTap's current floors (−98/−90 dB).

P.2: harness and CI (865e32d, 64509e1)

  • scripts/icount.py:
    • --exact requires identical counts.
    • --json-out and --compare-json support same-job A/B runs.
    • A recorded workload with no binary is now a failure.
    • A missing baselines file is now fatal.
    • Each workload's checksum is printed and recorded.
    • Hexagon workloads run from a fixed path, with a fixed argv[0] and an empty environment. qemu-hexagon copies these onto the guest stack, and static musl's startup walks them, so the old counts included them.
    • The Hexagon baselines are re-recorded (64509e1). The first isolated run measured every scenario 3,656–3,724 instructions lower (−0.0008% to −0.0020%). There is a ledger entry in docs/PERFORMANCE.md, and the README table was regenerated.
  • Test names and labels. Tests carry an async. ctest prefix and an async label, including the bare-metal entry.
  • New OutputHash suite. It prints FNV-1a hashes of the datapath output. They are compared within one job, never pinned.
  • CI:
    • Push runs only on main; branches run through pull_request and dispatch. This removes the double runs on PR branches.
    • Cancellation spares main and the migration PR, but still applies to other PRs, including this one.
    • permissions: contents: read.
    • The QEMU and ratchet jobs are pinned to ubuntu-24.04, with the image OS added to the plugin-qemu cache key and image and toolchain versions logged.
    • --no-tests=error on every ctest call.
    • The QEMU jobs upload their full test logs.
  • New Pico 2 build job.
  • ci-arm64: the TSan ring stress filtered on SpscRing, which matched nothing, and now filters on spsc_ring.

P.3: notebooks (3e39365)

  • requirements.in / requirements.lock. The notebook environment is pinned with hashes, and the same file is in RatioTap. The README points to it.
  • The ctypes bridges now rebuild the C ABI library every run. The rebuild is incremental and quiet. Before, a stale library could be loaded without anyone noticing.
  • notebooks/figure_digest.py prints a data digest for each figure. A changed curve then shows up as changed text. The rbj notebook's wall-clock timing cell is tagged nondeterministic.
  • All four notebooks re-executed.
    • The comparison and rbj numbers reproduce exactly.
    • Two notebooks' committed outputs were stale. They predate the compensated prototype design (2026-07-04). The demo SNR is now 125.9 dB (it read 126.4). The block-size study's wideband floors are now 61.5 / 58.1 / 54.0 dB (they read 60.9 / 58.2 / 53.1). The prose quoting these numbers is updated: README, book, COMPARISON.md and the notebook summaries.

Why

The migration gates key tests by unique names and compare exact icount, output hashes, full on-target logs and notebook outputs. All of that has to exist before the snapshot. Several checks were also passing while testing nothing: the arm64 TSan filter, the Pico builds, the trace tool and the blame-ignore file. The snapshot must not record those as green.

Verification

  • Host builds:
    • GCC with -Werror: 77/77 tests pass, all named async.* and labelled async.
    • Clang with -Werror: clean.
    • clang-tidy-18: clean on the new test.
  • Hashes are stable across runs and identical between GCC and Clang. On M33 under QEMU the suite takes about 60 s. The Q15/Q31 hashes match x86-64 exactly; the float hashes differ, as expected. The M33 timeout goes from 30 to 40 minutes.
  • icount.py on M33 under real QEMU with the plugin:
    • All 7 workloads match exactly.
    • The A/B mode passes.
    • A missing workload fails, and so does a missing baselines file.
  • CI on the current head (3e39365): every job is green (17/17).
    • M33 correctness took 28 min, within the new 40-min timeout.
    • Hexagon correctness took 26 min.
    • The ratchet passes on all three targets against the re-recorded baselines.
  • Notebooks were executed with pip install --require-hashes -r requirements.lock on Python 3.11.15. The digest lines are identical across two re-executions.
  • Not verified: ci-arm64, because it is dispatch- and schedule-only.

Notes for the reviewer

  • Contract change (test naming only): ctest names gain an async. prefix. -R and -E substring filters are unaffected.
  • Notebooks re-executed. They are committed executed, and the changed numbers are listed above.
  • The concurrency expression names the migration branch explicitly, so the migration PR never cancels an in-flight run.

🤖 Generated with Claude Code

https://claude.ai/code/session_015VR1VC4SDGxHZQQsQvPBaA

- Port both Pico 2 examples to the current snake_case API and add the
  dsptap include path they need since the DspTap substrate move; both
  firmwares build again against Pico SDK 2.1.1.
- Make the book's trace tool compile against both the current tree and
  the pre-fix commit 045de5d (whose API predates the renames), and pass
  the dsptap include path from book_figures.py.
- Point .git-blame-ignore-revs at the real reformat commit (b84020e) and
  add the namespace-rename reflow (e2f5a48); the old entry named a commit
  that exists in no history.
- Quote RatioTap's current cross-validation floors (-98/-90 dB).

Step P.1 of docs/MONOREPO_PLAN.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VR1VC4SDGxHZQQsQvPBaA
- icount.py: --exact (integer equality), --json-out/--compare-json for
  same-job A/B measurement, a failure when a recorded workload has no
  binary, a fatal error on a missing baselines file, and the workload's
  checksum printed and recorded. Hexagon workloads now run from one fixed
  path with a fixed argv[0] and an empty environment: qemu-hexagon copies
  those onto the guest stack and static musl's startup walks them, so the
  binary path and the job's environment were leaking into the count.
  qemu-hexagon is resolved to an absolute path first, since the cleared
  environment has no PATH. Hexagon baselines are re-recorded separately.
- Tests carry an "async." ctest prefix and an async label (the bare-metal
  entry too), so names stay unique once they share a tree with RatioTap's.
- New OutputHash suite: FNV-1a hashes of the datapath's output for every
  format, preset and two rate offsets, printed for same-job comparison,
  never pinned (float differs across FMA/libm; Q15/Q31 measured identical
  on x86-64 and Cortex-M33). About 60 s on M33, so the M33 leg's timeout
  rises to 40 min.
- CI: push runs only on main plus pull_request and dispatch (a PR push no
  longer runs twice); cancellation spares main and the migration PR;
  contents: read permissions; ubuntu-24.04 pinned for QEMU and ratchet
  jobs, with the image OS in the plugin-qemu cache key and image/toolchain
  versions logged; --no-tests=error on every ctest; QEMU legs keep and
  upload full test logs.
- New Pico 2 firmware build job, so the examples cannot silently rot
  again.
- ci-arm64: the TSan ring stress filtered on SpscRing, which has matched
  nothing since the snake_case rename; it now selects spsc_ring and fails
  on an empty selection. style.yml gets the same triggers, permissions and
  a SHA-pinned checkout.

Step P.2 of the monorepo migration plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VR1VC4SDGxHZQQsQvPBaA
The previous commit runs Hexagon workloads from a fixed path with a
fixed argv[0] and an empty environment. The first isolated CI run (PR
#48) measured every scenario 3,656 to 3,724 instructions lower
(-0.0008% to -0.0020%): the path and environment strings static musl's
startup used to walk. Inside tolerance, but re-recorded so exact
comparisons start from the new harness; ledger entry in
docs/PERFORMANCE.md, README table regenerated. M33/M55 are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VR1VC4SDGxHZQQsQvPBaA
- requirements.in / requirements.lock: the notebook environment pinned
  with hashes (numpy 2.4.6, scipy 1.17.1, matplotlib 3.11.2, samplerate
  0.2.4 and soxr 1.1.0 as the committed comparison outputs record, plus
  jupyter/nbconvert); identical in RatioTap. README points at it.
- The ctypes bridges now rebuild the C ABI library incrementally on every
  run (quietly; the log prints only on failure) instead of only when it is
  missing, so a library left over from an older checkout can no longer be
  measured silently.
- notebooks/figure_digest.py wraps plt.show() to print a digest of each
  figure's plotted data, quantized to 9 significant digits of each
  array's peak, so a changed curve shows up as changed text; stable
  across re-executions. The rbj notebook's wall-clock timing cell is
  tagged "nondeterministic".
- All four notebooks re-executed in the pinned environment. The
  comparison and rbj numbers reproduce exactly. Two notebooks' committed
  outputs predated the compensated prototype design (2026-07-04) and had
  drifted: the demo's SNR is 125.9 dB (was 126.4) and the block-size
  study's wideband floors are 61.5/58.1/54.0 dB (were 60.9/58.2/53.1),
  its final-ppm snapshots moved with them. The prose that quotes them
  (notebook summaries, README, book, COMPARISON.md) now states the
  re-executed values.

Step P.3 of the monorepo migration plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VR1VC4SDGxHZQQsQvPBaA
@tap
tap marked this pull request as ready for review September 27, 2026 20:40
@tap
tap merged commit 5e2057f into main Sep 27, 2026
17 checks passed
tap pushed a commit that referenced this pull request Sep 27, 2026
The previous commit runs Hexagon workloads from a fixed path with a
fixed argv[0] and an empty environment. The first isolated CI run (PR
#48) measured every scenario 3,656 to 3,724 instructions lower
(-0.0008% to -0.0020%): the path and environment strings static musl's
startup used to walk. Inside tolerance, but re-recorded so exact
comparisons start from the new harness; ledger entry in
docs/PERFORMANCE.md, README table regenerated. M33/M55 are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VR1VC4SDGxHZQQsQvPBaA
@tap
tap deleted the claude/step-p-harness-hardening branch September 27, 2026 20:40
tap pushed a commit that referenced this pull request Sep 28, 2026
Moves submodules/sampleratetap 2b4dff1 -> 5e2057f, SampleRateTap's
main after its step-P PR (#48). That range changes no header under
include/ and keeps the same DspTap pin (0eb09fa), so the tree the
cross-validation compiles against is unchanged: 82/82 tests pass and
the four cross-validation lines are byte-identical to before.

The monorepo migration's step 0 snapshots against this pin, so the
cross-validation lines it records come from the same async tree the
import compiles (step P.4 of the plan).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VR1VC4SDGxHZQQsQvPBaA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants