Skip to content

feat: validate Stax profiling on Apple M1 - #46

Draft
fcostaoliveira wants to merge 39 commits into
mainfrom
feat/apple-m1-stax-pilot
Draft

fcostaoliveira wants to merge 39 commits into
mainfrom
feat/apple-m1-stax-pilot

Conversation

@fcostaoliveira

@fcostaoliveira fcostaoliveira commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

M1 Stax pilot

Companion to merged plan #38 and Linux pilot #45. This branch pins Stax on Apple M1, adds a private paired Redis pilot runner, and records native and rootless controls. The memtier mode matches the Linux Redis 8.6 workload shape: 10,000 prefilled 64-byte keys, five-second warmup, 30-second 1:1 SET/GET, 2 threads × 10 clients, pipeline 4. The runner now reports each adjacent baseline/Stax pair and its order as well as pooled medians.

The pinned Stax macOS runtime requests fixed cycles/instructions only. Its unused configurable-event helper rejects the second event on M1 when allocation is forced too late. A rootless KPEP probe maps the accepted L1D and branch events to sample slots 2 and 5, with eight config words for ten total counters. An opt-in experimental source patch fixes configuration and wires it into the server behind STAX_EXPERIMENTAL_APPLE_PMU=1; it builds and passes rootless configuration and negative controls. It has not measured programmable counters in a privileged capture.

The pinned helper exposes a world-writable socket without a peer check and can leave global kperf/kdebug state configured on setup error. The runbook uses a teardown guard and a foreground helper socket under a private 0700 directory. The runner checks socket ownership and reserves 1.5 GiB free disk before rounds and saves. This branch does not install a LaunchDaemon.

A real M1 Metal clock check placed 12 completed command-buffer intervals in the host Mach clock domain. A Swift/Rust bridge then sent 299 checksum-verified real M1 GPU spans through Stax using an unprivileged fake helper with zero kperf records. Stax saved a v3 archive and reproduced its views after reopening. The pinned compare --json mislabeled those 299 synthetic markers as CPU PET samples. The proposed small CLI patch in examples/stax-compare-cpu-samples.patch reports zero CPU PET samples and 299 target spans for the same archive; all 39 Stax CLI tests pass. The installed baseline CLI is unchanged, and the pilot checks the archived run summary for real CPU PET counts.

A separate fake helper required the exact M1 configurable-counter setup and emitted invented PET/PMC records at sample slots 2 and 5. The patched server and parser saved 78 such CPU samples alongside 299 real GPU spans. PR #47 imported all 78 with the expected invented values: cycles 100, instructions 200, L1D misses 13, branch mispredicts 17. The original compare counted 377 combined markers; the proposed patch counted 78 CPU samples and 299 target spans. A new parser regression test keeps the gap between configurable counter slots. These fixture values are not hardware measurements. A separate rootless fixture alternated synthetic kdebug CPU IDs 0 and 4; the pinned Stax PET parser and v3 archive discard that ID. It saved 347 invented CPU samples and 299 linked Metal spans, but the archive cannot support per-core PMU validation or P/E-specific interpretation. The plan now requires preserving core identity upstream or independently verified core-scoped captures before such claims. A source audit also found native dropped-record counts are logged in the helper/client but absent from Stax’s RPC run summary and v3 archive, so PR #47 correctly marks native loss accounting unavailable. Positive PET counts alone do not establish complete delivery.

The now-merged integration PR #47 was checked on this M1 at c3ba2de: its importer stored that Metal archive in SQLite with 299 target spans, zero CPU samples, and a zero CPU-hotspot denominator. A separate rootless end-to-end control used #47's capture adapter, real Stax CLI/server and the real M1 GPU app; it saved and imported another 299-span trace, with clean target exit. The test process bypassed the helper ownership preflight only for the zero-kperf fake helper. This does not validate privileged CPU/PMU capture.

Against #47 at b6d2897, a further rootless control used the app's actual submitting thread ID in invented CPU samples. Stax linked all 299 real M1 Metal spans to that synthetic stack within its 50 ms window; the largest link distance was 22.653 ms. The archive and SQLite importer retained 341 invented CPU samples, 299 linked spans and their origin thread ID. The public import/query path succeeded. This verifies the software join and persistence path, not real CPU stacks or hardware counters. The helper remains unprivileged, with no root daemon running. Two negative controls, with no CPU samples or invented samples on the wrong thread, each retained 299 spans and linked zero. The real no-helper preflight refused a new capture before launching the target or creating output, while archive import still worked. A six-run synthetic cadence control held the requested Stax frequency at 99 Hz: 20 ms fixture periods linked 299/298 of 299 Metal spans, 50 ms linked 287/286, and 100 ms linked 147/147. The unlinked 50/100 ms spans exceeded Stax’s 50 ms origin window, showing why observed sample cadence and link diagnostics matter for attribution.

Six additional unprofiled Redis 8.6/memtier repeats had zero GET misses. Throughput was about 537k–539k operations/s for four rounds, then 581k and 577k; the largest departure from the six-round median was 7.80%. The cause is unknown. The working live-pilot budget, set before a profiled result, is at most 10% median throughput loss and 20% median p99 latency increase; paired order effects can require more rounds.

A four-round rootless Redis/memtier pilot with the synthetic helper exposed two runner issues: the long-running stax-server --version probe timed out, and a server kept alive across baseline rounds retained large archive state. Both are fixed. Failure-path hardening now saves partial command output on timeout, bounds the Xcode version probe, keeps the artifact root private, and records helper preflight errors in metadata. A real no-helper M1 run exited before any Redis/Stax round, retained only private metadata with the reason, and printed a concise error. The plan also calls for a separate low-pipeline latency control because the Linux VM pilot found a large pipeline-one PING p99 increase despite small throughput loss. The corrected pilot used a new server/socket for each profiled round and none in baseline rounds, preserved all five checked views across two archive round trips, compared the archives offline, and had zero GET misses. Host throughput and p99 shifted in the final unprofiled round; synthetic samples cannot establish Stax overhead.

After #47 merged, a shared SQLite control stored a real M1 native fixed-counter Redis profile (23,844 stack samples) alongside a separate rootless Stax Metal archive (341 invented CPU samples, 299 real GPU spans). Apple and Stax CLI queries returned their own runs and rejected cross-collector run IDs. The workloads and capture times differ, so this checks storage coexistence rather than comparative performance.

A fresh post-merge native M1 event probe also passed with 556 ten-position vectors and a verified checksum. Its public SQLite import/query reported 553 alu_work leaf samples and retained native E/P core labels. TMA remains unavailable because event positions and formulas are uncalibrated. To close Stax's separate core-identity gap, this branch now includes a proposed pinned-Stax patch that carries the raw kdebug CPU ID into optional v3 saved-sample fields; old archives reopen with unknown identity. A rootless fake-daemon run archived 174 invented CPU samples for each ID 0 and 4 plus 299 real Metal spans with unknown CPU. The topdown importer validates and preserves the optional ID on CPU PET rows only; public import and sample queries retained all 348 synthetic IDs. Synthetic target-span markers expose neither CPU identity nor PMU fields, even if those keys appear in their saved sample. Newly stored normalized results use schema version 2 for this row shape; older stored results remain queryable. The patched server also reopened an older unmodified 341-CPU-sample archive. These controls prove software preservation only, not physical core residency or PMU scope.

A further source review found that XNU's kperf KPC thread samples are cumulative snapshots. Pinned Stax forwards them as if they were interval deltas. A rootless cumulative fixture produced 348 invented samples: stock pipeline values summed to 6,072,600 cycles while a proposed per-thread delta patch yielded 34,700 after withholding the first baseline. Both preserved 299 real Metal spans. The patch also discards all-zero vectors emitted when a thread counter buffer is unavailable, then rebases; its eight parser library tests pass. The public topdown importer/query retained the patched archive's trace_cpu_id and marked PMU semantics unverified. The v3 archive has no conversion provenance; even a correctly differenced thread counter cannot be assigned wholesale to one trace CPU across migrations. Conversion provenance must live with the captured run: a separate open/save control proved Stax replaced a source archive's producer version with the saving server's version while preserving the old cumulative PMU values. See docs/stax-m1-kpc-accumulation-control.json, docs/stax-m1-save-provenance-control.json, and examples/stax-m1-kpc-delta.patch.

The branch now includes current main through PR #54. The independent M6 Metal control retained 299 real target spans and zero CPU samples. Intel Linux now has live Stax capture, replay and topdown queries, while its shared-host overhead budget remains unresolved; the separate Linux VM's longer latency follow-up still failed its declared low-pipeline p99 budget. These controls do not alter the M1 PMU acceptance status.

Status: draft. Privileged capture and hardware PMU validation remain unverified because no root helper is in use on this M1. Native M1 counters and fixed-stack captures already succeeded. No Apple TMA funnel or GPU hardware-counter claim is made.

Verification

  • Full repository suite after merging current main through PR Validate Stax on Intel Linux after enabling perf permissions #54: 2,153 passed, 107 skipped; Ruff clean. All six CI checks passed on the latest head.
  • Stax release build with full Xcode, 39 CLI tests, 7 target tests and 5 macOS parser tests passed. A separate upstream illustrative parser doc test does not compile.
  • Redis 8.2.2 and 8.6.0 have frame-pointer builds and matching dSYMs; memtier 2.5.2 reported consistent count, duration and rate.
  • Real M1 Metal bridge, fake helper, Stax target ingestion, archive round trip, proposed compare fix, and feat: integrate optional Stax samples, waits and target spans #47 SQLite import/capture controls passed. The controls use no CPU or GPU hardware counter data.

No Python project dependency was added. See docs/stax-m1-pilot.md for exact revisions, commands, hashes, and limitations.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant