gputrace: capture gates, CUDA/Metal parity toolchain, and analysis consolidation - #2

Draft

tmc wants to merge 537 commits into

mainfrom

This pull request is big! We're only showing the most recent 250 commits

Commits on Aug 13, 2026

perfettoviewer: serve local trace UI
tmc
committed
cbad10e
timeline: export native Perfetto traces
tmc
committed
c0dafab
perfetto: bound native trace output
tmc
committed
a5bf7bf
environment: define comparison projection
tmc
committed
859dbc3
perfetto: retain execution boundaries under budget
tmc
committed
a1bce46
timeline: keep APS aggregates off counter tracks
tmc
committed
95d4dc3
timeline: report MLX semantic coverage
tmc
committed
2ff8f47
mlxsemantic: distinguish receipts from sidecars
tmc
committed
1763bae
perfetto: stream preflighted packets
tmc
committed
27ad29e
timeline: export stable PerfettoSQL views
tmc
committed
2f3cd49
diff: fail closed on environment evidence
tmc
committed
0bdb615
timeline: focus Perfetto viewer precisely
tmc
committed
d1c8cd2
perfetto: annotate every timed event
tmc
committed
9ce57ff
perfetto: complete evidence manifest
tmc
committed
51cb6d1
perfetto: report loss by evidence class
tmc
committed
c577a92
skill: document native timeline workflow
tmc
committed
8194122
timeline: consolidate Perfetto workflow
tmc
committed
5d808e3
perfettoviewer: pin local UI revisions
tmc
committed
f1591cb
timeline: report unavailable manifest evidence
tmc
committed
8608792
timeline: close viewer listener on cancellation
tmc
committed
6baf1b3
perfetto: order track descriptors by hierarchy
tmc
committed
1a00860
timeline: retain untimed MLX semantic nodes
tmc
committed
71857a9
timeline: break out dispatch detail by encoder
tmc
committed
fdc2181
timeline: omit untimed dispatch detail tracks
tmc
committed
05ada2e
trace: attribute capture dispatches to pipelines
tmc
committed
168bf3b
timeline: separate untimed capture launches
tmc
committed
085e20a
skill: explain capture-only Perfetto evidence
tmc
committed
720eb48
timeline: export all pipeline compiler facts
tmc
committed
11be423
perfettosql: normalize recorded dispatches
tmc
committed
1882682
docs: describe Perfetto compiler facts
tmc
committed
05cac11
docs: specify shader source attribution
tmc
committed
b5333d7
perfettosql: expose dispatch evidence
tmc
committed
6ae06c2
docs: describe canonical dispatch evidence
tmc
committed
77b370d
perfettosql: normalize dispatch identifier types
tmc
committed
dd0669e
timeline: retain measured dispatch geometry
tmc
committed
a00c64a
timeline: retain dispatch capture structure
tmc
committed
2250671
docs: explain dispatch capture membership
tmc
committed
fafc2f0
perfettosql: report real dispatch sample coverage
tmc
committed
375cda0
docs: qualify dispatch sample attribution
tmc
committed
9347cc6
capture: preserve target standard input
tmc
committed
3c98514
hostcorrelation: refuse unproven clock joins
tmc
committed
9d35ad7
perfettosql: separate dispatch and function work
tmc
committed
85bf559
docs: describe function-level Perfetto view
tmc
committed
d8c3ea6
hostcorrelation: project bounded host events
tmc
committed
a9ac076
perfettosql: expose encoder aggregate evidence
tmc
committed
0a8ffbc
docs: describe encoder Perfetto evidence
tmc
committed
b639a98
timeline: attach verified host correlations
tmc
committed
70f860d
perfettosql: expose command buffer evidence
tmc
committed
33fc8e3
timeline: order encoder tracks chronologically
tmc
committed
15132ff
docs: describe command buffer evidence
tmc
committed
d1f12ac
hostcorrelation: derive clock bridge from samples
tmc
committed
5ad3724
perfettosql: expose capture manifest evidence
tmc
committed
6c58b64
docs: describe manifest SQL views
tmc
committed
12f5f28
timeline: expose raw profiler evidence
tmc
committed
9ce9d44
docs: describe raw profiler evidence
tmc
committed
e3a2c45
capture: retain live GPU command timing
tmc
committed
e3f82e9
perfettosql: retain function provenance
tmc
committed
cbdf049
docs: explain function evidence provenance
tmc
committed
d408fdd
capture: observe all command buffer constructors
tmc
committed
a82f04e
perfettosql: expose verified timing receipts
tmc
committed
6b28680
docs: describe verified timing views
tmc
committed
9cb9863
hostcorrelation: bind measured host intervals
tmc
committed
19dfd4e
perfettosql: retain MLX semantic evidence
tmc
committed
ff518c6
docs: describe semantic evidence views
tmc
committed
53fa825
perfetto: mark live timing as measured
tmc
committed
e3071e6
perfettosql: retain unattributed counter evidence
tmc
committed
1166ad5
docs: describe untimed Perfetto evidence
tmc
committed
40cfd37
metallib: decode embedded source archives
tmc
committed
319980f
metallib: inventory debug source records
tmc
committed
65fc70a
metallib: bind metadata to declared sections
tmc
committed
36c0724
metallib: bind function debug metadata
tmc
committed
10fc364
perfettosql: expose complete event arguments
tmc
committed
dea6607
docs: describe lossless event argument views
tmc
committed
876c94f
shader: use metallib source declarations
tmc
committed
c6736c7
shader: stop estimating source line cost
tmc
committed
daa9119
perfettosql: expose timing summary evidence
tmc
committed
6f3d5b4
docs: distinguish native timing summaries
tmc
committed
fa33724
shader: suppress advice without dispatch shape
tmc
committed
b089994
perfettosql: separate source and projected counts
tmc
committed
9f205e3
docs: explain cross-domain evidence counts
tmc
committed
b80d897
perfettosql: retain raw record identity
tmc
committed
fb40c29
docs: explain track event argument identity
tmc
committed
4256cc8
docs: define Perfetto release ownership
tmc
committed
ec5d8c1
xcodebindings: report source cost readiness
tmc
committed
0aaa126
docs: record instruction cost boundary
tmc
committed
eb057c8
perfettosql: expose wall clock conversion
tmc
committed
90fa3f9
docs: explain wall clock conversion fields
tmc
committed
adbea68
perfettosql: retain pipeline identity
tmc
committed
c76f9d2
docs: define pipeline identity scope
tmc
committed
8968fcb
perfettosql: retain continuous time evidence
tmc
committed
8d92344
docs: bound continuous time evidence
tmc
committed
6c408fa
perfettosql: retain replay performance state
tmc
committed
2b64434
docs: bound performance state evidence
tmc
committed
e6d24bf
perfettosql: retain replay restore intervals
tmc
committed
cd23894
docs: define replay restore interval scope
tmc
committed
ef5b95e
perfettosql: retain Metal device identity
tmc
committed
e6fc8d5
docs: define stream device identity evidence
tmc
committed
ce7a1b6
perfettosql: retain stream archive metadata
tmc
committed
0193f1e
docs: define stream archive metadata scope
tmc
committed
5ff3539
perfettosql: report stream table integrity
tmc
committed
57331c0
docs: define stream table integrity evidence
tmc
committed
a3e9723
perfettosql: inventory stream archive families
tmc
committed
af0aa1a
docs: define archive family inventory semantics
tmc
committed
d10d33a
perfettosql: reconcile archive family blobs
tmc
committed
a0d19dc
docs: define decoded archive family counts
tmc
committed
8bfff05
tools: validate untimed Perfetto dispatches
tmc
committed
c6bee5b
perfettosql: expose counter decode coverage
tmc
committed
59823c0
docs: define counter decode coverage
tmc
committed
1900d35
perfettosql: inventory APSData dictionaries
tmc
committed
26d9e92
docs: define APSData dictionary inventory
tmc
committed
ee8b287
perfettosql: identify raw profiler artifacts
tmc
committed
f9ed279
docs: define raw profiler artifact identity
tmc
committed
1eb30a3
perfettosql: expose raw timeline headers
tmc
committed
734223f
docs: define raw timeline header evidence
tmc
committed
960cfdf
capture: retain timing without trace commands
tmc
committed
660398a
perfetto: expose live kernel timing
tmc
committed
038a751
perfetto: decode fixed GPRWCNTR fields
tmc
committed
9fefd02
perfetto: retain raw counter payloads
tmc
committed
f79883d

Commits on Aug 14, 2026

perfetto: expose counter batch identity
tmc
committed
8c79a04
perfetto: retain counter pass catalog
tmc
committed
e3af023
timeline: use exact profiler sample carriers
tmc
committed
48ffba3
perfetto: retain counter trace identities
tmc
committed
536ca1c
perfetto: preserve unsigned counter trace IDs
tmc
committed
6eaec7f
perfetto: retain fixed streamData tables
tmc
committed
98198f7
perfetto: distinguish malformed streamData tables
tmc
committed
817d644
perfetto: retain streamData strings
tmc
committed
35adb0a
perfetto: retain pipeline compiler diagnostics
tmc
committed
a2266ac
perfetto: preserve pipeline statistic presence
tmc
committed
0651fd9
capture: finalize traces when Go exits
tmc
committed
3393cfc
perfetto: structure pipeline compiler remarks
tmc
committed
b1bf3f7
perfetto: retain compiler remark arguments
tmc
committed
659fee8
perfetto: retain counter encoder aggregates
tmc
committed
741e9c8
perfetto: retain attributed counter samples
tmc
committed
fa4d9c7
perfetto: identify APSData blobs
tmc
committed
866467a
perfetto: inventory streamData archives
tmc
committed
45e38bd
perfetto: retain archive value descriptors
tmc
committed
ebf1513
perfetto: project nested archive values
tmc
committed
3c2dd54
perfetto: expose shader address mappings
tmc
committed
e0e3643
perfetto: relate mappings to encoders
tmc
committed
9a254e8
perfetto: expose recorded profiler configuration
tmc
committed
e78871d
perfetto: expose recorded profiler setup
tmc
committed
35e0356
perfetto: inventory profiler carriers
tmc
committed
ac413af
perfetto: expose all root scalar values
tmc
committed
d449cea
perfetto: audit root scalar equality
tmc
committed
d075d3e
perfetto: audit profiler carrier sequences
tmc
committed
f398478
perfetto: audit shader binary content
tmc
committed
e634cba
perfetto: audit embedded payload content
tmc
committed
c27e839
perfetto: report native-label and sidecar name conflicts

tmc
and
claude
committed
2f1ec51
perfetto: add fixture qualification and UI smoke gates

tmc
and
claude
committed
faa828f
perfetto: freeze the v1 contract surface

tmc
and
claude
committed
26673e2
perfetto: add pipeline-first dispatch view
tmc
committed
ffe9cdb
perfetto: qualify pipeline-first rendering
tmc
committed
0c769c7

Commits on Aug 23, 2026

build: gate darwin-only packages behind //go:build darwin
tmc
committed
5ac48e2
test: skip darwin-behavior tests off darwin
tmc
committed
4846541
go.mod: depend on github.com/tmc/lib/nvidia
tmc
committed
2d79f44
cmd/gputrace: add nvidia command reporting GPU status via NVML
tmc
committed
4918437
internal/cuptitrace: convert CUPTI activity captures to Perfetto traces
tmc
committed
8bf93dc
cmd/gputrace: add cupti command for Linux kernel tracing
tmc
committed
f4bc8b1
docs: document the Linux/NVIDIA surface
tmc
committed
5a4c8e8
gpuevent: extract vendor-neutral activity model
tmc
committed
0a74e06
cmd/gputrace: add devices command with backend capability registry
tmc
committed
167e15c
gpuevent: derive per-kernel metrics and optimization findings
tmc
committed
1486d82
cmd/gputrace: add analyze command reporting optimization findings
tmc
committed
fcce09e
optimize: add reproducible workload run harness
tmc
committed
4aeaf09
optimize: compare runs with noise-aware verdicts
tmc
committed
6bc6f41
optimize: add findings-to-actions playbook
tmc
committed
36a4609
cmd/gputrace: add --suggest to analyze for playbook actions
tmc
committed
aac2e18
docs: document the agentic optimization loop and NVIDIA skill reference
tmc
committed
65605c0
cmd/gputrace: native Linux CUPTI capture via preloaded shim
tmc
committed
5b40707
cmd/gputrace: run the NVML sampler during capture --samples
tmc
committed
13debdc
docs: point agents at native capture and bundle inputs
tmc
committed
82e9238
optimize: compare kernel activity between captures
tmc
committed
74ea4ee

Commits on Aug 26, 2026

capture: fix nvml_sampler invocation; add extended sampler source
tmc
committed
8aec656
cupticapture: concurrent-first kernel activity and latency timestamps
tmc
committed
934bedb
gpuevent: decode API, clock-sync, and capture-meta records
tmc
committed
feced95
tools: host nvml_sampler in-tree
tmc
committed
6a26940
cupticapture: record hardware provenance in meta.json
tmc
committed
db5ce95
cmd/gputrace: surface launch overhead in analyze with --api capture
tmc
committed
513f990
cupticapture: per-PID event shards for multi-process targets
tmc
committed
d40d9a8
cupti: pprof export and richer Perfetto projection for CUDA captures
tmc
committed
38a0173
tools: relocate nvml_sampler source to its own package
tmc
committed
6f5e96f
cupticapture: memcpy memory kinds, guarded exit flush, shim cleanup
tmc
committed
f33a2d2
gpuevent: kernel richness, theoretical occupancy, stream tracks, new findings
tmc
committed
151ef87

Commits on Aug 27, 2026

gitignore: exclude local capture artifacts
tmc
committed
ed9dab5
gpuevent: span attribution and per-span decomposition
tmc
committed
55156f9
docs: add CUPTI capture roadmap
tmc
committed
0afd0c8
docs/research: GB10 findings and macOS counter-oracle notes
tmc
committed
f551786
gpuevent: translate unix-clock spans via clock_sync
tmc
committed
fe6a1cc

Commits on Aug 28, 2026

Commits on Aug 29, 2026

Commits on Aug 30, 2026

cupticapture: flush CUPTI periodically so uninterposable targets capture
tmc
committed
d43aa0b
cupticapture: flush with FlushAll, not FlushPeriod
tmc
committed
c232fb1
cmd/gputrace: stream the target's stderr during capture
tmc
committed
682c0d9
gpuevent: stack kernels under the spans that enclose them
tmc
committed
984463c
cudagraphdot: parse MLX CUDA-graph DOT dumps
tmc
committed
e5f1103
perfetto: name the device from the trace, not the writer
tmc
committed
1808edc
cmd/gputrace: export CUDA captures and graph dumps as pprof
tmc
committed
bc89089
cupticapture: query CUPTI's dropped-record count
tmc
committed
6263dc4
gpuevent: measure whether a capture kept every record
tmc
committed
28ff4a4
cmd/gputrace: state what a capture holds before reporting on it
tmc
committed
9537819
gpuevent: keep the last clock_sync, not the first
tmc
committed
260e20f
gpudoctor: tell a staged profiling setting from an absent one
tmc
committed
361053f
cmd/gputrace: profile each escalated kernel in its own ncu run
tmc
committed
c84a18b
cmd/gputrace, internal/gate: add gate command for capture validity
tmc
committed
29c8afa
cmd/gputrace: make absence reporting explicit in analyze, summary, and stats
tmc
committed
33168bb
cmd/gputrace, internal/gate, docs/research: refine gate output and document Metal argmax invariant
tmc
committed
2dc5c76
docs/research, cmd/gputrace: add real mlx-lm decode invariant provenance and profiler blit absence reporting
tmc
committed
3470514
docs/research, testdata: document CUDA workload invariants and add test harness
tmc
committed
14d80dd
testdata/harness: document nvcc build flags and usage modes
tmc
committed
00d0ae8
cmd/gputrace, internal/gate: add two-bundle gate --compare residency mode
tmc
committed
bf36070
gate: label replay-derived stationarity, add units and compare dispatch delta
tmc
committed
abf3b85
gate: nested-range monotonicity check via --ranges
tmc
committed
d52b8f7
capture, gate: host provenance sidecar and cross-session labeling
tmc
committed
e03c31f
cmd/gputrace: label cross-host capture comparisons in diff paths
tmc
committed
8dddb1d
gate: live command-buffer stationarity from the timing sidecar
tmc
committed
7601de6
trace: decode MTLResourceOptions from buffer-creation records
tmc
committed
72c027c
cmd/gputrace, counter: one decoder for Metal kernel display names
tmc
committed
7565150
docs: residency A/B observation, python mlx-lm vs mlx-go on Qwen3
tmc
committed
dfb7d9e
docs: record that unified-memory counters cannot work on GB10
tmc
committed
607022e
gpuevent: stop calling the two completeness checks independent
tmc
committed
e64174a
gpuevent: an empty capture is never complete
tmc
committed
381f8df
cmd/gputrace, gate: gofmt
tmc
committed
e8b5939
gputrace: address mlx-swift audit findings
tmc
committed
61fb831
gate: make block-size auto reachable and report the effective block
tmc
committed
c844d4a

Commits on Aug 31, 2026

gputrace: report storage-mode footprint and residency together
tmc
committed
b48563a
trace: count buffers across the whole capture
tmc
committed
fecdf03
gputrace: say which buffer population each command counts
tmc
committed
40a3953
gate: an overshoot is not a shortfall
tmc
committed
b4b53e7
gpuevent: compare captures by launch geometry, not by symbol
tmc
committed
db147ce
cupti: anchor exported traces to CLOCK_REALTIME so two can be compared
tmc
committed
dedfdd8
profiler, shaders: report shader compilation time
tmc
committed
fca6ee1
profiler: say how much of the counter stream this capture owns
tmc
committed
b543e94
analysis: buffer bindings group by CS ordinal, not by encoder
tmc
committed
044dcac
cupti: share tracks for repeated spans
tmc
committed
66a6185
profiler: name the absent compile records instead of implying a cache miss
tmc
committed
418c4e8
docs: record that Compile Performance measures no compiler phase
tmc
committed
3bef950
cupti: expose pre-span API launches
tmc
committed
2189499
gate: count the timing marks the sidecar could not supply
tmc
committed
2693b61
timeline: keep the -1 compiler sentinel out of the duration columns
tmc
committed
7cb1361
make: report the tests that did not run
tmc
committed
2968d08
docs: name the two perf fixtures and say not to commit them
tmc
committed
88def2c