Conversation
Write native GPU, counter, track, clock, and manifest packets while retaining Chrome JSON as a separate format. Validate strict MLX semantic sidecars and project their hierarchy into the selected clock domain. Add loopback-only --open and --serve viewer modes to the existing timeline command.
Keep finite conversion lossless by default. Under an explicit logical-byte limit, retain whole event groups by stable capture identity hash, preserve descriptors and dependency skeletons, and emit a stock-Perfetto-visible loss receipt.
Attach capture-backed GPU cycles and derived execution cost to encoder details with their ordinal attribution basis. Report the unjoined APS clock explicitly instead of plotting aggregate values as sampled busy-time counter series. Remove the unreachable generator that mixed compiler facts and unverified counter rows.
Return coverage from strict sidecar validation and include used and unused semantic nodes plus matched and unmatched GPU targets in JSON and the native Perfetto manifest. Expand negative tests for every rejected identity and reference shape.
Detect schema-less MLX runtime receipts before strict decoding and explain that trace identity and explicit GPU target links are required. Document why label similarity cannot supply those relationships.
Compute the exact framed output size before writing, then emit one packet at a time instead of buffering the complete trace. Preserve deterministic lossless and budgeted output and test that writes stay packet-sized.
Add --sql-out to the existing timeline command and emit versioned capture, dispatch, pipeline, semantic, counter-series, and unmatched views. Validate the module against trace_processor_shell and reconcile dispatch counts and duration with canonical JSON.
Attach a versioned environment projection to trace data and require exact workload, device/driver, runtime, capture-mode, and timing evidence before presenting controlled deltas. Add an explicit cross-environment override whose JSON remains labeled non-causal.
Add exact kernel selection, an explicit occurrence selector for repeated names, and validated initial time ranges to the existing timeline viewer. Encode navigation with Perfetto nanosecond deep-link parameters and reject ambiguous or invalid selections.
Add clock domain, timing source, and timing quality to native event details, including sidecar semantic events. Copy event arguments during projection so native enrichment does not mutate canonical JSON or Chrome output.
Record input identity availability, packet families, pinned Perfetto revision, source counts, and absent system evidence. Fix PerfettoSQL generic TrackEvent arguments to use debug-prefixed keys and extend native validation to reconcile its dispatch view with gpu_slice.
Extend constrained-export receipts with considered, retained, and dropped items and framed bytes per evidence class, retained descriptor skeletons, and dropped identity bounds. Emit every field in the stock-Perfetto-visible manifest.
Describe native Perfetto export, stable SQL views, exact viewer focus, strict sidecar identity, APS aggregate placement, and fail-closed environment comparison in the repository skill.
Keep native export, viewer launch, semantic attachment, resource policy, loss receipts, and SQL views under the existing timeline command. Document only current flags and test that no parallel top-level command is introduced.
Require self-hosted Perfetto UI directories to provide an entry point and a versioned revision manifest. Expose the selected revision in viewer metadata and status, and refuse revision changes during startup.
Identify the exporter and make capture, replay, counter catalog, decoder, raw artifact, and untimed-dispatch status explicit. Do not infer unavailable provenance from bundle names.
Close the listener unconditionally when the viewer exits. This covers cancellation that races with the Serve goroutine before http.Server.Shutdown can observe an active server.
Emit parent descriptors before children in deterministic sibling order. Reject missing parents and cycles before writing packets.
Emit required untimed declarations for every strict-sidecar node and separate them from timed target links in PerfettoSQL. Report links that cannot project onto the selected clock domain instead of silently omitting them.
Add presentation-only child tracks that expose every kernel under its encoder in lossless native Perfetto exports. Keep gpu_slice as the accounting source and omit duplicate detail tracks from constrained output.
The presentation projection is useful for measured dispatch slices, but capture-only traces can contain thousands of zero-duration label-derived events. Avoid creating one child track per such event and record the omission reason in the manifest.
Walk command-buffer records in capture order and associate each dispatch with the preceding pipeline state. Preserve unnamed archive identities and fail closed when no function name is available.
Render capture-only dispatch records as generic Perfetto instants with pipeline identity and geometry. Keep measured dispatches in gpu_slice, report CS labels as annotations, and omit unproved encoder rows.
Document that untimed capture launches are generic instants and that CS labels do not prove launch or encoder identity.
Use one helper for profiled and capture-backed pipeline statistics so native Perfetto dispatch details expose the complete static compiler record without adding inferred dynamic measurements.
Include capture-only TrackEvent launches in the canonical dispatch view and aggregate complete compiler facts across both native GPU and debug-prefixed argument encodings.
gate --timing <sidecar> scores stationarity from the GT_TIMING_OUT sidecar's per-command-buffer GPU wall clock instead of replay-derived streamData timing, closing the gap where a live mid-run excursion was invisible to replay scoring. Granularity is command buffers, not invariant-matched dispatches; output says so. Fix a naming regression this surfaced: the sidecar interposer labels command buffers gputrace.live.cb.<n>, and that label appears as a CS record in the capture. AnalyzeKernels mistook it for an encoder, so pipeline attribution died and every kernel name became the command buffer label - an invariant that matched 40/40 without the sidecar matched 0/40 with it. Exclude the interposer's own labels from encoder detection.
The 8 bytes at Culul+0x18 hold the MTLResourceOptions value passed at buffer creation, verbatim. Established with a controlled capture (testdata/storage-mode-probe) creating buffers at options 0x0, 0x20, 0x100, 0x120, 0x200 plus a heap buffer: every value round-trips exactly. Previously the field was read by nothing and api_calls printed a hardcoded options:HazardTrackingModeUntracked for every buffer - a label, not a decode. Decode the field into InitCall.ResourceOptions, render it with FormatResourceOptions, and add Trace.BufferStorageModes counting buffer-creation records by storage mode. gputrace gate surfaces the breakdown as a staging observation (storage: buffer storage modes: 2177 shared ...) and distinguishes a bundle with no buffer-creation records from one recording zero buffers of a mode. Findings and provenance in docs/research/METAL_STORAGE_MODE_RECORDS.md.
Audit of the Metal readers found the same class of drift that bit the CUDA side: six mutually incompatible inline fallbacks for unnamed pipelines, split across two different keys (PipelineIndex vs PipelineID), so the same dispatch got a different identity in timing than in timeline and cross-command joins silently missed. shaders dropped unnamed pipeline rows entirely while counting their time in the share denominator, and listed bare library UUIDs as shader names that kernels classifies and separates. Consolidate on counter.DispatchInfo.DisplayName, add PipelineStats.DisplayName for pipeline-keyed sites, and share the library-UUID/archive-name classification with the shaders report. gate's profiler branch now matches invariants against DisplayName so an unnamed dispatch is matchable rather than invisible. timeline warns when a chrome/perfetto export carries no dispatch events at all although the bundle recorded dispatches. Full findings in docs/research/METAL_READER_DRIFT_AUDIT.md.
First direct Metal observation of the CUDA campaign's staging check: both runtimes allocate every buffer shared (2,177 vs 3,180 creation records, zero private/managed/memoryless in either arm), so storage mode does not separate them and the weightLoadStream unified-memory premise survives its first check. The Go arm allocates 46% more buffers for the same decode; structural diffs show a SwiGLU fusion gap, a bfloat16-vs-float32 logsumexp precision divergence, and different GEMM selection. Observation-only: MLX version parity across arms not established, all deltas descriptive.
CUPTI_ACTIVITY_KIND_UNIFIED_MEMORY_COUNTER was on the roadmap behind a --um flag, deferred on a guess that a coherent-memory part might report nothing. Probing it settles the question harder than the guess did: with profiling permitted, every one of the eight counter kinds fails cuptiActivityConfigureUnifiedMemoryCounter with CUPTI_ERROR_UM_PROFILING_NOT_SUPPORTED -- error 28, the system/architecture one, not the per-device error and not a permission error. Configuration fails before any allocation, so there is nothing to enable and no workload that could produce a record. The device attributes explain it: PageableMemoryAccess and PageableMemoryAccessUsesHostPageTables are both set, so the GPU walks the host page tables, managed memory never migrates, and there is no migration or fault event to count. The counters measure a mechanism this part does not have. Also correct the neighbouring dram__ claim, which still said the counters read n/a. The whole domain is missing from --query-metrics; ncu.go already says so and the doc did not.
The type doc said the drop count and the graph invariant "fail independently". They do not, in the case that matters. A launch that left no record anywhere is invisible to both: it leaves every node of its graph even, and a record stranded in a buffer that was never completed was never dropped, so the counter reads zero honestly. That pairing is exactly what let a capture missing 20-47% of its records report itself healthy and diff as a 43.9% kernel-time win. Two blind instruments are not a sighted one. Say so at the type, and name the instrument that does see it: a workload invariant, where an op firing once per token makes a missing launch produce a wrong count rather than a silent one. gputrace gate -k scores that; this type is what you check when you have not got it. Doc only. The user-facing verdicts already carried the caveat.
MeasureCompleteness answered "looks complete" for a capture holding no
records at all: nothing was dropped and there were no graphs to
cross-check, so nothing contradicted it. That is the one case where the
type agreed with every other self-report on the way down. Zero records
is the terminal state of the stranding failure, not a separate error --
the tracer arms, the flush period returns success, the drop counter
reads zero honestly, and nothing is ever handed back. Measured on a GB10
MLX decode: as the activity buffer grew the recorded kernel count fell
monotonically, 119 of 129 argmax launches at 1 MiB, 102 at the 4 MiB
default, and at 16 MiB CUPTI requested two buffers, completed zero, and
wrote a capture with nothing in it.
Completeness now carries the record count and is never complete without
one. Its remedy for the empty case names the flush interval and warns
against the buffer knob by name, because raising it is what produces
this state and the overflow remedy points the opposite way from the
stranding one.
Making Records load-bearing means the zero value no longer reads as
healthy, which is documented on the field and asserted, since two tests
had been building Completeness{} by hand to mean a clean capture.
Struct tags and one-line method bodies in the gate command and its compare result were left misaligned, so gofmt -l flagged two committed files. Whitespace only; no behavior change.
The mlx-swift end-to-end run surfaced four defects. host-receipt rejected the whole host-events file when any interval started before the sidecar's first clock sample; sampling begins at the first Metal device, so early phases like model load are routinely unbindable. Receipt now withholds unbindable intervals (reported on stderr) and binds the rest, refusing only when nothing is bindable. gate --block-size was inert on real decodes: the dynamic floor max(blockSize, len/8) swallowed every explicit value. An explicit size is now honored exactly and zero means auto; the flag help also said 'token block size' though the live path blocks over command-buffer gaps. kernels and shaders pointed users at a nonexistent --profile flag for archive-named functions; the working advice is 'gputrace profile-replay'. The capture --timing-sidecar help now states it requires --run-id.
61fb831 honored an explicit --block-size but left the coercion in Evaluate that turned 0 into 16 before EvaluateStationarity ever ran, so the auto branch was dead code through the CLI and 'auto' silently meant 16. Drop the coercion and the 16 default in DefaultOptions; zero now flows through as auto. Two consequences land deliberately with it. The evaluability threshold was 2*BlockSize, which the coercion pinned at 32 gaps; auto keeps that 32-gap floor (two 16-gap blocks) while an explicit size needs two of its own blocks, never fewer than 16 gaps. And the result's block_size was the requested value, which omitempty would have dropped for every auto run; it is now the effective block actually used. Found by the mlx-swift audit follow-up: on a 64-token decode auto and --block-size 157 now agree (1259 gaps / 8), and auto differs from 16.
Storage modes and residency sets were both already parsed and neither was surfaced anywhere a person reads. Storage modes reached exactly one caller, the gate's staging note. Residency appeared only as individual lines in "api-calls" and "dump", so the pattern that matters -- a set created, never requested, never added to a queue -- had to be spotted by eye in a call listing. They go in one command because they are one finding seen from two directions. An all-shared allocation profile and an uncommitted residency set both mean the process is leaving placement and residency to the driver, and reading either alone makes a default look like a decision. On a real capture: 1272 buffers, 1.1 GB, all shared, one newResidencySet and no requestResidency. InitCall gains Length, so the allocated footprint can be read by a program and not only by a person parsing Info. Trace.InitCalls decodes the init section without requiring a command buffer. ParseAPICallList refuses a capture holding none, which is right for an API-call listing and wrong here: a capture can allocate buffers and create residency sets without this decoder recognising one command buffer, and failing the read reports an absent decode as an absent program. Two limits print with the numbers rather than waiting to be discovered. These counts are not bounded by the decoded-dispatch fraction "api-calls" reports. Buffer and residency records are found by scanning the whole capture for markers, independent of dispatch decoding, so a capture reporting "Decoded API subset: 0 of 39014" can still have a complete picture. Believing otherwise discards a sound number. The narrower real limit is that the scan finds the shapes it knows. Residency-set membership is not decoded, so there is no wired-bytes figure separate from the allocated one. A number labelled wired that was really allocated would be worse than no number. With no set committed, every allocation is under automatic residency and the allocated total is the working upper bound. A capture that decodes nothing is reported as a property of the capture. "0 requestResidency" from an empty decode and from a real observation are different results, and only the second is a finding.
ResidencyReport read InitCalls, which truncated the capture at the first CUUU command-buffer marker. Programs allocate throughout a run, not only during startup, so the report counted a prefix and presented it as a total. On resid-go the first CUUU sits 0.2% into the file and 3066 of 3180 buffer records were dropped, understating 1.2 GB as 300 MB. The undercount was not uniform across captures, which made it worse than a low number: resid-go lost 96% and swift64 lost 37%, manufacturing a 3.8x Go-versus-Swift allocation gap that does not exist. Both runtimes allocate 1.2 GB. Two captures also agreed at 1272 buffers purely because they shared an init-phase size, so the truncation produced false cross-runtime corroboration as readily as false difference. The doc comment and the command output both asserted a whole-capture scan and disclaimed a bound that was not the real one, so the report told the reader it could not have the error it had. InitCalls is renamed ResourceCalls to match what it now reads. Two guards, since one confident number was the failure mode. The test helper writes a CUUU marker after the first record: every residency test passed before because the synthetic captures contained no CUUU at all, asserting a property of a file shape that does not occur. And the report now carries an independent BufferStorageModes count, reporting any divergence with both numbers rather than resolving it, because which scanner is right is exactly what is unknown at that point. Found by 917F2BD0 against real captures.
"residency" and "buffers" both print a buffer count under the same word and mean different things. residency counts buffer-creation records; buffers counts distinct resources and their aliases. On resid-go that is 3180 against 204, a 10x gap with no note on either side, and the byte totals reconcile at 1.2 GB against 1.11 GB because both are right about different populations. The residency cross-check added in fecdf03 does not cover this. It pairs the record decoder with BufferStorageModes, and both read the same Culul records, so they agree by construction on the question they check: they catch a decoder that mis-walks the stream and cannot catch a misreading of what the records represent. The uncorrelated number is the one from "buffers", and it was the one left unlabeled -- a safeguard pointing somewhere other than the failure it was added for, which is the shape of the bug it was added after. Labels both sides and says in the residency output what its cross-check does and does not cover. The column header is now "records" rather than "buffers". No counting changes. Raised by 917F2BD0.
EvaluateCompleteness tested "matched >= want-Slack", which is one-sided
and reads as two-sided. Only "matched == want" was an exact branch, so
every count ABOVE the expectation fell into the slack case and passed
there: MissingCount went negative and the reason claimed a flush
residual the overshoot had never been compared against. A 250-mark
capture against 101 expected reported
PASS completeness ok 250/101 arg_reduce (--149, within flush
residual of 2)
which is a shortfall claim, a negative count, and a slack comparison
that did not happen, on a capture where nothing was missing.
The direction matters because the two mean different things. A
shortfall is lost work and belongs against the slack. An overshoot is
work the invariant did not predict, and completeness -- which asks
whether records are missing -- is satisfied. It still must not borrow
the shortfall's wording.
MissingCount and ExcessCount are now separate and both positive. They
were one signed field, which is what let a negative shortfall exist at
all.
The ratio is reported rather than discarded: the gate held both terms
and printed only their difference, as a negative. It is named
DispatchRatio, an observed dispatch-to-expected-mark ratio, and not an
acceptance rate. The two coincide only if the invariant op fires once
per draft candidate, which is a property of the model and not something
a capture establishes. Naming it for the inference would rebuild the
defect the split removes.
Two causes produce an overshoot and a capture cannot separate them: an
op that legitimately fires more than once per token, as under
speculative decoding, or an invariant symbol matching more than the
caller meant. The reason says overshoot and gives the ratio; it does
not choose between them.
A status-only test cannot see any of this, since the overshoot passed
before and passes now. The new tests assert the arithmetic and the
wording.
There was a problem hiding this comment.
🟡 Changes recommended
Clean builds are blocked by the machine-local dependency replacement, and timing, shader, and MTLB correctness defects remain.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Consolidates GPU capture, profiling, gating, comparison, and evidence workflows across Metal and CUDA.
Changes:
- Adds capture/profiler APIs, host provenance, benchmark integration, and backend discovery.
- Improves timing, shader, counter, Perfetto, replay, and trace-diff accuracy.
- Expands tests, fixtures, research documentation, and formatting checks.
File summaries
| File | Description |
|---|---|
tracebench/example_test.go |
Demonstrates metric reporting. |
tools/perfetto_native_validate_test.go |
Tests Perfetto validation counts. |
tools/nlm-corpus-README.md |
Documents research corpus layout. |
testdata/xcode-oracle-static-tokens2to3/vertices.txt |
Adds Xcode vertex oracle data. |
testdata/xcode-oracle-static-tokens2to3/shaders.txt |
Adds Xcode shader oracle data. |
testdata/xcode-oracle-static-tokens2to3/PROVENANCE.md |
Records oracle provenance. |
testdata/storage-mode-probe/README.md |
Documents storage-mode probe. |
testdata/harness/README.md |
Documents CUDA test harness. |
skills/gputrace/references/linux-nvidia.md |
Adds NVIDIA workflow guidance. |
skills/gputrace/agents/openai.yaml |
Configures agent metadata. |
profilereplay/profilereplay.go |
Exposes profiler replay API. |
Makefile |
Adds formatting and aggregate checks. |
internal/xcodebindings/streamdata_test.go |
Adds real-fixture stream probing. |
internal/xcodebindings/streamdata_darwin_test.go |
Tests Objective-C dictionary keys. |
internal/xcodebindings/shader_cost_darwin_test.go |
Tests shader cost rows. |
internal/xcodebindings/shader_binary.go |
Adds shader-binary wrapper. |
internal/xcodebindings/shader_binary_test.go |
Tests wrapper validation. |
internal/xcodebindings/shader_binary_private_darwin.go |
Enumerates private shader binaries. |
internal/xcodebindings/reproducible_test.go |
Adds reproducibility helper. |
internal/xcodebindings/reproducible_helpers_test.go |
Tests reproducibility helper. |
internal/xcodebindings/framework_path_darwin_test.go |
Tests framework path resolution. |
internal/xcodebindings/counter_data_encoding_darwin_test.go |
Verifies counter ABI encodings. |
internal/xcodebindings/aps_buffers_darwin_test.go |
Tests APS input validation. |
internal/tracebundle/payload_test.go |
Tests bundle classification. |
internal/trace/timing_stats.go |
Reformats timing fields. |
internal/trace/dependencies.go |
Consolidates buffer-name scanning. |
internal/trace/cs.go |
Exposes label classification helpers. |
internal/trace/command_buffer_test.go |
Tests unavailable encoder counts. |
internal/trace/api_calls_test.go |
Tests resource-option decoding. |
internal/timing/unavailable.go |
Defines unavailable-timing semantics. |
internal/timing/profiler.go |
Consolidates profiler discovery and sorting. |
internal/timing/metrics_test.go |
Tests removal of synthetic timing. |
internal/timing/helpers.go |
Removes redundant helper. |
internal/shader/metrics_private_stub.go |
Adds portable private-metric stub. |
internal/shader/metrics_private_darwin.go |
Applies pipeline binary metrics. |
internal/shader/metrics_darwin.go |
Extracts live-register maxima. |
internal/shader/attribution_declaration_test.go |
Tests declaration-level attribution. |
internal/replay/state.go |
Reuses buffer-name parser. |
internal/replay/metal_test.go |
Tests replay validation wording. |
internal/replay/gputools_replay_support_darwin_test.go |
Tests replay initialization. |
internal/replay/gputools_replay_darwin_test.go |
Tests private replay ABI guards. |
internal/replay/counter_metrics.go |
Marks encoder attribution. |
internal/replay/bridge_pure_blit.go |
Removes obsolete blit bridge. |
internal/profilerraw/finddir.go |
Centralizes profiler-directory discovery. |
internal/profilereplay/lock_test.go |
Tests replay locking. |
internal/perfettosql/module.go |
Embeds the SQL projection. |
internal/perfettosql/doc.go |
Documents PerfettoSQL package. |
internal/perfetto/wire.go |
Adds protobuf wire helpers. |
internal/osa/bridge.go |
Restricts AppleScript bridge to Darwin. |
internal/optimize/capture_compare.go |
Exposes capture comparison. |
internal/metallib/parser.go |
Decodes extended MTLB sections. |
internal/metallib/parser_test.go |
Tests MTLB section parsing. |
internal/hostcorrelation/testdata/separate-runs.json |
Adds correlation fixture. |
internal/graph/dot_test.go |
Tests graph provenance boundaries. |
internal/gpuevent/backend_test.go |
Tests backend registry and CUPTI loading. |
internal/gpuevent/backend_other.go |
Disables Metal off Darwin. |
internal/gpuevent/backend_darwin.go |
Reports Darwin Metal availability. |
internal/gate/compare_test.go |
Tests staging comparison. |
internal/evidence/report_test.go |
Tests structural count semantics. |
internal/environment/environment_test.go |
Tests environment comparability. |
internal/difftrace/testdata/report_golden.json |
Updates diff-report fixture. |
internal/difftrace/quick.go |
Adds environment comparison output. |
internal/difftrace/parser_test.go |
Tests raw-trace limitations. |
internal/cupticapture/hash.go |
Adds SHA-256 helper. |
internal/counter/streamdata_encoder_test.go |
Verifies encoder-index offset. |
internal/counter/store_constant_stats_test.go |
Tests constant-calculation metrics. |
internal/counter/sampling_test.go |
Tests raw sample retention. |
internal/counter/objc_values_test.go |
Tests Objective-C input validation. |
internal/counter/csv_import.go |
Stops propagating CSV-only occupancy. |
internal/counter/comprehensive_test.go |
Updates metric expectations. |
internal/counter/binary_extract_test.go |
Removes occupancy logging. |
internal/counter/aps_private_probe_test.go |
Adds opt-in APS probe. |
internal/counter/aps_error_darwin_test.go |
Tests APS errors. |
internal/counter/aps_counter_shard_stub.go |
Adds non-Darwin APS stub. |
internal/counter/aps_counter_shard_darwin.go |
Adds Darwin APS decoding. |
internal/capture/meta.go |
Writes Metal capture provenance. |
internal/analysis/stats.go |
Refines statistics and symlink handling. |
internal/analysis/stats_test.go |
Tests symlink exclusion. |
internal/analysis/buffer_timeline_test.go |
Tests attribution-aware reporting. |
internal/analysis/buffer_access_test.go |
Tests advice suppression. |
internal/admit/report.go |
Renders admission reports. |
gputrace.go |
Exposes missing-capture error. |
gpubench/README.md |
Documents benchmark client. |
gpubench/go.mod |
Defines standalone benchmark module. |
gpubench/example_test.go |
Demonstrates benchmark integration. |
go.mod |
Adds performance and NVIDIA dependencies. |
exp/purego.go |
Adds pure-Go Metal capture. |
exp/purego_test.go |
Tests pure-Go capture. |
exp/example_test.go |
Adds interposer examples. |
exp/doc.go |
Documents experimental package. |
examples/source_mapping/main.go |
Removes synthetic timing fallback. |
docs/TRACE_DIFF_WORKFLOW.md |
Documents divergence options. |
docs/research/PERF_VS_NONPERF_TRACES.md |
Updates parsing status and links. |
docs/research/INSTRUMENTS_TIMING_INVESTIGATION.md |
Corrects format reference. |
docs/research/GPU_PROFILING_APIS_DISCOVERED.md |
Records private-API findings. |
docs/research/FIELD_OFFSET_QUICK_REFERENCE.md |
Consolidates reference links. |
docs/research/BUFFER_FEATURES_STATUS.md |
Updates implementation paths. |
docs/research/BINARY_FORMAT_REFERENCE.md |
Consolidates format references. |
docs/README.md |
Adds benchmark and CUPTI docs. |
cmd/gputrace/main.go |
Centralizes CLI error handling. |
cmd/gputrace/cmd/xcode_payload_darwin.go |
Reports Xcode payload capabilities. |
cmd/gputrace/cmd/xcode_export_postcondition_darwin.go |
Verifies export completion. |
cmd/gputrace/cmd/xcode_export_postcondition_darwin_test.go |
Tests export postconditions. |
cmd/gputrace/cmd/xcode_counters_test.go |
Tests counter option validation. |
cmd/gputrace/cmd/xcode_bindings.go |
Clarifies binding probe output. |
cmd/gputrace/cmd/xcode_bindings_test.go |
Tests probe boundary wording. |
cmd/gputrace/cmd/tree_test.go |
Tests tree nesting. |
cmd/gputrace/cmd/tree_output_test.go |
Tests tree output routing. |
cmd/gputrace/cmd/timing_test.go |
Tests profiler timing semantics. |
cmd/gputrace/cmd/timeline_mio_stub.go |
Adds unsupported-platform stub. |
cmd/gputrace/cmd/timeline_mio_darwin.go |
Reads Xcode GPU time. |
cmd/gputrace/cmd/summary_test.go |
Tests canonical summary vocabulary. |
cmd/gputrace/cmd/span_table.go |
Adds span decomposition table. |
cmd/gputrace/cmd/shaders_xcode_cost_unsupported.go |
Guards Xcode cost off Darwin. |
cmd/gputrace/cmd/shaders_xcode_cost_darwin_test.go |
Tests Xcode cost output. |
cmd/gputrace/cmd/shader_source.go |
Discloses attribution provenance. |
cmd/gputrace/cmd/shader_source_test.go |
Tests provenance formatting. |
cmd/gputrace/cmd/shader_metrics_stub.go |
Adds portable shader-metric stub. |
cmd/gputrace/cmd/shader_metrics_private.go |
Joins metrics to pipelines. |
cmd/gputrace/cmd/replay_counters_test.go |
Tests replay-counter modes. |
cmd/gputrace/cmd/replay_counters_stub.go |
Adds unsupported replay stub. |
cmd/gputrace/cmd/replay_counters_darwin.go |
Implements real counter replay. |
cmd/gputrace/cmd/profiler_input.go |
Adds profiler replay hints. |
cmd/gputrace/cmd/profiler_input_test.go |
Tests profiler hints. |
cmd/gputrace/cmd/profile_replay_test.go |
Tests replay CLI constraints. |
cmd/gputrace/cmd/pprof_test.go |
Tests missing-timing disclosure. |
cmd/gputrace/cmd/platform_unsupported_test.go |
Updates unsupported command tests. |
cmd/gputrace/cmd/platform_darwin.go |
Routes performance display command. |
cmd/gputrace/cmd/perfcounters_validate.go |
Removes unsupported occupancy validation. |
cmd/gputrace/cmd/optimize_compare.go |
Adds noise-aware comparison command. |
cmd/gputrace/cmd/mtlb_test.go |
Tests MTLB error propagation. |
cmd/gputrace/cmd/mtlb_stats.go |
Clarifies MTLB statistics. |
cmd/gputrace/cmd/mtlb_list.go |
Routes MTLB output correctly. |
cmd/gputrace/cmd/mtlb_info.go |
Routes MTLB details correctly. |
cmd/gputrace/cmd/kernels_attribution.go |
Handles unavailable dispatch attribution. |
cmd/gputrace/cmd/kernels_attribution_test.go |
Tests attribution markers. |
cmd/gputrace/cmd/human_limit.go |
Adds bounded human output. |
cmd/gputrace/cmd/human_limit_test.go |
Tests output limits. |
cmd/gputrace/cmd/host_receipt.go |
Adds host-correlation receipts. |
cmd/gputrace/cmd/host_receipt_test.go |
Tests receipt command. |
cmd/gputrace/cmd/help_test.go |
Tests consolidated CLI help. |
cmd/gputrace/cmd/graph.go |
Clarifies graph evidence scope. |
cmd/gputrace/cmd/fences.go |
Labels fence inference as heuristic. |
cmd/gputrace/cmd/dump.go |
Reports dispatch decoding coverage. |
cmd/gputrace/cmd/diff_flags_test.go |
Updates quick-diff option tests. |
cmd/gputrace/cmd/devices.go |
Adds backend inventory command. |
cmd/gputrace/cmd/dependencies_test.go |
Tests limited dependency graphs. |
cmd/gputrace/cmd/counter_metadata_other.go |
Adds portable metadata fallback. |
cmd/gputrace/cmd/counter_metadata_darwin.go |
Enriches counters from Xcode. |
cmd/gputrace/cmd/counter_metadata_darwin_test.go |
Tests counter enrichment. |
cmd/gputrace/cmd/correlate.go |
Removes unsupported occupancy claims. |
cmd/gputrace/cmd/command_buffers_test.go |
Tests output routing. |
cmd/gputrace/cmd/collect_xcode_profile_status_test.go |
Tests profile readiness. |
cmd/gputrace/cmd/collect_xcode_profile_screenshot_test.go |
Tests screenshot verification. |
cmd/gputrace/cmd/collect_xcode_profile_replay.go |
Strengthens replay binding evidence. |
cmd/gputrace/cmd/collect_xcode_profile_open_test.go |
Tests profiler-only rejection. |
cmd/gputrace/cmd/collect_xcode_profile_list.go |
Adds structured error markers. |
cmd/gputrace/cmd/collect_xcode_profile_export_backoff_test.go |
Tests UI probe backoff. |
cmd/gputrace/cmd/collect_xcode_profile_close_test.go |
Tests exact window tracking. |
cmd/gputrace/cmd/collect_xcode_profile_checkbox.go |
Improves target-window lookup. |
cmd/gputrace/cmd/clear_buffers.go |
Clarifies destructive behavior. |
cmd/gputrace/cmd/capture_test.go |
Tests executable path resolution. |
cmd/gputrace/cmd/capture_input.go |
Joins capture sample input. |
cmd/gputrace/cmd/buffers_test.go |
Tests writer-based output. |
cmd/gputrace/cmd/buffers_sort_test.go |
Tests deterministic sorting. |
cmd/gputrace/cmd/buffer_access.go |
Discloses attribution limitations. |
cmd/gputrace/cmd/bench_test.go |
Tests benchmark units. |
cmd/gputrace/cmd/automation_cancel.go |
Escapes AppleScript strings. |
cmd/gputrace/cmd/automation_cancel_quote_test.go |
Tests AppleScript escaping. |
cmd/gputrace/cmd/api_calls_test.go |
Tests bounded API-call output. |
capture/capture.go |
Exposes capture API. |
api_trace.go |
Expands repeated capture parsing API. |
api_timing.go |
Removes synthetic timing API. |
api_surface_test.go |
Updates facade coverage. |
.gitignore |
Ignores generated capture artifacts. |
.github/workflows/ci.yaml |
Adds formatting validation. |
Review details
- Files reviewed: 147/592 changed files
- Comments generated: 9
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ) | ||
|
|
||
| replace github.com/tmc/lib/nvidia => /home/tmc/go/src/github.com/tmc/lib/nvidia |
Comment on lines
+43
to
+44
| path := filepath.Join(bundle, MetaFileName) | ||
| if err := os.WriteFile(path, append(data, '\n'), 0o644); err != nil { |
| high = value | ||
| } | ||
| } | ||
| metrics.HighRegister = int(high) |
Comment on lines
+33
to
+38
| protocol.EnumerateBinariesForPipelineStateEnumerator(pipelineState, func(binary *gtshaderprofiler.GTMioShaderBinaryData) { | ||
| if binary == nil || binary.ID == 0 { | ||
| return | ||
| } | ||
| binaries = append(binaries, &ShaderBinaryData{id: binary.ID}) | ||
| }) |
Comment on lines
+68
to
+69
| header.StringTable = header.FunctionTableSize | ||
| header.BytecodeOffset = header.PublicMetadata |
Comment on lines
+18
to
+27
| sf, err := os.Open(samplesPath) | ||
| if err == nil { | ||
| defer sf.Close() | ||
| sc, err := gpuevent.DecodeJSONL(sf) | ||
| if err == nil { | ||
| cap.Samples = append(cap.Samples, sc.Samples...) | ||
| } | ||
| } else if !os.IsNotExist(err) { | ||
| return cap, err | ||
| } |
| // metalAvailable reports whether the Metal capture stack is usable. | ||
| // On darwin the interposer and replay stack ship with gputrace itself, | ||
| // so availability follows the platform rather than a probe. | ||
| func metalAvailable() bool { return true } |
Comment on lines
+30
to
+34
| Eleven tab-separated Counters-tab exports, one per counter group. Only the | ||
| compute and memory groups carry data: this is a compute-only workload, so | ||
| `vertex-shader.txt`, `fragment-shader.txt`, `primitives.txt`, `vertices.txt`, | ||
| and the two fragment-stage files are present for completeness and are empty of | ||
| compute rows. |
| @@ -0,0 +1,3 @@ | |||
| module github.com/tmc/gputrace/gpubench | |||
CompareCaptures grouped launches by kernel name and reported a mean per name. The same symbol is routinely launched at many geometries in one run, and a mean across them describes no launch that occurred. On a real pair this is not a rounding matter. fp_qmv_single carries +255.07 ms across 18 geometries whose per-launch ratios run from 1.007 to 1.63; the name-level mean read as "7.8% slower per launch at identical launch count and occupancy", which was quoted, acted on, and eventually retracted. A later attempt to stratify by duration instead failed the same way, because duration is not what separates these populations either: at 320 blocks the kernel is 1.18 and at 1280 blocks it is 1.007, at 55 and 159 microseconds respectively. Group instead on the full launch key -- name, grid, block, registers, shared memory. Registers and shared memory belong in it because they change how many blocks are resident, which is usually the thing a duration comparison is trying to hold fixed. Report blocks per launch beside duration. The penalty tracks how much of the device a launch fills, so the block count is the column that makes the pattern legible, and it was not printed anywhere. The name-level table stays, since "where did the time go" is a real question and the symbol is the right unit for it. What changes is that a name spanning several geometries now prints its shape count where the mean used to be. Withholding it is the point: the number was not imprecise, it was unreadable, and nothing about how it was displayed invited doubt. Shape rows also mark unequal launch counts. When the arms ran a geometry a different number of times, the total mixes per-launch cost with work done, and only the mean separates them. The tests pin the misleading case rather than the arithmetic: two geometries moving in opposite directions, where the symbol-level mean reports the opposite of what happened to most launches.
Perfetto refused to place two captures on one timeline, reporting clock_sync_failure_undeferrable_packet_loss and dropping every event of the second: "a sequence-scoped clock (64-127) is used before the ClockSnapshot defining it arrives". The cause is not ordering. Every exported trace declared its sequence-scoped clock against a builtin at (0, 0), and Normalize rebases each capture to its own first event, so two traces both began at zero with nothing relating them. Each loaded alone and no pair could merge. The shim already records what is needed. It writes a clock_sync record pairing CLOCK_REALTIME with the CUPTI clock, ReadCapture decodes it into Capture.ClockSync, and the cupti command then unpacked the capture into events, samples and apis and rebuilt a bare Capture from them -- which dropped the sync record one line after reading it. Build from the decoded capture instead. The snapshot now carries the wall time of source timestamp zero and names CLOCK_REALTIME as the primary trace clock, so two captures land in their true relative positions: the e4b pair resolves 9.757 s apart, which is the real gap between them in the hold that produced them. A capture with no sync record keeps its own clock rather than asserting a wall time nothing measured. Such a trace still loads; it just cannot be merged, which is the honest representation of what is known about it. The delta is applied rather than assumed. On the machines seen so far CUPTI activity timestamps are already unix-epoch nanoseconds and the two differ by a few hundred nanoseconds, but nothing in CUPTI's contract promises that epoch. A pair whose arithmetic would underflow an unsigned subtraction yields no anchor rather than a wrapped one. trace_processor now reports zero undeferrable packet loss on both arms, with slice counts unchanged at 187687 and 177969. It does report one unrelatable clock domain, its own seeded BOOTTIME, which this capture has no reading for; relating it would mean inventing a correspondence that was never measured.
Compilation is host-side work. It lands in no dispatch duration, no encoder span, and no execution cost, so a run that compiles inside its measured window pays a price none of those numbers report. The archive records it and the only reader was the Perfetto exporter, which meant checking whether a warmup worked required exporting a trace and reading its arguments. profiler grows a Shader Compilation section with the capture total, the slowest pipeline, the compile-cache split, and per-phase compiler timings; --kernels gains a per-pipeline Compile line. shaders carries CompilationTimeMs and FunctionWasCached in its csv and json output. They stay out of the --all table, which reproduces Xcode's All Shaders columns and does not have them. Absent and zero are kept apart throughout. FunctionWasCached is a pointer because a trace routinely times a compile without recording a cache result, and the archive writes -1 for a phase it did not measure, which a plain sum would turn into a total smaller than its own parts. The CSV test located its columns by offset from the end of the row. Two new columns moved them, so it now looks them up by header name, which is what it meant.
The GRC counter stream is machine wide. A capture accounts for part of it and the rest belongs to whatever else was on the GPU, so the Execution Cost table describes the attributed subset, not the stream. AttributedFraction computed that share already and no command called it, which left the table reading as though it covered everything. On one fixture the capture owns 912 of 4,989 samples. The encoder counts are printed as two labelled numbers rather than one. The ids carrying counter samples, the ids declared in Encoder Infos, and the capture's own compute encoders are three different populations -- 48, 96, and 3 on that same trace -- and a single unlabelled "encoders" line here would contradict the compute-encoder line four rows above it. That is the conflation 40a3953 removed from the buffer commands; there is no reason to rebuild it here.
The key that buffer bindings are collected under was named EncoderID and rendered as "Encoder 3". It is neither: it is a running count of CS records at the point the binding was seen. On parity-asymmetric-perfdata, a capture with three compute encoders carries ten CS records and produces groups at ordinals 3, 6, 9 and 10 -- four groups against three encoders, in a numbering that shares no value with the 0, 1, 2 that profiler, timing, and the timeline report. That mattered because the name invited a specific wrong join: pair EncoderAccesses with dispatch timing on encoder index and name the buffer the hot kernel moves bytes through. Equality between the two would be coincidence, and the result would be a confident sentence about which kernel touched which buffer that is not true of the capture. AnalyzeBufferAccess already recorded that matching bucket counts cannot prove attribution; the field name and the report contradicted it. Renamed to what it holds: BindingGroupInfo.CSOrdinal, BindingGroups, GroupOrdinals, BufferAlias.Groups, AttributedGroups. The report says "Binding Groups: 4 (keyed by CS ordinal, not encoder index)", lists "CS ordinal 3" rather than "Encoder 3", and states outright that joining these to encoder indices would mislabel which kernel touched which buffer. No mapping is offered. Recovering an encoder from an ordinal needs the CS grouping rule, and one capture showing three groups of three plus a trailing one does not establish it. JSON keys change with the fields; the names were wrong and a compatible spelling of a wrong name is not worth keeping.
… miss Item 03's fields turned out to be exercised after all -- 17 perfdata bundles carry a Compile Performance dictionary -- and first contact with them found two readings that mislead. On qwen25-05b-rotmask-warm-tokens2-4 the section read "14 of 14 pipelines" and "13 cached, 0 compiled". Thirteen plus zero is thirteen, so the fourteenth pipeline is one the archive holds no cache flag for: v_copybfloat16bfloat16 compiled in 3.598 ms and carries no Compile Performance dictionary at all. Left as a subtraction the reader performs in their head, that remainder reads as a cache miss in a run whose own name says warm. It is now counted and printed as "1 with no cache record". The phase timings are worse. Across every trace on hand -- 1,088 phase fields over 17 bundles -- not one is a measurement; every one is the -1 sentinel. The aggregate already excluded them, but it said so in a trailing footnote and printed no phase rows, which reads as "this section has no phase data" rather than "the archive carried these fields and measured none of them". When nothing was measured it now says so on its own line. That is worth stating plainly because it bounds what the fields can do. They were wanted to separate a compile-cache miss from a stuck optimizer. No capture here can do that: the phase timings are unmeasured everywhere, and FunctionWasCached is true on every pipeline that records it, so its discriminating power is untested rather than demonstrated.
Six of the dictionary's seven fields are the -1 sentinel on every capture reachable from this machine, and the seventh has never been observed taking both its values. Documented so the next reader inherits the measurement rather than the plausible rationale, which is that these fields separate a compile-cache miss from a slow optimizer. They cannot: the optimizer half is unmeasured everywhere, and Function was cached is true wherever it appears, so it discriminates nothing that has been seen. Established in two passes over disjoint path sets. Seventeen -perfdata bundles carrying the dictionary: 1,088 phase fields, none measured, no pipeline compiled. The 501 remaining .gputrace and .gpuprofiler_raw paths: 67 carrying the dictionary, 20 distinct captures after deduplicating nested shards, both zeros again. Two traps recorded with it. A bundle and the shard nested inside it are two paths holding one capture, so a path total overstates the sample about fourfold; the zeros survive that but the sample size does not. And grep for "Compile Performance" over the bundle returns nothing on a capture that demonstrably holds 19.434 ms of compile time, because the keys live inside the NSKeyedArchiver shard -- the control was run, and a negative from that method means the method cannot see the data. Also recorded: one pipeline in qwen25-05b-rotmask-warm-tokens2-4 compiles in 3.598 ms with no Compile Performance entry at all, the only such case found. It is a missing record and not a cache miss, which is what subtracting cached plus compiled from the pipeline count would make it look like. Compilation time in milliseconds is the field here with real signal. Capture totals span 9.100 ms to 225.974 ms, but that is a sum over pipelines and the two ends differ in pipeline count as well as cost, so the note says to compare per pipeline.
readSidecarMarks skipped three things and reported none of them. One is uninteresting: a record of another kind is not a mark and never was. The other two are marks the run produced and the reader could not use -- a command_buffer record whose gpu_start_seconds is absent or non-positive, and a line that will not parse. Dropping those quietly is not free, because of what the caller does next. EvaluateStationarity takes consecutive differences over the marks that survive, so a hole does not shorten the trajectory: it merges two gaps into one of roughly twice the duration. Enough of them landing in one block moves that block's median away from the base median, which is exactly the signal the threshold is looking for. A stationary run can be failed for records the sidecar never carried, and the verdict gives the reader nothing to suspect it with. Counted and surfaced now, on a pass as well as a fail. On a fail it is the first thing to rule out. On a pass it bounds how much of the trajectory the verdict actually saw. Found by looking for the shape rather than by tripping over it. Three defects today shared it -- the -1 compiler phases summed as durations, a missing cache record recoverable only by subtraction, and a pre-span launch API rendered as absent -- across NSKeyedArchiver parsing, profiler output, and the CUPTI span path, which share no code. Checked the other candidates in the same sweep: gate's BlitDelta, difftrace's EffectiveGPUTime, and the livetiming optional fields all guard correctly and say "n/a" rather than zero. This one did not.
The Perfetto exporter nil-guarded these six fields and then wrote whatever the pointer held, including the archive's -1 for a phase it did not measure. That put a non-duration into a column named _ns, on a surface whose entire purpose is SQL aggregation: MIN(compiler_backend_ns) returns -1 as the fastest backend pass, AVG is dragged below zero, and nothing in the schema says why. It is not a latent risk. On every capture measured -- 17 bundles carrying the dictionary in one pass, 20 further distinct captures in a disjoint second -- all six fields are -1 on every pipeline. The column is not at risk of holding a sentinel; it holds nothing else. The three states the archive distinguishes all survive, which is the point. A field the archive omitted leaves the duration and the flag both NULL. A recorded zero still reads 0. A recorded -1 leaves the duration NULL and sets <field>_unmeasured. What changes is that the duration columns now only ever hold durations. Found by the sharper of the two rules to come out of today, and it is not the one I proposed. Mine was that these defects live where absence was never modelled. That is true of three of them and would have walked straight past this one: streamdata.go:84 states outright that the pointers distinguish zero, false, and -1 from absent, so absence was modelled deliberately and the defect happened anyway, because the producer put the sentinel inside the pointer. The guard was never what was being defeated. The rule that finds this is: an in-band sentinel entering a field that already has an out-of-band absence representation. Verified against a real trace processor rather than the skipped default. The integration test now asserts all three states in one row: backend recorded -1, total recorded 0, optimization absent.
go test prints ok for a package whether its tests ran or sat out, so the suite reports green end to end while a large share of it never executes. Every "suite green" claimed in this tree today was true and meant less than it read. Measured on a default go test ./...: 1722 passed, 113 skipped, 0 failed. Setting two variables that were already present on this machine -- TRACE_PROCESSOR_SHELL and GPUTRACE_TEST_TRACE -- gives 1754 passed, 81 skipped, 0 failed. Thirty-two tests, no new fixtures and no code changes; the coverage was never asked for. The Perfetto shell had been sitting in ~/tmp since August. make test now pipes through internal/cmd/testcensus, which renders the package lines, prints failures in full, preserves the exit code, and ends with a tally plus the packages the skips concentrate in. make test-gated reports which opt-in variables are set. Neither changes what runs; they change what the run is willing to say about itself. TESTING.md documents TRACE_PROCESSOR_SHELL, which was missing from it despite being the largest single gate at 16 tests and the one most likely to already be satisfied, since the shell is a download rather than a capture. Its existing "77 skips to 62" note is left in place and marked as predating this measurement rather than quietly overwritten. This is the same defect the tool spends its time finding in captures, one level up in the tooling: a skipped test and a passing test are the same line, so absence renders as a value. It is not a bug and nothing here was broken. It changes what the suite's own verdict means.
TESTING.md said GPUTRACE_REQUIRE_PERF_FIXTURES makes "missing optional perf fixtures under the legacy test paths" fail instead of skip. The intent was recorded; the files were not, so the phrase gave a reader no way to act on it and no way to find what it referred to. They are two paths, and they account for 33 of internal/counter's 51 remaining skips -- the largest single block in the suite and, awkwardly, in the package whose parser produced the compile-performance findings landed earlier today. That size is exactly what makes committing them tempting. Do not. No -perf bundle has ever been tracked here; git log --all --diff-filter=A over those paths returns nothing, so this is not a fixture that went missing, it is one that was always meant to be local. Replay output carries the timing of the machine that produced it, and committed it would become a fixture that disagrees with itself on the next person's hardware and fails for reasons unrelated to their change -- a worse defect than 33 skips, and one that survives longer. The structural bundles beside them are committed because they carry no timing. Their absence from .gitignore is not evidence they were meant to be added, which is what I first concluded from it. Records the generation commands so the coverage is one command away for anyone who wants it, without the trap attached.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rolls up the api-forwarders line (517 commits, fast-forward from main): the
capture interposer and CUPTI capture paths, the
gatecommand (completeness,stationarity incl. live timing sidecar, nested-range monotonicity, storage-mode
observation), host-provenance sidecars and cross-host compare labeling,
MTLResourceOptions decoding from capture records, kernel display-name
consolidation across readers, and the docs/research investigations backing
each format claim.
Not yet releasable as a tagged version — two dependency publishes are pending:
go.modcarries a machine-specificreplace github.com/tmc/lib/nvidia => /home/tmc/...;tmc/lib is unpushed (local tag
nvidia/v0.1.0exists).tmc/apple(v0.6.18) lacksobjc/objcinspect,private/appleneuralengine, andprivate/xcode/gtshaderprofiler; those liveonly on the apple private-frameworks worktree branch.
Until those land, builds require the local
go.worksetup. Full test suite isgreen on darwin/arm64 and the branch is synced across m4max and dgx-ts.