Skip to content

feat(insights): TraceServices-backed trace analysis, plus fix insights-capture never writing a trace - #355

Open
Mistweaver wants to merge 4 commits into
softdaddy-o:mainfrom
Mistweaver:pr/insights-traceservices-analysis
Open

feat(insights): TraceServices-backed trace analysis, plus fix insights-capture never writing a trace#355
Mistweaver wants to merge 4 commits into
softdaddy-o:mainfrom
Mistweaver:pr/insights-traceservices-analysis

Conversation

@Mistweaver

Copy link
Copy Markdown
Contributor

Summary

insights-analyze only ever called stat() on the .utrace file — the tool's own description said full analysis "requires TraceAnalysis module integration (future enhancement)". TraceAnalysis and TraceServices were already linked by SoftUEBridgeEditor for the Rewind Debugger, so no build-rule change was needed to do it properly.

This also fixes a pre-existing bug that made insights-capture produce no trace at all — see part 2, which is why the feature was untestable end-to-end.

Part 1 — real analysis

The tool now opens the trace with IAnalysisService::Analyze and reads the same providers the Unreal Insights UI does, under FAnalysisSessionReadScope:

analysis_type provider
basic_info unchanged — file metadata only, still parses nothing so it stays instant on large captures
frame_stats IFrameProvider — per frame type, avg/median/p90/p95/p99, FPS, hitch counts
top_functions ITimingProfilerProvider::CreateAggregation over every CPU thread and GPU queue
call_tree CreateButterflyGenerateCalleesTree/GenerateCallersTree, rooted at a named timer
counters ICounterProvider — min/max/average/last
csv_stats ICsvProfilerProvider — CSV captures embedded in the trace
threads IThreadProvider
bottlenecks composite — frame health, worst frames, heaviest timers by both inclusive and self time

All accept an optional [start_time, end_time] window plus top_n / hitch_threshold_ms, exposed on the CLI as --start-time / --end-time / --top-n / --hitch-threshold-ms, with --timer-name / --direction / --max-depth for call_tree and --column-filter for csv_stats.

Two design points that matter for agent callers:

  • Percentiles and self time, not just averages. An average frame time hides exactly the hitches you're hunting, and inclusive time alone cannot distinguish an expensive call tree from an expensive function.
  • csv_stats exists because CSV stats are a different instrumentation system. WorldTickMisc and Ticks/* are CSV profiler stats, not CPU timers, so they never appear in top_functions or call_tree — which reads as a tool failure. Worse, WorldTickMisc is declared CSV_SCOPED_TIMING_STAT_EXCLUSIVE (LevelTick.cpp:1481), making it a residual bucket: time in UWorld::Tick no other CSV stat claimed. A large value names an accounting gap, not a slow function. Asking call_tree for it returns that explanation rather than a bare "not found".

Errors are actionable throughout: an unknown timer name returns substring near-misses; a trace with no timing data explains that CPU scopes are compiled out of Shipping entirely (CPUPROFILERTRACE_ENABLED is 0 when UE_BUILD_SHIPPING), so a Shipping capture can never contain them.

ITimingProfilerProvider::ReadTimers was deprecated in 5.8 for GetTimerReader(), which does not exist in 5.7, so timer lookup is version-guarded.

Part 2 — insights-capture silently wrote nothing

It issued Trace.Start "<path>" <channels>. UE splits console command arguments on whitespace without honouring quotes, so any project path containing a space — and Unreal Projects is the UE default — arrives as 3+ args and is rejected:

Cmd: Trace.Start "../../../Users/.../Unreal Projects/.../SmokeTest" cpu,gpu,frame
LogConsoleResponse: Warning: Invalid arguments. Usage: Trace.File [Path] [ChannelSet]

GEngine->Exec still returns true, because the command was handled — so both start and stop reported success while producing no file. It now calls FTraceAuxiliary::Start/Stop directly: the path is a real parameter with no parsing, the return value is a genuine success bool, the reported path is absolute (ProjectSavedDir() is relative to the engine binaries dir), .utrace is appended if missing, and stop reports trace_file_exists.

Verification

Smoke-tested against an 85 MB trace captured from a live VR Preview session, which caught four defects that all compiled cleanly and were only reachable at runtime — each is fixed here and pinned by a regression test:

  1. Crash: Trying to read from session outside of a ReadScope. IAnalysisSession::GetName()/GetDurationSeconds() read like plain accessors but are locked session reads; three call sites ran before the scope opened.
  2. inf in frame_stats: the last frame is still open when tracing stops, and one infinite duration poisoned total/max/average and drove average_fps to 0. Non-finite durations are now excluded and reported as incomplete_frames_skipped.
  3. Misleading timer_count: it came from ITable::GetRowCount(), already capped by TableEntryLimit, so --top-n 12 reported 12 for a trace holding 11,486 timers. Renamed returned_count.
  4. top_timers_by_self_time was not ranked by self time. TableEntryLimit is applied after SortBy, so limiting the aggregation by inclusive time reduced the self-time list to a re-sort of the heaviest call trees. At --top-n 8 it bottomed out at 13.6 ms of self time while omitting CharacterMesh at 53,042 ms — second-largest in the trace — plus RHIThreadLock_Wait (35,515 ms), xrWaitFrame (31,664 ms) and two others. It now aggregates unlimited (EntryLimit > 0 gates the cap, so 0 is unlimited) and ranks each metric over the full set, reporting timers_considered.

Builds clean — 0 errors and 0 warnings from these files — on FantasyMakerVREditor (UE 5.7) and StarKnightsEditor (UE 5.8, forced clean rebuild so everything is back in unity blobs). python -m pytest tests/ — 740 passed, 3 skipped.

Sample output from the VR trace, call_tree on Tick_Engine:

Tick_Engine                       incl=68869.6ms  self=453.1ms
  UWorld_Tick                     incl=61796.8ms  self=980.0ms
     OpenXrWaitFrame              incl=31685.1ms  self=2.4ms
        xrWaitFrame               incl=31664.5ms  self=31664.5ms
     TickCompletionEvents         incl=25013.9ms  self=44.9ms
     FActorComponentTickFunction::ExecuteTick   incl=3491.5ms
        Cable                     incl=2146.5ms   self=2146.5ms  x6332

Notes

🤖 Generated with Claude Code

Mistweaver and others added 4 commits August 7, 2026 19:27
insights-analyze previously only stat()-ed the .utrace file and returned its
size and timestamps - the tool itself said full analysis "requires
TraceAnalysis module integration". TraceAnalysis and TraceServices were
already linked by the editor module (for the Rewind Debugger), so the trace
can be parsed directly.

The tool now opens the trace with IAnalysisService::Analyze and reads the same
providers the Unreal Insights UI does, under an FAnalysisSessionReadScope:

  basic_info     unchanged - file metadata only, still parses nothing so it
                 stays instant on large captures
  frame_stats    IFrameProvider: per frame type count, avg/median/p90/p95/p99,
                 FPS and hitch counts
  top_functions  ITimingProfilerProvider::CreateAggregation over every CPU
                 thread and GPU queue, ranked by total inclusive time
  counters       ICounterProvider: min/max/average/last per counter
  threads        IThreadProvider
  bottlenecks    composite - frame health, the worst frames to jump to, and
                 the heaviest timers by both inclusive and self time

Percentiles and self time are the point: an average frame time hides hitches,
and inclusive time alone cannot distinguish an expensive call tree from an
expensive function.

All analyses accept an optional [start_time, end_time] window plus top_n and
hitch_threshold_ms, exposed on the CLI as --start-time/--end-time/--top-n/
--hitch-threshold-ms.

Verified: builds clean (0 errors, 0 warnings from this file) on both
FantasyMakerVREditor (UE 5.7) and StarKnightsEditor (UE 5.8).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two additions driven by a real profiling dead-end: a packaged frame showing
34.5 ms of "WorldTickMisc" that a flat timer aggregation could not decompose.

**call_tree** - ITimingProfilerProvider::CreateButterfly, exposed as
GenerateCalleesTree / GenerateCallersTree rooted at a named timer. This is the
hierarchical "what actually runs inside this scope" view that top_functions
cannot give. Callers pass a timer name, which is resolved to an id through the
timer reader; a failed lookup returns substring near-misses instead of just
"not found". Depth and per-node child count are bounded, and truncation is
reported via omitted_children/depth_limited so a cut tree is never mistaken
for a leaf.

**csv_stats** - ICsvProfilerProvider, which is where the CSV-only stats live.
This matters because WorldTickMisc and Ticks/* are NOT CPU timers: they are
CSV profiler stats and never appear in top_functions or call_tree, so looking
for them there reads as a tool failure. Worse, WorldTickMisc is declared
CSV_SCOPED_TIMING_STAT_EXCLUSIVE, making it a residual bucket - time in
UWorld::Tick that no other CSV stat claimed - so a large value names an
accounting gap, not a slow function. The returned hint says so and points at
call_tree on the corresponding CPU scope instead.

csv_stats always returns every column name (cheap, from the table layout) so
an agent can discover what to filter on, then summarises only the columns
matching column_filter in a single pass over rows.

Both paths now also report the Shipping caveat in their error text: CPU scopes
are compiled out when UE_BUILD_SHIPPING (CPUPROFILERTRACE_ENABLED is 0), so a
Shipping capture can never contain them and no re-analysis recovers them.

ITimingProfilerProvider::ReadTimers was deprecated in 5.8 for GetTimerReader(),
which does not exist in 5.7, so timer lookup is version-guarded.

Verified: 0 errors and 0 warnings from this file on both FantasyMakerVREditor
(UE 5.7) and StarKnightsEditor (UE 5.8); the 5.7 pass was a full unity rebuild.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Smoke-tested against an 85 MB trace captured from a live VR Preview session.
All four of these compiled cleanly and were only reachable at runtime.

1. Crash: "Trying to read from session outside of a ReadScope". Three sites
   called IAnalysisSession::GetName()/GetDurationSeconds() before opening the
   FAnalysisSessionReadScope (both frame analyses, plus ResolveAgainstSession()
   in Execute). They read like plain accessors but are locked session reads and
   TraceServices asserts. This took down the tool on every non-trivial analysis.

2. insights-capture silently produced no trace at all. It issued
   "Trace.File <path> <channels>", but UE splits console arguments on
   whitespace without honouring quotes, so any project path containing a space
   ("Unreal Projects") arrives as 3+ args and is rejected with "Invalid
   arguments". GEngine->Exec still returns true, because the command *was*
   handled, so both start and stop reported success while writing nothing.
   Now calls FTraceAuxiliary::Start/Stop directly: the path is a real parameter,
   the return value is a genuine success bool, the reported path is absolute,
   and stop reports trace_file_exists.

3. frame_stats reported inf. A frame still open when tracing stops has no end
   time; a single such frame poisoned total/max/average and drove average_fps
   to 0. Non-finite durations are now excluded and reported as
   incomplete_frames_skipped rather than silently dropped.

4. top_functions reported a misleading timer_count. It came from
   ITable::GetRowCount(), which TableEntryLimit has already capped, so
   --top-n 12 reported 12 for a trace containing 11,486 timers. Renamed to
   returned_count, with a note when the cap is hit.

Verified: 0 errors, 0 warnings from these files on FantasyMakerVREditor
(UE 5.7) and StarKnightsEditor (UE 5.8). 742 Python tests pass, with each
defect pinned by a regression test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
top_timers_by_self_time was not the top timers by self time. The aggregation
was requested with SortBy=TotalInclusiveTime and TableEntryLimit=top_n, and
TraceServices applies that limit *after* sorting, so the self-time list was
only a re-sort of the N heaviest call trees. A timer whose self time leads but
whose inclusive time does not was excluded from the very list meant to find it.

Measured on the VR trace at --top-n 8: the old self-time list bottomed out at
FEngineLoop::Tick with 13.6 ms of self time, while CharacterMesh - 53,042 ms
of self time, second-largest in the whole trace - was absent entirely, along
with RHIThreadLock_Wait (35,515 ms), xrWaitFrame (31,664 ms),
FEngineLoop_UpdateTimeAndHandleMaxTickRate (27,987 ms) and ParallelDraw
(14,061 ms). The list was reporting sub-millisecond entries while omitting
multi-second ones.

AnalyzeBottlenecks now aggregates with TableEntryLimit=0 (verified unlimited:
TimingProfiler.cpp gates the cap on EntryLimit > 0) and sorts the full row set
independently for each metric, emitting top_n from each. It reports
timers_considered so the ranking base is visible: 1148 for this trace, versus
the 8 it previously ranked over.

top_functions is unaffected - it is honestly labelled sorted_by
total_inclusive_ms and its limit matches its ordering - but it now shares the
ReadAggregationRows helper rather than duplicating the reader loop.

Verified: 0 errors, 0 warnings on FantasyMakerVREditor (UE 5.7) and
StarKnightsEditor (UE 5.8); re-run against the same trace confirms
CharacterMesh now ranks second by self time. 742 Python tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant