feat(insights): TraceServices-backed trace analysis, plus fix insights-capture never writing a trace - #355
Open
Mistweaver wants to merge 4 commits into
Open
Conversation
insights-analyze previously only stat()-ed the .utrace file and returned its
size and timestamps - the tool itself said full analysis "requires
TraceAnalysis module integration". TraceAnalysis and TraceServices were
already linked by the editor module (for the Rewind Debugger), so the trace
can be parsed directly.
The tool now opens the trace with IAnalysisService::Analyze and reads the same
providers the Unreal Insights UI does, under an FAnalysisSessionReadScope:
basic_info unchanged - file metadata only, still parses nothing so it
stays instant on large captures
frame_stats IFrameProvider: per frame type count, avg/median/p90/p95/p99,
FPS and hitch counts
top_functions ITimingProfilerProvider::CreateAggregation over every CPU
thread and GPU queue, ranked by total inclusive time
counters ICounterProvider: min/max/average/last per counter
threads IThreadProvider
bottlenecks composite - frame health, the worst frames to jump to, and
the heaviest timers by both inclusive and self time
Percentiles and self time are the point: an average frame time hides hitches,
and inclusive time alone cannot distinguish an expensive call tree from an
expensive function.
All analyses accept an optional [start_time, end_time] window plus top_n and
hitch_threshold_ms, exposed on the CLI as --start-time/--end-time/--top-n/
--hitch-threshold-ms.
Verified: builds clean (0 errors, 0 warnings from this file) on both
FantasyMakerVREditor (UE 5.7) and StarKnightsEditor (UE 5.8).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two additions driven by a real profiling dead-end: a packaged frame showing 34.5 ms of "WorldTickMisc" that a flat timer aggregation could not decompose. **call_tree** - ITimingProfilerProvider::CreateButterfly, exposed as GenerateCalleesTree / GenerateCallersTree rooted at a named timer. This is the hierarchical "what actually runs inside this scope" view that top_functions cannot give. Callers pass a timer name, which is resolved to an id through the timer reader; a failed lookup returns substring near-misses instead of just "not found". Depth and per-node child count are bounded, and truncation is reported via omitted_children/depth_limited so a cut tree is never mistaken for a leaf. **csv_stats** - ICsvProfilerProvider, which is where the CSV-only stats live. This matters because WorldTickMisc and Ticks/* are NOT CPU timers: they are CSV profiler stats and never appear in top_functions or call_tree, so looking for them there reads as a tool failure. Worse, WorldTickMisc is declared CSV_SCOPED_TIMING_STAT_EXCLUSIVE, making it a residual bucket - time in UWorld::Tick that no other CSV stat claimed - so a large value names an accounting gap, not a slow function. The returned hint says so and points at call_tree on the corresponding CPU scope instead. csv_stats always returns every column name (cheap, from the table layout) so an agent can discover what to filter on, then summarises only the columns matching column_filter in a single pass over rows. Both paths now also report the Shipping caveat in their error text: CPU scopes are compiled out when UE_BUILD_SHIPPING (CPUPROFILERTRACE_ENABLED is 0), so a Shipping capture can never contain them and no re-analysis recovers them. ITimingProfilerProvider::ReadTimers was deprecated in 5.8 for GetTimerReader(), which does not exist in 5.7, so timer lookup is version-guarded. Verified: 0 errors and 0 warnings from this file on both FantasyMakerVREditor (UE 5.7) and StarKnightsEditor (UE 5.8); the 5.7 pass was a full unity rebuild. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Smoke-tested against an 85 MB trace captured from a live VR Preview session.
All four of these compiled cleanly and were only reachable at runtime.
1. Crash: "Trying to read from session outside of a ReadScope". Three sites
called IAnalysisSession::GetName()/GetDurationSeconds() before opening the
FAnalysisSessionReadScope (both frame analyses, plus ResolveAgainstSession()
in Execute). They read like plain accessors but are locked session reads and
TraceServices asserts. This took down the tool on every non-trivial analysis.
2. insights-capture silently produced no trace at all. It issued
"Trace.File <path> <channels>", but UE splits console arguments on
whitespace without honouring quotes, so any project path containing a space
("Unreal Projects") arrives as 3+ args and is rejected with "Invalid
arguments". GEngine->Exec still returns true, because the command *was*
handled, so both start and stop reported success while writing nothing.
Now calls FTraceAuxiliary::Start/Stop directly: the path is a real parameter,
the return value is a genuine success bool, the reported path is absolute,
and stop reports trace_file_exists.
3. frame_stats reported inf. A frame still open when tracing stops has no end
time; a single such frame poisoned total/max/average and drove average_fps
to 0. Non-finite durations are now excluded and reported as
incomplete_frames_skipped rather than silently dropped.
4. top_functions reported a misleading timer_count. It came from
ITable::GetRowCount(), which TableEntryLimit has already capped, so
--top-n 12 reported 12 for a trace containing 11,486 timers. Renamed to
returned_count, with a note when the cap is hit.
Verified: 0 errors, 0 warnings from these files on FantasyMakerVREditor
(UE 5.7) and StarKnightsEditor (UE 5.8). 742 Python tests pass, with each
defect pinned by a regression test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
top_timers_by_self_time was not the top timers by self time. The aggregation was requested with SortBy=TotalInclusiveTime and TableEntryLimit=top_n, and TraceServices applies that limit *after* sorting, so the self-time list was only a re-sort of the N heaviest call trees. A timer whose self time leads but whose inclusive time does not was excluded from the very list meant to find it. Measured on the VR trace at --top-n 8: the old self-time list bottomed out at FEngineLoop::Tick with 13.6 ms of self time, while CharacterMesh - 53,042 ms of self time, second-largest in the whole trace - was absent entirely, along with RHIThreadLock_Wait (35,515 ms), xrWaitFrame (31,664 ms), FEngineLoop_UpdateTimeAndHandleMaxTickRate (27,987 ms) and ParallelDraw (14,061 ms). The list was reporting sub-millisecond entries while omitting multi-second ones. AnalyzeBottlenecks now aggregates with TableEntryLimit=0 (verified unlimited: TimingProfiler.cpp gates the cap on EntryLimit > 0) and sorts the full row set independently for each metric, emitting top_n from each. It reports timers_considered so the ranking base is visible: 1148 for this trace, versus the 8 it previously ranked over. top_functions is unaffected - it is honestly labelled sorted_by total_inclusive_ms and its limit matches its ordering - but it now shares the ReadAggregationRows helper rather than duplicating the reader loop. Verified: 0 errors, 0 warnings on FantasyMakerVREditor (UE 5.7) and StarKnightsEditor (UE 5.8); re-run against the same trace confirms CharacterMesh now ranks second by self time. 742 Python tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
insights-analyzeonly ever calledstat()on the.utracefile — the tool's own description said full analysis "requires TraceAnalysis module integration (future enhancement)".TraceAnalysisandTraceServiceswere already linked bySoftUEBridgeEditorfor the Rewind Debugger, so no build-rule change was needed to do it properly.This also fixes a pre-existing bug that made
insights-captureproduce no trace at all — see part 2, which is why the feature was untestable end-to-end.Part 1 — real analysis
The tool now opens the trace with
IAnalysisService::Analyzeand reads the same providers the Unreal Insights UI does, underFAnalysisSessionReadScope:analysis_typebasic_infoframe_statsIFrameProvider— per frame type, avg/median/p90/p95/p99, FPS, hitch countstop_functionsITimingProfilerProvider::CreateAggregationover every CPU thread and GPU queuecall_treeCreateButterfly→GenerateCalleesTree/GenerateCallersTree, rooted at a named timercountersICounterProvider— min/max/average/lastcsv_statsICsvProfilerProvider— CSV captures embedded in the tracethreadsIThreadProviderbottlenecksAll accept an optional
[start_time, end_time]window plustop_n/hitch_threshold_ms, exposed on the CLI as--start-time/--end-time/--top-n/--hitch-threshold-ms, with--timer-name/--direction/--max-depthforcall_treeand--column-filterforcsv_stats.Two design points that matter for agent callers:
csv_statsexists because CSV stats are a different instrumentation system.WorldTickMiscandTicks/*are CSV profiler stats, not CPU timers, so they never appear intop_functionsorcall_tree— which reads as a tool failure. Worse,WorldTickMiscis declaredCSV_SCOPED_TIMING_STAT_EXCLUSIVE(LevelTick.cpp:1481), making it a residual bucket: time inUWorld::Tickno other CSV stat claimed. A large value names an accounting gap, not a slow function. Askingcall_treefor it returns that explanation rather than a bare "not found".Errors are actionable throughout: an unknown timer name returns substring near-misses; a trace with no timing data explains that CPU scopes are compiled out of Shipping entirely (
CPUPROFILERTRACE_ENABLEDis 0 whenUE_BUILD_SHIPPING), so a Shipping capture can never contain them.ITimingProfilerProvider::ReadTimerswas deprecated in 5.8 forGetTimerReader(), which does not exist in 5.7, so timer lookup is version-guarded.Part 2 —
insights-capturesilently wrote nothingIt issued
Trace.Start "<path>" <channels>. UE splits console command arguments on whitespace without honouring quotes, so any project path containing a space — andUnreal Projectsis the UE default — arrives as 3+ args and is rejected:GEngine->Execstill returnstrue, because the command was handled — so bothstartandstopreported success while producing no file. It now callsFTraceAuxiliary::Start/Stopdirectly: the path is a real parameter with no parsing, the return value is a genuine success bool, the reported path is absolute (ProjectSavedDir()is relative to the engine binaries dir),.utraceis appended if missing, andstopreportstrace_file_exists.Verification
Smoke-tested against an 85 MB trace captured from a live VR Preview session, which caught four defects that all compiled cleanly and were only reachable at runtime — each is fixed here and pinned by a regression test:
Trying to read from session outside of a ReadScope.IAnalysisSession::GetName()/GetDurationSeconds()read like plain accessors but are locked session reads; three call sites ran before the scope opened.infinframe_stats: the last frame is still open when tracing stops, and one infinite duration poisoned total/max/average and droveaverage_fpsto 0. Non-finite durations are now excluded and reported asincomplete_frames_skipped.timer_count: it came fromITable::GetRowCount(), already capped byTableEntryLimit, so--top-n 12reported 12 for a trace holding 11,486 timers. Renamedreturned_count.top_timers_by_self_timewas not ranked by self time.TableEntryLimitis applied afterSortBy, so limiting the aggregation by inclusive time reduced the self-time list to a re-sort of the heaviest call trees. At--top-n 8it bottomed out at 13.6 ms of self time while omittingCharacterMeshat 53,042 ms — second-largest in the trace — plusRHIThreadLock_Wait(35,515 ms),xrWaitFrame(31,664 ms) and two others. It now aggregates unlimited (EntryLimit > 0gates the cap, so0is unlimited) and ranks each metric over the full set, reportingtimers_considered.Builds clean — 0 errors and 0 warnings from these files — on
FantasyMakerVREditor(UE 5.7) andStarKnightsEditor(UE 5.8, forced clean rebuild so everything is back in unity blobs).python -m pytest tests/— 740 passed, 3 skipped.Sample output from the VR trace,
call_treeonTick_Engine:Notes
csv_statshas been exercised against the empty case (capture_count: 0) but not yet against a trace containing real CSV captures — UE ignored thecsvtrace channel in my capture, which is consistent with CSV profiling needing to be started separately rather than merely having its channel enabled.🤖 Generated with Claude Code