Pytorch backend: analysis by DonAurelio · Pull Request #541 · argonne-lcf/THAPI

DonAurelio · 2026-09-16T20:52:19Z

Add overload_name() to the local RecordFunction header and use it to qualify traced operator names. Without it, calls to abs(Tensor) and its abs.out(Tensor, out) overload — the former commonly invoking the latter internally — both showed up as plain aten::abs, indistinguishable from aten::abs reentrantly calling itself; they now appear as aten::abs and aten::abs.out respectively.
Fix corrupted operator durations for nested and reentrant calls by replacing the single per-thread entry timestamp with a per-thread stack.
Add a PyTorch version compatibility check that warns when the traced interpreter's torch version falls outside the range this backend was validated against. This check only catches a change in the RecordScope enum's member count; it will not detect a same-size reordering of RecordScope's members, nor a reordering or type change of RecordFunctionCallback's fields.
Add a LTTNG_UST_PYTORCH_LIBRARY_PATH environment variable so the backend can locate PyTorch's library path directly instead of only detecting it by executing python3 on PATH, supporting calling conventions such as iprof -- venv/bin/python3 model.py.

- libPyTorchInterval.so: babeltrace2 filter turning lttng_ust_pytorch op_entry/op_exit into generic interval lttng:host messages - btx_pytorch_model.yaml: hand-written upstream model (cxi-style, no --matching needed for 2 fixed events) - btx_pytorchinterval_callbacks.cpp: pairs entry/exit via EntryState, keyed by {hostname, vpid, vtid} - Adds BACKEND_PYTORCH to backend_e (utils/xprof_utils.hpp) Verified against a real trace (854 op_entry/op_exit pairs): to_interval produces 854 correctly-named, correctly-timed lttng:host messages. Pending: tally name/level registration, default --backends wiring (next commit); timeline needs no changes (backend-agnostic, verified). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

- utils/xprof_utils.hpp: adds pytorch to pretty_backend_name_g and backend_levels_g at level 6 (its own tier, above itt(5), since RecordFunction ops wrap cuda/ze/omp/mpi calls beneath them) - xprof/xprof.rb.in, utils/babeltrace_thapi.in: add pytorch:6 to the default --backends list so tally/timeline pick it up without an explicit --backends flag Verified against the same real trace (854 op_entry/op_exit pairs): `tally` now prints a dedicated BACKEND_PYTORCH section (854 calls, 27.75ms total) instead of "Wrong Backend passed" warnings; other backends' tally output (ze/cl) is unaffected. Timeline needs no changes -- confirmed backend-agnostic in the prior commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

TApplencourt · 2026-09-17T18:42:10Z

+//                                 never construct one ourselves, only
+//                                 receive a reference from the real library,
+//                                 so no field layout is needed here -- both
+//                                 resolve against their real out-of-line
+//                                 symbols in libtorch_cpu.


I'm not sure this comment is usefull at all.
Why should we ever call anything from the libtorch intenral?

TApplencourt · 2026-09-17T18:43:46Z

+  const int64_t entry_ts = err ? ts : stack.back();
+  if (!err)
+    stack.pop_back();
+


You have a ternary with err, and a if with error.
Can be merged in one block

TApplencourt · 2026-09-17T18:46:03Z

+// PyTorch identifies an operator by TWO strings: a schema name (e.g.
+// "aten::abs") and an overload name (e.g. "" for the default overload,
+// "out" for the variant that writes into a caller-supplied output tensor).
+// fn.name() alone returns only the schema name, so two different overloads
+// of the same op are indistinguishable in the trace. This matters because
+// PyTorch's own operators frequently call one overload from another: the
+// default abs(Tensor) allocates an output tensor and then calls abs.out()
+// to actually compute the result, so a trace keyed on fn.name() alone shows
+// "aten::abs" entering, then "aten::abs" entering AGAIN before the first
+// one exits -- indistinguishable from the op reentering itself, even though
+// it is really two different overloads, one nested inside the other.
+// Emitting fn.name() + "." + fn.overload_name() keeps the two distinguishable.


A little too verbose.
In other too keep the tracer minimal I think it will be best too have 2 field for the tracepoint overload_name and name
And we can do the join inside babeltrace2 pluging

TApplencourt · 2026-09-17T18:48:06Z

+  return (overload[0] == '\0') ? fn.name() : std::string(fn.name()) + "." + overload;
+}

 // ENTRY: fires BEFORE the op runs. LTTng adds time + vpid/vtid via context.


via context?
Why the exit doesn't have such descriptn? It does the same.
So I guess just put
LTTng adds time + vpid/vtid via context. Before ENTRY and EXIT.

Or just remove all this verbosity. on_entry, on_exit sound obvious to me

TApplencourt · 2026-09-17T18:52:51Z

+
+    if torch_lib.nil?
+      torch_info = exec('python3 -c "import torch; print(torch.__file__); print(torch.__version__)"', debug: false)
+      torch_file, torch_version = torch_info.strip.split("\n")


just split should be enougn. No need to strip and split

TApplencourt · 2026-09-17T18:54:34Z

+      torch_lib = torch_file && File.join(File.dirname(torch_file), 'lib')
+
+      if torch_lib.nil?


Kind of a weird pattern.

Same ternary and then if

Do the join inside the else branch

TApplencourt · 2026-09-17T18:55:51Z

+        torch_version_parts = torch_version.split(/[.+ab-]/).reject(&:empty?).first(3).map(&:to_i)
+        # 1.10.0 and older fail this check: RecordScope had fewer members
+        # (6 vs. our 10), shrinking RecordFunctionCallback's scopes_ bit-array.
+        # This only catches size/count changes -- a same-size reordering of
+        # members (e.g. swapping two fields of equal size) would pass
+        # undetected, breaking ABI compatibility.
+        unless ([1, 11, 0]..[2, 14, 0]).cover?(torch_version_parts)
+          LOGGER.warn("THAPI: untested PyTorch version #{torch_version}")
+        end


You can use something like Gem::Version.new('0.4.1') > Gem::Version.new('0.10.1') should work out of the box and be more reasable

TApplencourt · 2026-09-17T18:56:51Z

+        # 1.10.0 and older fail this check: RecordScope had fewer members
+        # (6 vs. our 10), shrinking RecordFunctionCallback's scopes_ bit-array.
+        # This only catches size/count changes -- a same-size reordering of
+        # members (e.g. swapping two fields of equal size) would pass
+        # undetected, breaking ABI compatibility.


Can we just said 1.10.0 fail because RecordScope had fewer members
What is the point of this ABI explanation break? It's not a tutorial on ABI breakage

TApplencourt · 2026-09-17T18:57:11Z

+        # This only catches size/count changes -- a same-size reordering of
+        # members (e.g. swapping two fields of equal size) would pass
+        # undetected, breaking ABI compatibility.
+        unless ([1, 11, 0]..[2, 14, 0]).cover?(torch_version_parts)


2.14.0 is the latest version?

TApplencourt · 2026-09-17T18:59:59Z

+      end
+    end
+
+    unless torch_lib.nil?


Why a

LOGGER.warn('No torch module found for python3, pytorch backend will not be enabled')

And then yet another unless torch_lib.nil? ? Too many branches, doing all the same things

TApplencourt · 2026-09-17T19:01:07Z

-    else
+    # Supported patterns:
+    # * iprof -- python3 model.py (torch env is active).
+    # * iprof -- venv/bin/python3 module.py (non-active torch env, with LTTNG_UST_PYTORCH_LIBRARY_PATH)


I guess we support

LTTNG_UST_PYTORCH_LIBRARY_PATH=${LIBTORCH_USED_BY_VENV} iprof -- venv/bin/python3 module.py

Will be clearer

TApplencourt · 2026-09-18T14:22:47Z

Also I think pytorch may work on the github-CI machine? So we can add a integration tests if you don't mind. :)

Aurelio Vivas and others added 6 commits September 14, 2026 23:48

pytorch: fix entry/exit duration corruption on nested/reentrant ops

a4d774c

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

validate pytorch version

f56ba94

LTTNG_UST_PYTORCH_LIBRARY_PATH

020785b

Formatting

5c02d90

TApplencourt reviewed Sep 17, 2026

View reviewed changes

Navigation Menu

Pytorch backend: analysis - #541

Pytorch backend: analysis#541
DonAurelio wants to merge 6 commits into
argonne-lcf:develfrom
DonAurelio:pytorch-backend-analysis

DonAurelio commented Sep 16, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt Sep 17, 2026

Uh oh!

TApplencourt commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		torch_lib = torch_file && File.join(File.dirname(torch_file), 'lib')

		if torch_lib.nil?

Conversation

DonAurelio commented Sep 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TApplencourt commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants