Describe the bug
A plain PyTorch decoder stack compiles and runs correctly at 12 layers and fails to compile at 16 with:
[INTERNAL_ERROR] [NCC_ILNK020] tensor_map verification failed after linking
The failure is in the compiler, not at runtime. We have ruled out the obvious causes:
- Not memory. Zero allocation failures in the log. The same graph at 12 layers compiles and runs; reducing batch size or sequence length does not help, and the error is a link-stage verification failure rather than an allocation failure.
- Not the number of graphs. An unrelated device sync in our code was initially splitting the model into one graph per layer. Removing it produced a single graph and 16 layers still fails. The only thing that changed is how quickly: about 5 minutes instead of 78.
- Not custom kernels. It reproduces with every NKI kernel disabled, on a pure PyTorch path with no custom calls in the graph.
- Not a simple per-graph layer cap. A 24-layer model split into two 12-layer graphs with xm.mark_step() between them also fails, even though a standalone 12-layer model compiles. A chunk inside a longer stack is not equivalent to the same layer count on its own.
- Not depth or width on their own. We wrote a synthetic decoder of the same shape -- multi-stream residual, softmax attention, MoE as one batched matmul -- and it compiles cleanly at 12, 16 and 24 layers, including at the real width (hidden 4096, 64 heads, intermediate 2048). The script is attached; it needs only torch and torch_xla. So something in our specific layers is implicated, not the size of the graph.
Model Name
GLM-5.3-Flash (glm5_next architecture), 313B parameters. Our own PyTorch implementation of the published architecture, not a Neuron-provided model.
Describe the workload type
Inference (text generation). Tensor parallel 32, expert parallel 32, BF16, batch 1, torch-xla / PJRT.
One layer is: a manifold-constrained hyper-connection block carrying 4 parallel residual streams, RMSNorm, then either a gated linear-attention layer or an MLA attention layer, then either a dense MLP or a 288-expert MoE evaluated as a batched bmm over the 9 experts local to the rank.
Instance Type
trn2.48xlarge, logical-neuroncore-config: 2
Release version
aws-neuronx-dkms 2.30.2.0
aws-neuronx-runtime-lib 2.34.10.0-ac18d186d
aws-neuronx-collectives 2.34.10.0-74eaafac6
aws-neuronx-tools 2.32.28.0-526c2b7f6
neuronx-cc 2.27.5334.0+f702b353
torch 2.11.0
torch-xla 2.11.0
libtorch-neuronx-lite 2.11.0.1.0.1284+f49d8626
nki 0.6.0+31049202112.g85070674
vllm-neuron 0.24.0.1.1.0
Reproduction Steps
We do not have a self-contained reproducer, and we have spent real effort
establishing what does not trigger it. repro_ilnk020.py (attached, needs only
torch and torch_xla, one NeuronCore, no checkpoint) builds a synthetic decoder and
adds our unusual constructs one at a time. Every configuration compiles:
| configuration |
result |
| 12 / 16 / 24 layers, hidden 1024 |
compiles |
| 16 layers, hidden 2048, 16 heads |
compiles |
| 16 layers, hidden 4096, 32 heads, inter 2048 |
compiles |
| 16 layers, hidden 4096, 64 heads, inter 2048 |
compiles |
| + unrolled gated-linear recurrence, hidden 512, seq 32 |
compiles |
| + recurrence, hidden 1024, seq 128 |
compiles |
| + recurrence, hidden 4096, seq 128 |
compiles |
| + unrolled causal depthwise convolution, hidden 4096 |
compiles |
| + recurrence and convolution, hidden 4096, seq 128 |
compiles |
So the following are not sufficient causes, alone or in the combinations above:
depth, width, head count, a multi-stream residual, a mixture-of-experts as one
batched matmul, an unrolled linear-attention recurrence at the real chunk size, and
an unrolled depthwise convolution.
The real model fails at 16 layers with all of these present. We have not reduced the
remaining difference, which is chiefly the MLA attention path (compressed KV latent,
per-head up-projections) and the exact operator mix of the real layers.
What we can give you from the real model:
- Build the 45-layer model described under "workload type" at BF16, tp=32.
- Compile with 12 layers: succeeds, output matches our CPU reference at
cos_sim 1.0000000.
- Compile the identical model with 16 layers: fails with
NCC_ILNK020 after about
5 minutes.
We are happy to share the model code under NDA. The attached script bisects in
90 seconds to 9 minutes per configuration against roughly 40 minutes for the real
model, so if you can say which construct you suspect we can extend it and send you
a runnable case quickly.
Regression Issue
Possible Solution
No suggested fix. As a workaround we cut the layer stack into graphs of at most 8
layers with xm.mark_step() between them, which costs nothing measurable: 16 layers
with one boundary versus three scores cos_sim 0.9998559 vs 0.9998600 against the same
reference.
Logs/Context/Additional Information
The error names the linker, so the first assumption is always memory, and it took us
some time to establish that it is not. If the message could distinguish "graph too
large" from "internal inconsistency" that alone would save people the detour.
Describe the bug
A plain PyTorch decoder stack compiles and runs correctly at 12 layers and fails to compile at 16 with:
The failure is in the compiler, not at runtime. We have ruled out the obvious causes:
Model Name
GLM-5.3-Flash (glm5_next architecture), 313B parameters. Our own PyTorch implementation of the published architecture, not a Neuron-provided model.
Describe the workload type
Inference (text generation). Tensor parallel 32, expert parallel 32, BF16, batch 1, torch-xla / PJRT.
One layer is: a manifold-constrained hyper-connection block carrying 4 parallel residual streams, RMSNorm, then either a gated linear-attention layer or an MLA attention layer, then either a dense MLP or a 288-expert MoE evaluated as a batched bmm over the 9 experts local to the rank.
Instance Type
trn2.48xlarge, logical-neuroncore-config: 2
Release version
aws-neuronx-dkms 2.30.2.0
aws-neuronx-runtime-lib 2.34.10.0-ac18d186d
aws-neuronx-collectives 2.34.10.0-74eaafac6
aws-neuronx-tools 2.32.28.0-526c2b7f6
neuronx-cc 2.27.5334.0+f702b353
torch 2.11.0
torch-xla 2.11.0
libtorch-neuronx-lite 2.11.0.1.0.1284+f49d8626
nki 0.6.0+31049202112.g85070674
vllm-neuron 0.24.0.1.1.0
Reproduction Steps
We do not have a self-contained reproducer, and we have spent real effort
establishing what does not trigger it.
repro_ilnk020.py(attached, needs onlytorch and torch_xla, one NeuronCore, no checkpoint) builds a synthetic decoder and
adds our unusual constructs one at a time. Every configuration compiles:
So the following are not sufficient causes, alone or in the combinations above:
depth, width, head count, a multi-stream residual, a mixture-of-experts as one
batched matmul, an unrolled linear-attention recurrence at the real chunk size, and
an unrolled depthwise convolution.
The real model fails at 16 layers with all of these present. We have not reduced the
remaining difference, which is chiefly the MLA attention path (compressed KV latent,
per-head up-projections) and the exact operator mix of the real layers.
What we can give you from the real model:
cos_sim 1.0000000.
NCC_ILNK020after about5 minutes.
We are happy to share the model code under NDA. The attached script bisects in
90 seconds to 9 minutes per configuration against roughly 40 minutes for the real
model, so if you can say which construct you suspect we can extend it and send you
a runnable case quickly.
Regression Issue
Possible Solution
No suggested fix. As a workaround we cut the layer stack into graphs of at most 8
layers with
xm.mark_step()between them, which costs nothing measurable: 16 layerswith one boundary versus three scores cos_sim 0.9998559 vs 0.9998600 against the same
reference.
Logs/Context/Additional Information
The error names the linker, so the first assumption is always memory, and it took us
some time to establish that it is not. If the message could distinguish "graph too
large" from "internal inconsistency" that alone would save people the detour.