About this report: This analysis was produced automatically by Hermes (an autonomous AI agent) during a systematic debugging session of the H3 (MiniMax-H3) workflow in ComfyUI. All measurements, patches and conclusions below were made by the agent on a rented cloud GPU instance (vast.ai, RTX 3090 24GB, accessed via SSH). The debug markers cited (H3DBG2/H3DBG3) are print statements the agent inserted into the model code to obtain the measurements.
Expected Behavior
The H3 image-to-video workflow (MiniMax-H3, ComfyUI native implementation) should produce a coherent 5-second video from a reference image + text prompt (e.g. a sheep character with clear structure, motion, and a latent that converges to the reference distribution, std ~0.96).
Actual Behavior
The output video is pure noise / color texture — no recognizable subject, no structure. This happens regardless of sampler formula, step count, attention backend, or quantization path. The final latent stays at noise level (std 1.05-1.72 instead of ~0.96), and the first-step prediction is nearly orthogonal to the input (cosine -0.02 vs. reference -0.7).
Sample frames (all different configurations, all broken):
Steps to Reproduce
- ComfyUI master (pin
344b4398) + comfy-kitchen >= 0.2.27, no custom nodes required.
- Download the official models from
Comfy-Org/MiniMax-H3:
diffusion_models/minimax_h3_fl2va_pruned_int8_convrot.safetensorstext_encoders/qwen3vl_32b_minimax_h3_bf16.safetensorsvae/minimax_h3_fl2va_pruned_video_vae.safetensors and the audio VAE
- Run the official H3 i2v API workflow: 28 steps, 544x960, 5s (12 frames), guidance-distilled, shift 12/3.
- GPU: RTX 3090 24GB, CUDA 12.x, driver 535+.
Debug Logs
The H3DBG2/H3DBG3 markers cited in the intro are debug print statements injected into comfy/ldm/minimax/model.py at specific points of the forward pass, using scripted find-and-replace patches (one patch per measurement point, applied via SSH to a rented cloud GPU instance — vast.ai, RTX 3090 24GB — server restarted, workflow submitted, values collected from the server log):
H3DBG2 in: std 1.001-1.021 (input stays noise-level across ALL steps)
H3DBG2 out: -0.077 0.351 | cosine: -0.0004 (first step, sigma=1 — prediction orthogonal to input)
H3DBG2 out: -0.05 0.32 | cosine: -0.7007 (after adding reference scale_noise interpolation — late steps now correlate)
H3DBG3 sampler-out VIDEO-Teil: 0.0543 1.0521 (final latent std 1.05 — noise)
The reference pipeline's first-step cosine is ~-0.7 ("denoise direction" out of the noise); the ComfyUI chain measures -0.02 at sigma=1 even though the input at sigma=1 is mathematically identical (x_t = noise).
How the instrumentation was built
- H3DBG2-in: just before the transformer blocks — mean/std of the model input latent
x
- H3DBG2-out: right after the output heads — mean/std of the prediction plus the cosine correlation between the prediction and the input latent
- H3DBG3: in the sampler after the last step — statistics of the final denoised latent
The same technique was used for every other check listed below (AdaLN table statistics, attention outputs, RoPE positions).
Concrete deviations found — where ComfyUI's code does not do what it should
-
Missing input interpolation (scale_noise) — Measured: the DiT input stays at pure-noise level across all steps (std 1.001–1.021, constant). The reference pipeline interpolates the noisy latent with the reference-image latent: x_t = sigma*noise + (1-sigma)*latent_image (the scale_noise step in the reference's before_denoise.py). In the ComfyUI chain the sampler passes only pure noise to the model — the interpolation step is absent. Consequence (measured): the first prediction is nearly orthogonal to the input (cosine -0.02 at sigma=1 vs. -0.7 reference). I patched the interpolation into the model forward; the correlation then rises to -0.70 in late steps, but the output is still noise.
-
AdaLN projection padded with random values — The curve-form conversion compresses the full 2688-dim time-modulation weights to an 8-dim basis (adaln_proj weight shape (96768, 8)). On load, the 8-dim weights are widened to 16-dim to match the 16-dim curve table — but columns 8–15 are not loaded from the file: they carry random initialization values (measured absmax 10.44, while the real 8 basis columns are ~1). Those random columns corrupt the per-timestep modulation. I neutralized them (set to zero) — the correlation improved but the output is still noise.
-
Prediction magnitude ~4x too small — Even with the interpolation in place, the DiT velocity prediction measures std 0.32 vs. ~1.4 expected from the reference. The sampler cannot converge: the final latent stays at noise level (std 1.05–1.72 vs. ~0.96 reference). This is consistent across every formula, step count and attention variant tested.
Why the quantization/conversion layer (comfy-kitchen) is the prime suspect
The H3 support in ComfyUI comes from the comfy-kitchen conversion: the model architecture (comfy/ldm/minimax/model.py), the weight loading/dispatch (comfy/ops.py, comfy/utils.py QUANT_ALGOS) and the quantized checkpoint itself (minimax_h3_fl2va_pruned_int8_convrot.safetensors, int8 with block scales) are all produced by comfy-kitchen. The remaining evidence points there:
- The ~4x prediction damping survives every Python-side correction. After ruling out formulas, steps, attention, input interpolation, AdaLN corruption and latent normalization, the only untested layer left is the int8 dequantization path of the checkpoint: the block scales / dequant algorithm of the converted int8 file.
- The measured dequantized weight statistics are plausible (std 0.084, int8 range -127..127) — so the scales themselves are not obviously wrong, yet the network output is 4x too small, which is what a scaling mismatch inside the quantized graph (e.g. wrong block-size grouping, wrong scale application order, or a kernel path that silently truncates) would produce.
- Important nuance: forcing float32 dequantization (
full_precision_matrix_mult) produced identical values to the int8 kernels — so the runtime kernels execute the dequant correctly; the suspicion is on the conversion data (the scales baked into the checkpoint), not on the kernel dispatch. On an A100/H100 (full bf16 path, no sm_86 fallbacks) this checkpoint class is the reference environment — on RTX 3090 we cannot get a clean bf16 comparison.
Issues checked that do NOT explain it
Each of these was verified by measurement against the reference pipeline; none explains the noise output:
- Missing
scale_noise input interpolation — patched in; cosine improves from -0.02 to -0.70 in late steps, but the output remains noise.
- AdaLN projection random-padded columns — neutralized; output remains noise.
- Sampler formulas — 5 variants (CONST, X0-as-output, sign flips, sigma compensation, velocity scaling): identical output.
- Step counts — 20/28/40/50: identical output.
- Attention backend — fused kernels vs. plain corrected attention: bit-identical values.
- int8 kernels vs. forced float32 dequant — identical values (kernels are numerically correct).
- VAE latent normalization — ComfyUI already normalizes with
latents_mean/latents_std, identical to reference encoders.
- M-RoPE positions — identical to reference (
get_rope_index logic, mrope [24,20,20], interleaved, theta 5e6).
- AdaLN time modulation health — table 4097x16 verified U-shaped, modulation scale 0.44 within expected range.
- Context segments / token tags — text 262 + vision 578 tokens, tags video=0/text=1/audio=2, no attention mask: all match the reference.
Related H3 issues on this repo — checked, none explains this
Related comfy-kitchen issues (Comfy-Org/comfy-kitchen)
- #98 — CUTLASS INT8 dequant selects poor config for tall MiniMax H3 shapes on RTX A6000: the automatic
cutlass_int8_dequant dispatch picks a poor configuration for the exact H3 production projection shapes we also hit (M=80661, N=21504/5376/28672, K=5376/7168/14336; ConvRot true, groupsize 256) on an sm86 GPU. Notably this issue reports the problem on an RTX A6000 — the same compute capability (sm86) as our RTX 3090. It is filed as a performance problem (INT8 no faster than BF16), not a correctness problem — which matches our measurement that forced float32 dequantization produces identical values. Still, it confirms that the H3 INT8 dequant path on sm86 is known to behave poorly, and we cannot rule out that a related scale/config defect in that path causes the ~4x output damping we observe.
- #92 — eager backend advertises int8_linear on MPS but dispatches to CUDA-only torch._int_mm: dispatch bug in the int8 linear path. Not directly applicable (we run CUDA, not MPS), but listed to show the int8 dispatch layer has known correctness hazards.
- #55 — INT8 Krea2 models broken on Tesla T4 (SM75): CUDA invalid argument or corrupted outputs: native INT8 checkpoints on a non-A100 GPU produce exactly our symptom class — "deterministic tiled psychedelic noise instead of a valid image" — while FP8/NF4/GGUF versions of the same model work on the same system. This is the strongest precedent that INT8 dequant kernels are compute-capability-sensitive in comfy-kitchen (there on SM75, possibly on SM86 as well).
Related discussions on the Comfy-Org/MiniMax-H3 Hugging Face repo
- #18 — How about my RTX 3090 for running H3? Q4/Q2 or INT8 which version: Kijai (Comfy Org) confirms the INT8 variant is the intended choice for the RTX 3090 ("Int8 indeed"). Other users report working int8_convrot generations on 3090-class hardware (RTX 5060 Ti / 5080 reports, albeit slow) — so the failure is not GPU-wide on sm86; it narrows the cause to something specific in the conversion/chain (the same int8 checkpoint produces noise here).
- #36 — Problem with convrot model: a convrot-specific CUDA crash (unknown error) at 10s duration on RTX 5090, worked around by switching the PyTorch/CUDA runtime (cu128 → cu130). Different symptom (crash vs. noise), but it demonstrates that the convrot kernel path is sensitive to the CUDA/runtime environment.
- #30 — Why MiniMax H3 Ruins Faces on Wide Shots?: structured quality artifacts on wide shots. Different symptom class (structured degradation, not full noise) — not this bug.
Other — Environment note
"Reference" throughout this report refers to the official MiniMax H3 diffusion pipeline code (the reference implementation used to validate each component): the model's before_denoise.py (sampler/timestep convention and RoPE constants), encoders.py (latent normalization with latents_mean/latents_std), get_rope_index (M-RoPE position computation) and the transformer block code. Every measurement in this report was compared against that reference implementation.
- Environment: all debugging was performed on a rented cloud GPU instance (vast.ai, RTX 3090 (sm_86), 24GB, CUDA 12.x) accessed via SSH — there is no local GPU involved. The reference notes "bf16 only on A100+". We could not produce a clean bf16 comparison on this GPU — filing this issue to confirm whether the 3090's kernel/quantization fallbacks (or the comfy-kitchen int8 conversion data) are the cause, since every Python-side check passes.
Follow-up verification on Ada GPU (L40, sm_89) — root cause confirmed
After filing this issue, the identical workflow (ComfyUI pin 344b4398, unpatched, official minimax_h3_fl2va_pruned_int8_convrot.safetensors weights, 28 steps, 576×1024, 5s, euler/simple, cfg 1.0, shift 12/3) was run on a rented L40 (Ada, sm_89, 48GB) via vast.ai:
- Result: a coherent, clean 5-second video — the sheep character with clear structure and motion, no noise, no texture artifacts (frames visually verified).
- None of the debug patches were applied — pure ComfyUI code + official weights.
- Runtime: ~9.4 min (18.7 s/it) on the L40.
This confirms the root cause is specific to the RTX 3090 / sm_86 environment, not the weights, the workflow, the sampler, or the comfy-kitchen conversion data. The dispatch mechanism is explained by comfy/quant_ops.py: the comfy-kitchen CUDA/CUTLASS backend is disabled when torch.version.cuda < 13.0 (our 3090 environment ran pytorch cu128), and the Triton backend is disabled on NVIDIA by default — so the entire INT8 chain executed in the eager (fallback) mode on the 3090. The same weights on the Ada GPU (where the fallback path behaves differently) produce correct output, narrowing the fault to the eager/fallback kernel path on sm_86.
Additional performance data from the L40 (same seed, same workflow): eager 561s, SageAttention 562s (no gain — attention is not the bottleneck), comfy-kitchen Triton backend 331s (1.7× faster). Upgrading pytorch to cu130 would additionally activate the CUTLASS backend.
Resolution status: works correctly on Ada (sm_89); the failure is environment-specific to the RTX 3090 (sm_86) kernel/backend fallback path. The debug patches (input-interpolation, velocity scaling, AdaLN zeroing) were 3090-specific workarounds and are NOT needed on the L40.
Expected Behavior
The H3 image-to-video workflow (MiniMax-H3, ComfyUI native implementation) should produce a coherent 5-second video from a reference image + text prompt (e.g. a sheep character with clear structure, motion, and a latent that converges to the reference distribution, std ~0.96).
Actual Behavior
The output video is pure noise / color texture — no recognizable subject, no structure. This happens regardless of sampler formula, step count, attention backend, or quantization path. The final latent stays at noise level (std 1.05-1.72 instead of ~0.96), and the first-step prediction is nearly orthogonal to the input (cosine -0.02 vs. reference -0.7).
Sample frames (all different configurations, all broken):
Steps to Reproduce
344b4398) +comfy-kitchen >= 0.2.27, no custom nodes required.Comfy-Org/MiniMax-H3:diffusion_models/minimax_h3_fl2va_pruned_int8_convrot.safetensorstext_encoders/qwen3vl_32b_minimax_h3_bf16.safetensorsvae/minimax_h3_fl2va_pruned_video_vae.safetensorsand the audio VAEDebug Logs
The H3DBG2/H3DBG3 markers cited in the intro are debug print statements injected into
comfy/ldm/minimax/model.pyat specific points of the forward pass, using scripted find-and-replace patches (one patch per measurement point, applied via SSH to a rented cloud GPU instance — vast.ai, RTX 3090 24GB — server restarted, workflow submitted, values collected from the server log):The reference pipeline's first-step cosine is ~-0.7 ("denoise direction" out of the noise); the ComfyUI chain measures -0.02 at sigma=1 even though the input at sigma=1 is mathematically identical (x_t = noise).
How the instrumentation was built
xThe same technique was used for every other check listed below (AdaLN table statistics, attention outputs, RoPE positions).
Concrete deviations found — where ComfyUI's code does not do what it should
Missing input interpolation (
scale_noise) — Measured: the DiT input stays at pure-noise level across all steps (std 1.001–1.021, constant). The reference pipeline interpolates the noisy latent with the reference-image latent:x_t = sigma*noise + (1-sigma)*latent_image(thescale_noisestep in the reference'sbefore_denoise.py). In the ComfyUI chain the sampler passes only pure noise to the model — the interpolation step is absent. Consequence (measured): the first prediction is nearly orthogonal to the input (cosine -0.02 at sigma=1 vs. -0.7 reference). I patched the interpolation into the model forward; the correlation then rises to -0.70 in late steps, but the output is still noise.AdaLN projection padded with random values — The curve-form conversion compresses the full 2688-dim time-modulation weights to an 8-dim basis (
adaln_projweight shape (96768, 8)). On load, the 8-dim weights are widened to 16-dim to match the 16-dim curve table — but columns 8–15 are not loaded from the file: they carry random initialization values (measured absmax 10.44, while the real 8 basis columns are ~1). Those random columns corrupt the per-timestep modulation. I neutralized them (set to zero) — the correlation improved but the output is still noise.Prediction magnitude ~4x too small — Even with the interpolation in place, the DiT velocity prediction measures std 0.32 vs. ~1.4 expected from the reference. The sampler cannot converge: the final latent stays at noise level (std 1.05–1.72 vs. ~0.96 reference). This is consistent across every formula, step count and attention variant tested.
Why the quantization/conversion layer (comfy-kitchen) is the prime suspect
The H3 support in ComfyUI comes from the comfy-kitchen conversion: the model architecture (
comfy/ldm/minimax/model.py), the weight loading/dispatch (comfy/ops.py,comfy/utils.pyQUANT_ALGOS) and the quantized checkpoint itself (minimax_h3_fl2va_pruned_int8_convrot.safetensors, int8 with block scales) are all produced by comfy-kitchen. The remaining evidence points there:full_precision_matrix_mult) produced identical values to the int8 kernels — so the runtime kernels execute the dequant correctly; the suspicion is on the conversion data (the scales baked into the checkpoint), not on the kernel dispatch. On an A100/H100 (full bf16 path, no sm_86 fallbacks) this checkpoint class is the reference environment — on RTX 3090 we cannot get a clean bf16 comparison.Issues checked that do NOT explain it
Each of these was verified by measurement against the reference pipeline; none explains the noise output:
scale_noiseinput interpolation — patched in; cosine improves from -0.02 to -0.70 in late steps, but the output remains noise.latents_mean/latents_std, identical to reference encoders.get_rope_indexlogic, mrope [24,20,20], interleaved, theta 5e6).Related H3 issues on this repo — checked, none explains this
Related comfy-kitchen issues (Comfy-Org/comfy-kitchen)
cutlass_int8_dequantdispatch picks a poor configuration for the exact H3 production projection shapes we also hit (M=80661, N=21504/5376/28672, K=5376/7168/14336; ConvRot true, groupsize 256) on an sm86 GPU. Notably this issue reports the problem on an RTX A6000 — the same compute capability (sm86) as our RTX 3090. It is filed as a performance problem (INT8 no faster than BF16), not a correctness problem — which matches our measurement that forced float32 dequantization produces identical values. Still, it confirms that the H3 INT8 dequant path on sm86 is known to behave poorly, and we cannot rule out that a related scale/config defect in that path causes the ~4x output damping we observe.Related discussions on the Comfy-Org/MiniMax-H3 Hugging Face repo
Other — Environment note
"Reference" throughout this report refers to the official MiniMax H3 diffusion pipeline code (the reference implementation used to validate each component): the model's
before_denoise.py(sampler/timestep convention and RoPE constants),encoders.py(latent normalization withlatents_mean/latents_std),get_rope_index(M-RoPE position computation) and the transformer block code. Every measurement in this report was compared against that reference implementation.Follow-up verification on Ada GPU (L40, sm_89) — root cause confirmed
After filing this issue, the identical workflow (ComfyUI pin
344b4398, unpatched, officialminimax_h3_fl2va_pruned_int8_convrot.safetensorsweights, 28 steps, 576×1024, 5s, euler/simple, cfg 1.0, shift 12/3) was run on a rented L40 (Ada, sm_89, 48GB) via vast.ai:This confirms the root cause is specific to the RTX 3090 / sm_86 environment, not the weights, the workflow, the sampler, or the comfy-kitchen conversion data. The dispatch mechanism is explained by
comfy/quant_ops.py: the comfy-kitchen CUDA/CUTLASS backend is disabled whentorch.version.cuda < 13.0(our 3090 environment ran pytorch cu128), and the Triton backend is disabled on NVIDIA by default — so the entire INT8 chain executed in the eager (fallback) mode on the 3090. The same weights on the Ada GPU (where the fallback path behaves differently) produce correct output, narrowing the fault to the eager/fallback kernel path on sm_86.Additional performance data from the L40 (same seed, same workflow): eager 561s, SageAttention 562s (no gain — attention is not the bottleneck), comfy-kitchen Triton backend 331s (1.7× faster). Upgrading pytorch to cu130 would additionally activate the CUTLASS backend.
Resolution status: works correctly on Ada (sm_89); the failure is environment-specific to the RTX 3090 (sm_86) kernel/backend fallback path. The debug patches (input-interpolation, velocity scaling, AdaLN zeroing) were 3090-specific workarounds and are NOT needed on the L40.