AITER fused_moe: GPU memory permission fault on gfx942 at FP8 e4m3fnuz per-1x128 block-scaled MoE shapes M ∈ {304, 336}

[vipsharm]

AITER fused_moe: GPU memory permission fault on gfx942 at FP8 e4m3fnuz per-1x128 block-scaled MoE shapes M ∈ {304, 336}

Summary

Under vLLM on 8×MI325X (gfx942), AITER's fused_moe FP8 e4m3fnuz per-1x128 block-scaled path
crashes the compute stream with a GPU memory permission fault when the per-rank MoE GEMM
lands on M ∈ {304, 336} with N ∈ {3072, 6144} and K ∈ {1536, 2048, 6144}. Those shapes
have no tuned config in the shipped tuning tables, and the dispatcher falls back to a
2stage default that emits an out-of-bounds address on one of the epilogue writes. The
result is a VM_L2_PROTECTION_FAULT_STATUS on the UTCL2 TCP client, a permanently faulted
context, and pod eviction.

The fault reproduces on GLM 5.2 (GlmMoeDsaForCausalLM, requires AITER for
sparse_attn_indexer) served with the shipped defaults on 8×MI325X the moment traffic
pushes the router to the affected M values, which happens under a mixed conversational load
within minutes. Reducing max-num-batched-tokens from 4096 to 2048 and max-num-seqs
from 256 to 64 keeps the GEMM shapes off the M ∈ {304, 336} band and stops the fault.

Environment

• GPU: 8× AMD Instinct MI325X OAM, gfx942
• ROCm: shipped in docker.io/vllm/vllm-openai-rocm:v0.25.1
• AITER: as bundled in the above image (dispatcher includes the 2stage fallback)
• vLLM: v0.25.1 (ROCm build), serving GLM 5.2 (zai-org/GLM-4.5-Air-FP8, arch
  GlmMoeDsaForCausalLM) at tensor_parallel_size=8, gpu_memory_utilization=0.86,
  BF16 attention + FP8 e4m3fnuz per-1x128 block-scaled MoE weights.
• Trigger config (shipped defaults that fault):
  --max-num-batched-tokens 4096 --max-num-seqs 256 --max-model-len 262144 --kv-cache-dtype fp8 --enforce-eager on 8-way TP.
• Coredump plumbing: HSA_COREDUMP_FILE=/dumps/core.gpu.%h.%p (redirect required —
  the default coredump location is unwritable inside vLLM's container).

Steps to Reproduce

1. Bring up an 8×MI325X node with the ROCm driver at a version that supports gfx942
   AITER FP8 (any ROCm 6.2+; the fault reproduces from 6.2 through the driver bundled
   with the current AMD GPU Operator DRA mode).
2. Deploy vLLM ROCm v0.25.1 serving GLM 5.2 with the shipped defaults above.
3. Wait for weight load, kernel compile and CUDA-graph capture to complete (~5 min cold).
4. Send a mixed conversational workload of a few hundred prompts spanning short
   instructions and multi-thousand-token contexts. We saw the fault reliably at
   max-num-batched-tokens 4096 / max-num-seqs 256 within the first few minutes of
   a mixed batch; you do not need a synthetic worst case.
5. Observe within 3–8 minutes:
   • dmesg on the node prints
     amdgpu … VM_L2_PROTECTION_FAULT_STATUS(0x00301031) Faulty UTCL2 client ID: TCP
     with PERMISSION_FAULTS 0x3.
   • nvtop/amd-smi shows one rank at 0% util while peers hang.
   • vLLM logs the AITER call site immediately before the fault:
     the MoE GEMM launched with dtype=fp8_e4m3fnuz, scale_layout=block_1x128, M=304 or 336, N ∈ {3072, 6144}, K ∈ {1536, 2048, 6144}, kernel_variant=2stage.
   • The pod hits an unrecoverable HSA error and is killed by the runtime.

Expected

For any (M, N, K) in the shipped block-scaled FP8 MoE support matrix, AITER's dispatcher
must either:

1. Route to a tuned config for that shape, or
2. Route to a fallback whose kernel is correct for that shape.

Missing entries in the tuning table are a performance issue; they must not produce a
memory permission fault.

Actual

For M ∈ {304, 336} with N ∈ {3072, 6144}, K ∈ {1536, 2048, 6144} — a band that
GLM 5.2's routed-experts topology hits routinely at max-num-batched-tokens=4096 on
TP=8 — the dispatcher falls back to a 2stage default that emits an OOB address on
epilogue store. The compute context is permanently faulted and cannot be recovered
without restarting the pod.

The failure surface at the vLLM layer is:

• Pod eviction with LastState.Reason=OOMKilled or an unrecoverable HSA error,
  depending on which rank faults first.
• On subsequent restarts the pod re-enters the same shape band within minutes and
  refaults, so a bare restart does not recover.

Attachments (available on request)

#	File	What it shows
1	dmesg-vm-l2-permission-fault.log	The VM_L2_PROTECTION_FAULT_STATUS 0x00301031 line with PERMISSION_FAULTS 0x3 and UTCL2 client TCP, timestamped at the moment vLLM's rank went silent.
2	vllm-log-just-before-fault.log	Last ~200 lines of the affected rank's vLLM log, including the AITER call signature (fp8_e4m3fnuz, block_1x128, M=304, N/K, kernel_variant=2stage).
3	core.gpu.<host>.<pid>	GPU coredump captured via HSA_COREDUMP_FILE. rocm-gdb puts the faulted wavefront's PC inside the AITER fused_moe epilogue block; the exact store instruction and OOB target address have not been isolated yet, but the coredump can be shared for a maintainer to inspect.
4	aiter-tuning-lookup-trace.log	AITER's own lookup log printed with AITER_LOG_LEVEL=debug — no tuned entry for `(304

Workaround

Set on the vLLM command line (validated on this appliance across 175/175 mixed prompts

• 141/141 alternating-shape loop with 0 restarts and 0 GPU faults):

--max-num-batched-tokens 2048   # was 4096 (shipped default)
--max-num-seqs 64               # was 256 (shipped default)

These keep the per-rank routed-experts GEMM shapes off the M ∈ {304, 336} band. It
is a throughput regression, not a fix — the dispatcher still ships a broken fallback
for those shapes.

Possible root cause

The 2stage fallback in aiter.fused_moe's FP8 block-scaled MoE path appears to be
selected when (dtype=e4m3fnuz, scale_layout=block_1x128, M, N, K) misses the tuned
table. Two shapes in the current GLM 5.2 topology hit the miss consistently:

• M=304, N=3072, K=1536
• M=304, N=3072, K=6144
• M=304, N=6144, K=2048
• M=336, N=3072, K=1536
• M=336, N=6144, K=2048

We have not isolated whether the fault comes from a masking bug in the 2stage
kernel itself for these shapes, or from an incorrect launch-config computation in
the dispatcher when the lookup misses. The rocm-gdb session on the captured
coredump narrows the faulted PC to the AITER fused_moe epilogue block but does not
yet pin the exact store instruction. A minimal repro (bare AITER call outside
vLLM) can be prepared on request.