[Issue]: No guard for gfx1250 in flydsl_moe_stage1 when running DeepSeek-V4-Flash with SGLang

[akao-amd]

Problem Description

As a part of the enablement effort, we evaluate DeepSeek-V4-Flash on SGLang [PR #32754](https://github.com/sgl-project/sglang/pull/32754}. With recent AITER main branch to serve the MoE backend, we bump into the error:

ExpandIntegerOperand Op #2: t196: ch = llvm.amdgcn.raw.ptr.buffer.load.lds<(dereferenceable load (s128) from %ir.52, align 1, addrspace 8), (dereferenceable store (s4096) into %ir.108, align 1, addrspace 3)> # D:1 t169, TargetConstant:i64<3452>, t306, t193, TargetConstant:i32<16>, t177, Constant:i32<0>, TargetConstant:i32<0>, TargetConstant:i32<0>

LLVM ERROR: Do not know how to expand this operator's operand!
Fatal Python error: Aborted

Thread 0x00007a7847ffe6c0 (most recent call first):
  File "/usr/lib/python3.12/threading.py", line 359 in wait
  File "/usr/lib/python3.12/threading.py", line 655 in wait
  File "/opt/venv/lib/python3.12/site-packages/tqdm/_monitor.py", line 69 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007a79f7fff6c0 (most recent call first):
  File "/usr/lib/python3.12/threading.py", line 359 in wait
  File "/usr/lib/python3.12/threading.py", line 655 in wait
  File "/opt/venv/lib/python3.12/site-packages/tqdm/_monitor.py", line 69 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Current thread 0x00007aa6e561d300 (most recent call first):
  File "/opt/venv/lib/python3.12/site-packages/flydsl/compiler/jit_function.py", line 772 in _run_pipeline
  File "/opt/venv/lib/python3.12/site-packages/flydsl/compiler/jit_function.py", line 924 in compile
  File "/opt/venv/lib/python3.12/site-packages/flydsl/compiler/jit_function.py", line 1549 in __call__
  File "/opt/venv/lib/python3.12/site-packages/flydsl/compiler/jit_function.py", line 1661 in _compile_impl
  File "/opt/venv/lib/python3.12/site-packages/flydsl/compiler/jit_function.py", line 1717 in __call__
  File "/sgl-workspace/aiter/aiter/ops/flydsl/moe_kernels.py", line 996 in _run_compiled
  File "/sgl-workspace/aiter/aiter/ops/flydsl/moe_kernels.py", line 1741 in _flydsl_moe_stage1_impl
  File "/sgl-workspace/aiter/aiter/ops/flydsl/moe_kernels.py", line 1942 in flydsl_moe_stage1
  File "/sgl-workspace/aiter/aiter/fused_moe.py", line 1451 in _flydsl_stage1_wrapper
  File "/sgl-workspace/aiter/aiter/fused_moe.py", line 3261 in fused_moe_2stages
  File "/sgl-workspace/aiter/aiter/fused_moe.py", line 1032 in _fused_moe_impl
  File "/sgl-workspace/aiter/aiter/fused_moe.py", line 679 in fused_moe_
  File "/sgl-workspace/aiter/aiter/jit/utils/torch_guard.py", line 211 in wrapper
  File "/sgl-workspace/aiter/aiter/jit/utils/torch_guard.py", line 317 in outer_wrapper
  File "/opt/venv/lib/python3.12/site-packages/torch/_ops.py", line 1269 in __call__
  File "/sgl-workspace/aiter/aiter/jit/utils/torch_guard.py", line 294 in wrapper_custom
  File "/sgl-workspace/aiter/aiter/fused_moe.py", line 549 in fused_moe
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/aiter.py", line 198 in run
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/runner.py", line 197 in run
  File "/sgl-workspace/sglang/python/sglang/srt/layers/quantization/fp8.py", line 2440 in apply
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 1532 in run_moe_core
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 1491 in forward_impl
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 1458 in forward
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 1144 in forward_normal
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 922 in forward
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v4.py", line 2278 in _run_moe_ffn_dp_sync
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v4.py", line 2167 in forward
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790 in _call_impl
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779 in _wrapped_call_impl
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v4.py", line 3078 in forward
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v4.py", line 3282 in forward
  File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124 in decorate_context
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 1193 in run_once
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/runner_backend/full_cuda_graph_backend.py", line 110 in capture_one
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 1231 in capture_one_shape
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 1114 in _capture_one_stream
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 1045 in capture
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py", line 466 in __init__
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner_components/cuda_graph_setup.py", line 478 in capture_decode_graph
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner_components/cuda_graph_setup.py", line 184 in capture_cuda_graphs
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1011 in init_cuda_graphs
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 433 in init_cuda_graphs
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 997 in init_all_cuda_graphs
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 1014 in init_model_worker
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 524 in __init__
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 5196 in run_scheduler_process
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108 in run
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap
  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 135 in _main
  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 122 in spawn_main
  File "<string>", line 1 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, psutil._psutil_linux, pybase64._pybase64, charset_normalizer.md, charset_normalizer.cd, requests.packages.charset_normalizer.md, requests.packages.chardet.md, requests.packages.charset_normalizer.cd, requests.packages.chardet.cd, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, PIL._imaging, PIL._imagingft, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, regex._regex, yaml._yaml, markupsafe._speedups, _cffi_backend, _cyutility, scipy._cyutility, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._batched_linalg, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._internal_matfuncs, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg.cython_lapack, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpacklib, scipy.sparse.linalg._propack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial.transform._rotation_cy, scipy.spatial.transform._rigid_transform_cy, scipy.optimize._direct, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, msgspec._core, zmq.backend.cython._zmq, tvm_ffi.core, lxml._elementpath, lxml.etree, setproctitle._setproctitle, Cython.Plex.Actions, Cython.Plex.Scanners, Cython.Compiler.Scanning, hip_utils, pyarrow.lib, pandas._libs._cyutility, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.properties, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, grpc._cython.cygrpc, google._upb._message (total: 147)
[rank1]:[W825 10:26:44.052465837 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=10, addr=[localhost]:51424, remote=[::ffff:0.0.0.0]:46541): Connection reset by peer
Exception raised from recvBytes at /__w/rockrel/rockrel/external-builds/pytorch/pytorch/torch/csrc/distributed/c10d/Utils.hpp:679 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x7cdedc0f605d in /opt/venv/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x685d322 (0x7cdefae3b322 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #2: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x273 (0x7cdefae39203 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x4a5 (0x7cdedf552935 in /opt/venv/lib/python3.12/site-packages/torch/lib/libtorch_hip.so)
frame #4: <unknown function> + 0xecdb4 (0x7ce57d0c6db4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: <unknown function> + 0x9cb84 (0x7ce58063cb84 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: <unknown function> + 0x129d6c (0x7ce5806c9d6c in /lib/x86_64-linux-gnu/libc.so.6)

[rank1]:[W825 10:26:44.056376840 ProcessGroupNCCL.cpp:1828] [PG ID 0 PG GUID 0 Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Connection reset by peer
[2026-08-25 10:26:44] kill_process_tree called: parent_pid=8114, include_parent=False, pid=8114
Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/engine.py", line 1787, in _wait_for_scheduler_ready
    data = scheduler_pipe_readers[i].recv()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/sgl-workspace/sglang/python/sglang/launch_server.py", line 74, in <module>
    run_server(server_args)
  File "/sgl-workspace/sglang/python/sglang/launch_server.py", line 57, in run_server
    launch_server(server_args)
  File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/http_server.py", line 2798, in launch_server
    ) = Engine._launch_subprocesses(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/engine.py", line 1206, in _launch_subprocesses
    scheduler_init_result.wait_for_ready()
  File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/engine.py", line 910, in wait_for_ready
    infos = _wait_for_scheduler_ready(scheduler_pipe_readers, scheduler_procs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/entrypoints/engine.py", line 1789, in _wait_for_scheduler_ready
    raise _scheduler_died_error(i, scheduler_procs[i])
RuntimeError: Rank 0 scheduler died during initialization (exit code: -6). If exit code is -9 (SIGKILL), a common cause is the OS OOM killer. Run `dmesg -T | grep -i oom` to check.

To reproduce

Please download henryx/xsgl:v0.5.18-rocm7_14-mi455-20260825-pr32754-new-aiter and run recipe on it.

Recipe

SGLANG_DEFAULT_THINKING=1 AITER_GROUPED_CONTIGUOUS_TOKEN_THRESHOLD=16 SGLANG_DSV4_REASONING_EFFORT=max SGLANG_USE_ROCM700A=0 SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton AITER_BF16_FP8_MOE_BOUND=0 AITER_FORCE_A8W4=1 SGLANG_USE_AITER_MOE_GU_ITLV=0 SGLANG_OPT_FUSE_MHC_POST_PRE=0 ENABLE_CK=0 SGLANG_USE_AITER=1 AITER_GROUPED_FORCE_SPLIT_K1=1 python3 -m sglang.launch_server --model-path /data/DeepSeek-V4-Flash --trust-remote-code --tp 1 --attention-backend dsv4 --page-size 256 --mem-fraction-static 0.60 --swa-full-tokens-ratio 0.15 --disable-shared-experts-fusion --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --chunked-prefill-size 8192 --cuda-graph-max-bs 256 --max-running-requests 256 --disable-radix-cache --kv-cache-dtype fp8_e4m3

### Operating System

Ubuntu 22.04

### CPU

AMD

### GPU

MI455

### ROCm Version

ROCm 7.14

### ROCm Component

_No response_

### Steps to Reproduce

_No response_

### (Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

<details>
<summary>rocminfo --support output</summary>

Paste output here

</details>


### Additional Information

_No response_

[zufayu]

the same bugs #4986

[zufayu]

the same bugs #4987