[Tracking issue]: TurboQuant/HIGGS Attention follow-ups

[mgoin]

Tracking follow-up work on the TurboQuant/HIGGS KV cache attention backend initially landed in #38479.

Backend coverage

• Expand flash_attn_varlen_func to FA3/4, not just FA2
• Hybrid attention models (e.g. Qwen3.5, mamba+attention, interleaved SWA?)
• MLA support (through a new attention backend?)

Accuracy

• Long-context evals across presets (k8v4, t4nc, k3v4nc, t3nc): RULER, NIAH at 32K–1M, LongBench
• Per-layer sensitivity sweep to inform --kv-cache-dtype-skip-layers defaults
• Publish recommended config table (quality vs. compression vs. throughput) based on eval results
• Add new presets as the sweeps suggest (e.g. mixed-bit, per-layer schedules)

Feature compatibility

Things currently disabled or unverified with the TurboQuant backend; enable and test:

• Speculative decoding / Eagle
• KV connector / disaggregated serving (NIXL, LMCache, Mooncake)

Performance

• CUDA/cutedsl kernels to replace the triton kernels
• Validate AMD performance
• Revisit stream-overlap gating under CUDAGraph
• FP8 decode path parity on Hopper

cc @vibhavagarwal5

[gaby]

@mgoin @vibhavagarwal5 It would be good to know which models even support this.

I brought testing with modern models multiple times in #38479, but never got a straight answer.

Right now, most modern models are Hybrid/MoE, and they all fail to start when using the new KV quantization methods. Tested this using the nightly Docker image.

TurboQuant Related PRs:

• feat [Feature] TurboQuant: support hybrid models and uniform quantization #39931
• fix [Bugfix] turboquant FP8 store crash with bf16 models on Ampere GPUs #39908
• fix [KV Cache][TurboQuant] Add official 3-bit and 4-bit grouped TurboQuant modes #39890
• fix [Bugfix] Fix turboquant FP8 cast failure for BF16 models on Ampere GPUs #39988
• fix [Bugfix] Fix TurboQuant KV cache index-out-of-bounds in Triton decode kernel #40074
• fix fix: handle non-divisible page sizes in hybrid model KV cache unification #40128
• fix Fix TURBOQUANT backend selection in cuda.py #40060
• fix fix(kv-cache): allow TurboQuant on hybrid models #41123
• fix [Bugfix][Spec-Decode] TurboQuant K+1 spec-verify routing (fixes #40880) #40914
• fix [Attention][TurboQuant] Share dequant buffers, eliminate float16_copy #40941

[vibhavagarwal5]

Adding these to the list for tracking

• Support TurboQuant for YOCO + sliding-window models (e.g., Gemma 4 E4B) #40108
• [TurboQuant] enable FA3/FA4 for prefill paths #40092

[BowenBao]

Thanks @mgoin for this tracking issue.

On ROCm side

• [ROCm] Fix TurboQuant on ROCm: backend routing, flash-attn compat, int64 overflow #39953

Generic

• Enable TurboQuant for GPT-OSS #40663 (draft, depends on a version of #39931)

[bhoomit]

Re: Performance — #40655 fixes a critical memory issue where per-layer TQ decode buffer allocation causes OOM on single-GPU setups.

TL;DR: register_buffer() allocates scratch buffers per-layer (O(num_layers)), but layers execute sequentially so one shared set suffices. For Qwen3-32B + TQ k8v4 on H200: model load drops from 123 GiB → 62 GiB, baseline OOMs → 588K KV tokens at 52.7 tok/s.

[gaby]

Now that vllm 0.20.0 was release, it is not documented which models are supported. Every model I have tried so far fails to start.

[noonghunna]

Adding concrete data for two of the unchecked items in the feature compatibility + backend coverage lists: filed #40807 — TurboQuant + spec-dec + chunked-prefill crashes CUDA graph capture at turboquant_attn.py:570 (query_start_loc.tolist()). Reproduces on Qwen3-Next hybrid dense (Qwen3.6-27B) on Ampere. Workaround repo +bench data: github.com/noonghunna/qwen36-27b-single-3090

[noonghunna]

Reporting a concrete instance of the two unchecked boxes under 'Feature compatibility' and 'Backend coverage' (hybrid attention + spec-decode) failing in combination: #40831. MTP × TurboQuant on Qwen3-Next hybrid (DeltaNet+attn) produces degenerate token loops on tool calls, long-context recall, and streaming, while either feature alone works correctly. Full isolation matrix + probe logs in the linked issue.

[shanyulu]

Hi, I'd like to work on the MLA support item for the TurboQuant/HIGGS KV cache backend.

My proposed first slice is intentionally narrow:

1. CUDA-only first.
2. DeepSeek-style MLA only.
3. Quantize only the latent KV path, e.g. kv_c / ctkv.
4. Keep k_pe / RoPE-related components unquantized for numerical stability.
5. Start with a compatibility/repro test that shows current MLA + turboquant_k8v4 behavior.
6. Then add a minimal experimental backend path or guard, depending on maintainers' preference.

Does this match the intended direction for the “MLA support” item in this tracker, or would maintainers prefer this to be designed as a new MLA-specific attention backend?

[gaby]

@mgoin @vibhavagarwal5 Can you take a look at #39988