[Bug] FP8 block-quant loader rejects artifacts using 'weight_scale' rather than 'weight_scale_inv' naming

[pasta-paul]

[Bug] FP8 block-quant loader rejects artifacts whose safetensors save weight_scale rather than weight_scale_inv

Summary

A class of compressed-tensors-quantized artifacts saves FP8 block-quant scale tensors under the attribute name weight_scale (no _inv suffix), with mathematically identical content to the weight_scale_inv form vLLM's FP8 block-quant loader expects. The loader crashes with:

AttributeError: 'MergedColumnParallelLinear' object has no attribute
'weight_scale_inv'. Did you mean: 'weight_scale'?

The crash site (vllm/model_executor/kernels/linear/scaled_mm/marlin.py:73 and marlin_utils_fp8.py:106) accesses layer.weight_scale_inv directly. The DeepseekV4 weight renaming mapper (vllm/models/deepseek_v4/nvidia/model.py:1511) only renames .scale → .weight_scale_inv; it does not handle the case where the artifact already uses the longer name .weight_scale.

A defensive getattr(layer, "weight_scale_inv", layer.weight_scale) fallback would accept both naming conventions transparently.

Reproducer

canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP (DeepSeek-V4-Flash, W4A16 routed experts + FP8 block 128×128 attention + MTP draft head). Safetensors built by llmcompressor's model_free_ptq path which produces keys named <module>.weight_scale instead of <module>.weight_scale_inv.

from safetensors import safe_open
with safe_open(".../layers.0.attn.wkv.weight", framework="pt") as f:
    t = f.get_tensor("layers.0.attn.wkv.weight")
    print(t.dtype, t.shape)         # torch.float8_e4m3fn, (512, 4096)
    s = f.get_tensor("layers.0.attn.wkv.weight_scale")  # NOT weight_scale_inv
    print(s.dtype, s.shape)         # torch.bfloat16, (4, 32) — FP8 block 128×128 scales

The mathematical content is identical to the canonical form — the block-scaled FP8 weight reconstruction is weight * scale.repeat_interleave(128, dim=0).repeat_interleave(128, dim=1). Only the attribute name differs from the loader's expectation.

Why this matters

llmcompressor's newer model_free_ptq path (which bypasses the PreTrainedModel integration step and writes safetensors directly) emits this naming. Any downstream artifact built with that path hits the loader crash, even though the math is correct and the artifact was previously usable on older vLLM builds that were tolerant of the naming.

This is a class of bug, not a one-off. We've confirmed:

• canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP (post-dequant shipping fix)
• The original (pre-dequant) FP8 attention in the same artifact also exhibits this — all 33,239 FP8/W4A16 scale tensors use weight_scale (no _inv)
• canada-quant/DeepSeek-V4-Flash-W4A16-FP8 (Card A) has the same llmcompressor-produced naming

Proposed fix

Two-line defensive patch:

--- a/vllm/model_executor/kernels/linear/scaled_mm/marlin.py
+++ b/vllm/model_executor/kernels/linear/scaled_mm/marlin.py
@@ -70,11 +70,13 @@ class CompressedTensorsW8A8Fp8MarlinScaledMMLinearKernel(...):
     def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
         if self.block_quant:
+            # Accept either `weight_scale_inv` (canonical) or `weight_scale`
+            # (llmcompressor model_free_ptq naming). Math is identical.
+            scale = getattr(layer, "weight_scale_inv",
+                            getattr(layer, "weight_scale", None))
             weight, weight_scale_inv = process_fp8_weight_block_strategy(
-                layer.weight, layer.weight_scale_inv
+                layer.weight, scale
             )
             # Update layer with new values
             replace_parameter(layer, "weight", weight.data)
-            replace_parameter(layer, "weight_scale_inv", weight_scale_inv.data)
+            # Always register the result under `weight_scale_inv` so downstream
+            # forward-path code (Dynamo, BlockScaledMMLinearKernel) finds it.
+            if hasattr(layer, "weight_scale_inv"):
+                replace_parameter(layer, "weight_scale_inv", weight_scale_inv.data)
+            else:
+                layer.register_parameter(
+                    "weight_scale_inv",
+                    torch.nn.Parameter(weight_scale_inv.data, requires_grad=False),
+                )

The same pattern likely applies to marlin_utils_fp8.py:106 (prepare_fp8_layer_for_marlin) and possibly to the DeepseekV4 renaming mapper in vllm/models/deepseek_v4/nvidia/model.py:1511 — we can extend the proposal to those sites in the same PR.

Open question for kylesayrs

Is weight_scale vs weight_scale_inv a deliberate semantic distinction (e.g., weight_scale_inv is the multiplicative inverse used during dequant fastpath, vs weight_scale for divide-by-scale dequant), or are they fully interchangeable in the FP8 block-quant path? The process_fp8_weight_block_strategy function appears to treat them identically, but if there's a subtle difference, the right fix is the source quantization step naming the attribute consistently, not the loader fallback.

We're happy to extend this PR (or split into a child PR for the model-mapper) once that's clarified.

Cross-references

• canada-quant repo audit: https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp/blob/main/docs/findings/cardd_marlin_patches_built_artifact_blocker_2026_05_25.md
• This sat alongside our #40923 comment and #36889 reopen comment — same artifact, sibling bugs in the same Marlin path on SM 12.0.

cc @kylesayrs (compressed-tensors maintainer)

[jshaofa-ui]

Solution: vLLM #43564 — FP8 block-quant loader rejects 'weight_scale' naming

Issue Summary

FP8 block-quant loader crashes with AttributeError: 'MergedColumnParallelLinear' object has no attribute 'weight_scale_inv' when loading artifacts where scale tensors are named weight_scale instead of weight_scale_inv. The mathematical content is identical — only the attribute name differs. Affects artifacts built with llmcompressor's model_free_ptq path.

Root Cause Analysis

The crash occurs in two locations:

1. vllm/model_executor/kernels/linear/scaled_mm/marlin.py:73 — process_weights_after_loading accesses layer.weight_scale_inv
2. vllm/model_executor/kernels/linear/scaled_mm/marlin_utils_fp8.py:106 — prepare_fp8_layer_for_marlin accesses layer.weight_scale_inv

The DeepseekV4 weight renaming mapper (vllm/models/deepseek_v4/nvidia/model.py:1511) only renames .scale → .weight_scale_inv. It does not handle the case where the artifact already uses .weight_scale (no _inv suffix).

The llmcompressor library's model_free_ptq path produces safetensors with weight_scale naming, which is mathematically identical to weight_scale_inv but uses a different attribute name.

Proposed Fix

Fix 1: Defensive getattr fallback in marlin.py (primary fix)

# vllm/model_executor/kernels/linear/scaled_mm/marlin.py

class CompressedTensorsW8A8Fp8MarlinScaledMMLinearKernel(...):
    def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
        if self.block_quant:
            # Accept either `weight_scale_inv` (canonical) or `weight_scale`
            # (llmcompressor model_free_ptq naming). Math is identical.
            scale = getattr(layer, "weight_scale_inv",
                            getattr(layer, "weight_scale", None))
            if scale is None:
                raise ValueError(
                    f"Layer {layer.__class__.__name__} has neither "
                    f"'weight_scale_inv' nor 'weight_scale'. "
                    f"Available attributes: {[a for a in dir(layer) if 'scale' in a.lower()]}"
                )
            
            weight, weight_scale_inv = process_fp8_weight_block_strategy(
                layer.weight, scale
            )
            # Update layer with new values
            replace_parameter(layer, "weight", weight.data)
            # Always register the result under `weight_scale_inv` so downstream
            # forward-path code (Dynamo, BlockScaledMMLinearKernel) finds it.
            if hasattr(layer, "weight_scale_inv"):
                replace_parameter(layer, "weight_scale_inv", weight_scale_inv.data)
            else:
                layer.register_parameter(
                    "weight_scale_inv",
                    torch.nn.Parameter(weight_scale_inv.data, requires_grad=False),
                )

Fix 2: Same pattern in marlin_utils_fp8.py

# vllm/model_executor/kernels/linear/scaled_mm/marlin_utils_fp8.py

def prepare_fp8_layer_for_marlin(
    layer: torch.nn.Module,
    ...
) -> None:
    # Before:
    # weight_scale_inv = layer.weight_scale_inv
    
    # After:
    weight_scale_inv = getattr(
        layer, "weight_scale_inv",
        getattr(layer, "weight_scale", None)
    )
    if weight_scale_inv is None:
        raise ValueError(
            f"Layer {layer.__class__.__name__} missing scale tensor. "
            f"Expected 'weight_scale_inv' or 'weight_scale'."
        )
    
    # Rest of function unchanged...

Fix 3: Extend DeepseekV4 renaming mapper

# vllm/models/deepseek_v4/nvidia/model.py

# Current renaming (only handles .scale → .weight_scale_inv)
RENAME_MAP = {
    ".scale": ".weight_scale_inv",
}

# Extended renaming (also handles .weight_scale → .weight_scale_inv)
RENAME_MAP = {
    ".scale": ".weight_scale_inv",
    ".weight_scale": ".weight_scale_inv",  # NEW: llmcompressor naming
}

def deepseek_v4_weight_mapping(
    name: str,
    ...
) -> str:
    for old_suffix, new_suffix in RENAME_MAP.items():
        if name.endswith(old_suffix):
            return name[:-len(old_suffix)] + new_suffix
    return name

Fix 4: Weight loader level normalization (most comprehensive)

# In the weight loader (safetensors loader or similar)
def normalize_fp8_scale_names(
    state_dict: dict[str, torch.Tensor],
) -> dict[str, torch.Tensor]:
    """Normalize FP8 scale tensor names for compatibility.
    
    Handles artifacts that use 'weight_scale' instead of 'weight_scale_inv'.
    The mathematical content is identical — only the name differs.
    """
    normalized = {}
    for key, tensor in state_dict.items():
        if key.endswith(".weight_scale"):
            # Rename to canonical form
            canonical_key = key[:-len(".weight_scale")] + ".weight_scale_inv"
            normalized[canonical_key] = tensor
            logger.debug(
                f"Renamed FP8 scale tensor: {key} → {canonical_key}"
            )
        else:
            normalized[key] = tensor
    return normalized

# Apply during model loading
def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]) -> None:
    # Normalize scale names before loading
    normalized_weights = normalize_fp8_scale_names(dict(weights))
    # Continue with normal loading...

Recommended Approach

Immediate: Fix 1 + Fix 2 (defensive getattr fallback) — 2-line changes, fixes the crash
Short-term: Fix 3 (extend renaming mapper) — handles the specific DeepseekV4 case
Long-term: Fix 4 (loader-level normalization) — handles all artifacts at the source

Test Plan

1. weight_scale artifact: Load canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP → should not crash
2. weight_scale_inv artifact: Load canonical artifact → should work as before (regression test)
3. Missing scale: Load artifact with neither weight_scale nor weight_scale_inv → should give clear error
4. Both present: Load artifact with both names → should prefer weight_scale_inv (canonical)
5. Marlin kernel correctness: Verify inference output is identical regardless of naming convention

Impact Assessment

• Severity: P1 — complete loader crash for a class of artifacts
• Affected users: Anyone loading llmcompressor-produced FP8 artifacts (DeepSeek-V4, etc.)
• Fix complexity: Very low — 2-3 lines per file
• Risk: Minimal — defensive fallback, backward compatible

Submission

Post as comment on: #43564

[pasta-paul]

Update (2026-05-26): tried both unblock paths today; neither is sufficient. There's a third layer to this bug:

Attempt 1: Renamed .weight_scale → .weight_scale_inv in safetensors via header-only rewrite (215 keys, 21 min). Result: load failed earlier with KeyError: 'layers.0.attn.fused_wqa_wkv.weight_scale_inv'. The model's fused MergedColumnParallelLinear registers its scale parameter as weight_scale in params_dict, not weight_scale_inv. So the rename was the wrong direction — the artifact's .weight_scale naming matches the model's params_dict.

Attempt 2: Applied a defensive getattr(layer, "weight_scale_inv", layer.weight_scale) patch at scaled_mm/marlin.py:71-77 + register_parameter("weight_scale_inv", ...) to alias both names. Result: model loaded past process_weights_after_loading for the first time on current vLLM. But the forward pass crashed:

RuntimeError: b_scales dim 1 = 32 is not size_n = 1536

The kernel called expects per-channel scales of shape (1536,) (TP=4 partition of 6144), but receives the block-shape (4, 32) from FP8 block 128×128 quantization. Some forward-path kernel-dispatch code routes the FP8 block layer to a per-channel kernel rather than to a block-aware one — likely because the dispatch logic distinguishes "block-quant" vs "per-channel-quant" by checking for weight_scale_inv (present after our patch's alias) vs weight_scale (per-channel).

This is the third layer of the bug. Fix needs to be deeper than either a safetensors rename or a single getattr patch — possibly:

• Make the FP8 block-quant scheme always register the scale as weight_scale_inv regardless of the source artifact's naming, OR
• Audit every kernel-dispatch site that distinguishes block vs per-channel by attribute name, harden each with hasattr checks.

marlin_utils_fp8.py already has the right pattern at lines 138-141 and 190-193 (if "weight_scale" in dir(layer): ... elif "weight_scale_inv" in dir(layer): ...), but the specific dispatch site we hit is elsewhere — couldn't locate it in ~30 min of tracing. Stack trace pointed at vllm/v1/executor/multiproc_executor.py:962 calling through torch._ops.OpOverload.__call__ so the kernel is registered via the op dispatch system.

@kylesayrs — would you happen to know which kernel-dispatch site this falls into? If you can point at the file, we'll happily PR a fix with a real repro artifact.

Reproducer remains the same: canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP (or for a fresh dequant'd state, the model.safetensors.index.json after running scripts/rename_weight_scale_to_inv.py — though that rename goes the wrong direction per today's finding). Full diagnosis in docs/findings/cardd_deeper_kernel_dispatch_blocker_2026_05_26.md.

[pasta-paul]

Cross-link: #43655 (supersedes #43515) addresses the load-time half of this bug-class — the quant_config=None plumbing into compressor.fused_wkv_wgate and indexer.weights_proj plus a conditional dispatch fix for the two raw torch.mm sites at attention.py:375/:392. From our testing today both fixes are necessary; either alone is incomplete. With #43655 merged, the load-time failures we hit on this issue collapse to just the runtime kernel-dispatch question — which is the residual scope of this issue and is independent of #43655.

Posted a downstream-confirmation comment on the PR with our artifact key-by-key evidence: #43655 (comment).

[pasta-paul]

Status update + protocol correction (2026-05-26)

Two corrections to my earlier filings on this issue:

1. AI-disclosure (per AGENTS.md): This issue and its associated canada-quant comments were drafted with AI assistance (Claude Opus). All technical claims have been verified by hand against the source code at jasl/vllm@a02a3778f and against our published HuggingFace artifacts; the human submitter (@pasta-paul / canada-quant) understands and defends every claim. Adding this disclosure retroactively per the policy at https://github.com/vllm-project/vllm/blob/main/AGENTS.md#accountability.

2. Reviewer re-routing: I originally pinged @kylesayrs (compressed-tensors maintainer at llm-compressor), but per .github/CODEOWNERS the vLLM-side quantization owners are @mgoin, @robertgshaw2-redhat, @tlrmchlsmth, @yewentao256, @pavanimajety, @zyongye. @pavanimajety in particular has been actively shipping DSv4 + SM120 work (e.g. merged #42209 NVFP4 MoE for DSv4, #40082 FlashInfer b12x for SM120/121) so this issue is squarely in her current scope. cc @mgoin @pavanimajety @robertgshaw2-redhat.

Significant upstream movement we'd missed (also today)

A re-scan of vllm-project/vllm main since 2026-05-15 shows the following relevant PRs already merged:

Merged PR	Affects our scope?
#42209 Add NVFP4 MoE support for DSv4	Yes — Card B's NVFP4-MTP path may now serve cleanly on upstream main without jasl/vllm
#40082 FlashInfer b12x MoE + FP4 GEMM for SM120/121	Yes — adds the canonical SM_120 MoE path; complements/alternative to Marlin
#43554 Remove NormGateLinear (fused into mhc_pre via #43474)	Yes — our local packed_modules_mapping patch was on NormGateLinear; may no longer be needed
#43149 Extract DSv4 sparse MLA impl into model folder	Refactor — confirms model-folder is canonical
#42855 Fix DSV4 Base swiglu limit in FP8 path	Pro variant; not Flash, but adjacent

This shifts the recommendation for downstream operators significantly. Where last week we suggested jasl/vllm@a02a3778f as the SM_120 base build, today's vllm-project/vllm@main + #43655 (open, mergeable=CONFLICTING) may be the cleaner path — many of the patches that motivated the jasl fork have landed upstream.

Updated remaining scope for this issue

After accounting for the merges above, the bug-class this issue tracks reduces to:

1. Load-time weight_scale vs weight_scale_inv naming — fixable by either renaming in safetensors or by a defensive getattr fallback at scaled_mm/marlin.py:73 and compressed_tensors_w8a8_fp8.py:163. #43655 plumbs quant_config into compressor.fused_wkv_wgate + indexer.weights_proj which partially addresses this; the marlin-side getattr defensiveness still isn't covered there.
2. Forward-time kernel-dispatch decision — even after load succeeds, FP8 block layers register weight_scale_inv but some kernel-dispatch site distinguishes block-quant vs per-channel by attribute name and routes to a per-channel kernel that fails the shape check (b_scales dim 1 = 32 is not size_n = 1536). This site is downstream of process_weights_after_loading and we haven't located it precisely; help from a quantization maintainer would shortcut the search.
3. kylesayrs' design at #41276 takes the opposite position — ignored_layers += ["weights_proj", "fused_wkv_wgate"] plus NotImplementedError if those modules are quantized, citing accuracy concerns. There's a substantive maintainer-direction question about whether DSv4 attention compressor/indexer SHOULD be quantizable. We left a comment on [WIP] [DSV4] Quantization Support #41276 at [WIP] [DSV4] Quantization Support #41276 (comment) asking for guidance.

If kylesayrs' direction prevails: canada-quant artifacts need to be re-calibrated with these layers ignored (one-shot fix, no further vLLM changes). If #43655's direction prevails: this issue's residual scope is the kernel-dispatch fix in (2).

Happy to drive whichever direction once the design question is resolved.