DeepSeek-Coder v1 tokenizer produces incorrect output on transformers v5+ (gap in PR #44801's fix)

[SuryanshSS1011]

System Info

• transformers version: 5.10.2
• tokenizers version: 0.22.2
• Python version: 3.11.2
• Platform: Linux (Penn State HPC, RHEL 8.10)
• PyTorch version: not relevant (tokenizer-only repro)

Who can help?

@ArthurZucker @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("deepseek-ai/deepseek-coder-1.3b-instruct")
print(f"tokenizer class: {type(tok).__name__}")
 
for s in ["How are you doing?",
          "def fib(n):\n    if n < 2:\n        return n",
          "Hello, world! 1234",
          "   leading spaces"]:
    ids = tok.encode(s, add_special_tokens=False)
    decoded = tok.decode(ids)
    print(f"  match={decoded == s!s:<5}  input={s!r}  decoded={decoded!r}")

Output on transformers 5.10.2 + tokenizers 0.22.2:

tokenizer class: LlamaTokenizer
    match=False  input='How are you doing?'         decoded='Howareyoudoing?'
    match=False  input='def fib(n):\n    if n < ... decoded='deffib(n):ifn<2:returnn'
    match=False  input='Hello, world! 1234'         decoded='Hello,world!1234'
    match=False  input='   leading spaces'          decoded='leadingspaces'

All whitespace (spaces, newlines, tabs) is stripped on the encode → decode round-trip.

Root cause

PR #44801 (closes #44779) addressed this for DeepSeek-R1 / V2 / V3 / V4 / VL / OCR by adding them to MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS in src/transformers/models/auto/tokenization_auto.py. That set is keyed on model_type.

DeepSeek-Coder v1 (1.3B / 6.7B / 33B Instruct) declares model_type: llama in its config.json and tokenizer_class: LlamaTokenizerFast in tokenizer_config.json, so it falls through to LlamaTokenizer. But its tokenizer.json has the same ByteLevel-BPE pipeline as the other DeepSeek variants:

Model	normalizer	pre_tokenizer subtypes	decoder	model
deepseek-coder-1.3b-instruct	Sequence (empty)	Split×4, Digits, ByteLevel	ByteLevel	BPE
DeepSeek-Coder-V2-Lite-Instruct	Sequence (empty)	Split×5, Digits, ByteLevel	ByteLevel	BPE
DeepSeek-R1	Sequence (empty)	Split×3, ByteLevel	ByteLevel	BPE

LlamaTokenizer is SentencePiece-based and cannot consume a ByteLevel-BPE tokenizer.json correctly, and hence the stripped whitespace.

Why the existing fix doesn't trivially extend

Adding llama to MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS would force all llama-typed models onto TokenizersBackend, including genuine Llama models that should use LlamaTokenizer. So the fix needs a different shape.

Two possible directions:

1. Per-checkpoint override: explicit list of known-broken (model_id, expected_backend) tuples. This is narrow and surgical, but the list needs maintenance as new checkpoints land.
2. Pipeline-shape detection: when loading, if the resolved tokenizer class is SentencePiece-based but tokenizer.json has model.type=BPE + decoder.type=ByteLevel, force TokenizersBackend. More robust, slightly more code.
   Happy to implement either once maintainers confirm which is preferred.

Expected behavior

tok.decode(tok.encode(s)) should round-trip whitespace, as it does on transformers v4 and on the other DeepSeek variants in v5+ (DeepSeek-R1, DeepSeek-Coder-V2-Lite-Instruct — both load as TokenizersBackend and round-trip cleanly).

This is a tokenization-correctness bug, not a UX one — downstream tasks that depend on exact-token round-trip (code-generation evaluation, edit-distance against ground-truth, RL reward signals on tokenized output) silently produce wrong values.

[SuryanshSS1011]

Coverage update

I ran a broader probe across DeepSeek model families on transformers 5.10.2; same encode → decode round-trip test.

Results:

Same model_type: llama mechanism, expanded scope:

Checkpoint	Result
deepseek-coder-1.3b-{base,instruct}	3/3 mismatches
deepseek-coder-6.7b-{base,instruct}	3/3 mismatches
deepseek-coder-33b-{base,instruct}	3/3 mismatches
deepseek-llm-7b-{base,chat}	3/3 mismatches

All 8 declare model_type: llama and load as LlamaTokenizer. The original report's scope is "all DeepSeek-Coder v1 sizes + DeepSeek-LLM v1," not just deepseek-coder-1.3b-instruct.

Confirmed working (sanity baseline):

DeepSeek-V3.1, deepseek-vl2-tiny, DeepSeek-Coder-V2-Lite-Instruct, DeepSeek-V3 — all TokenizersBackend, 0/3 mismatches.

Possibly-related, different mechanism:

DeepSeek-VL-7b-chat is also broken (3/3 mismatches, loads as LlamaTokenizer), but via model_type: multi_modality rather than llama. The multi_modality model_type isn't in MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS. Same end-state, different routing. Happy to file separately if you'd prefer to keep that scoped distinct from this one, let me know.

[itazap]

will add these here! https://github.com/huggingface/transformers/pull/46091/changes#diff-d804e851851cdebeb8048938f1f8beec1cfa78bf7b1f06af86faa450f9d18defR408

[kpal002]

Opened PR #46580 with a fix.

Root cause: DeepSeek-Coder v1 / DeepSeek-LLM v1 checkpoints declare model_type: llama and tokenizer_class: LlamaTokenizerFast in their Hub config, but their tokenizer.json is actually a ByteLevel-BPE tokenizer (same architecture as TokenizersBackend). The existing MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS mechanism can't help here: it fires only when the registered class mismatches the hub class — but after stripping "Fast", LlamaTokenizer == LlamaTokenizer, so the check silently passes and the model falls through to LlamaTokenizer (SentencePiece-based), which strips all whitespace.

Adding "llama" to that set would break every genuine Llama checkpoint, so the fix introduces a separate CHECKPOINT_PREFIXES_WITH_INCORRECT_HUB_TOKENIZER_CLASS frozenset keyed on Hub checkpoint name prefixes (deepseek-ai/deepseek-coder-, deepseek-ai/deepseek-llm-). An early check in AutoTokenizer.from_pretrained routes any matching checkpoint directly to TokenizersBackend before the model_type lookup runs.

Also fixes deepseek-ai/deepseek-vl-7b-{base,chat} (model_type: multi_modality) by adding "multi_modality" to MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS.

Affected checkpoints:

Checkpoint	Was	Now
deepseek-ai/deepseek-coder-{1.3b,6.7b,33b}-{base,instruct}	LlamaTokenizer ❌	TokenizersBackend ✅
deepseek-ai/deepseek-llm-7b-{base,chat}	LlamaTokenizer ❌	TokenizersBackend ✅
deepseek-ai/deepseek-vl-7b-{base,chat}	LlamaTokenizer ❌	TokenizersBackend ✅