Tokenizer issue for gemma 3 HF model

[kirklandsign]

libc++abi: terminating due to uncaught exception of type std::runtime_error: Missing pattern for PreTokenizer of type Split
Aborted

Repro:

adb shell 'cd /data/local/tmp/llama; ls; ./llama_main --model_path gemma3_8da4w.pte --tokenizer_path gemma_tokenizer.json --prompt "where is paris"'

[guangy10]

This model has both added tokens and special tokens. Here it is on HF: https://huggingface.co/google/gemma-3-1b-it/tree/main

[jackzhxng]

This is just for gemma3?

[guangy10]

This is just for gemma3?

For what we've tried (qwen3, gemma3, smollm2)so far yes just seeing this crash for gemma3. smollm is having a different issue, all generated tokens are identical.

[jackzhxng]

Yeah I can take a look at this. How urgent is this?

[guangy10]

Yeah I can take a look at this. How urgent is this?

Thanks Jack. It's not very urgent. Basically the work is to help us making better UX to run HF models and understand perf gaps between running via pybind in python and via llm runner on-device

[jackzhxng]

@kimishpatel this is fixed now right?

[jackzhxng]

Update on this:

• [hf] Add new features to HF tokenizer #87 from @larryliu0820 tried to address the issue for the huggingface tokenizer, if we specify tokenizer.json, however the output is wrong
• Loading in a tokenizer.model as a sentencepiece tokenizer also produces incorrect output