Community contribution: Adding GGUF support for more architectures

[SunMarc]

Feature request

Recently, we have added the ability to load gguf files within transformers.

The goal was to offer the possibility to users to further train/fine-tune their gguf models.

See Workflow

1) Load gguf file in transformers: we dequantize the weights to fp32, then we load the weights to be used with PyTorch.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
filename = "tinyllama-1.1b-chat-v1.0.Q6_K.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

2. train/finetune

3. Convert the model back to gguf to use in the ggml ecosystem using convert_hf_to_gguf script or using gguf-my-repo space if you pushed your model on the hub :

tokenizer.save_pretrained('directory')
model.save_pretrained('directory')

!python ${path_to_llama_cpp}/convert-hf-to-gguf.py ${directory}

Let's try to add GGUF support for more architectures! Currently supported architectures are

• Llama
• Mistral
• Qwen2

It would be great to add the support for more architectures such as

• Phi3 Add support for GGUF Phi-3 #31844
• Qwen2Moe Add Qwen2Moe GGUF loading support  #33264
• Gemma2
• T5 Add T5 GGUF loading support #33389
• Falcon Add falcon gguf #33437
• Bloom Add gguf support for bloom #33473
• StableLM Add gguf support for StableLM #33793
• gpt2 Add gguf support for gpt2 #34044
• starcoder2 Add GGUF for starcoder2 #34094
• llama4
• Deepseekv3
• c4ai-command-a

... and many more (Feel free to suggest more architectures ! The model needs to integrated in transformers)

Adding this feature would require to follow the same protocol as in this PR :

1. Update GGUF_TENSOR_MAPPING and GGUF_CONFIG_MAPPING in order to map the tensor/config of the gguf file to the one on transformers.
2. Create a GGUFXXXConverter(XXXConverter) class to convert the gguf tokenizer to a transformers one.
3. Write tests

If you are interested to take up the challenge, comment below with the architecture name you want to integrate and open a PR!

Once you open a PR, feel free to ping @SunMarc @LysandreJik @ArthurZucker for a review !

Motivation

Support for more gguf models

Your contribution

Reviewing PRs and possibly adding the support for more models

[VladOS95-cyber]

@SunMarc I am going to take Qwen2Moe

[KingNish24]

@SunMarc I want to take Gemma2

[junejae]

@SunMarc May I suggest & take T5? Seems GGUF version of T5 encoder is highly used for getting along with FLUX.

[010kim]

@SunMarc Hello! Unless someone else is working on this model already, may I take MiniCPM-V?

[SunMarc]

@SunMarc May I suggest & take T5? Seems GGUF version of T5 encoder is highly used for getting along with FLUX.

Added @junejae !

@SunMarc Hello! Unless someone else is working on this model already, may I take MiniCPM-V?

Hi @010kim, thanks for the interest ! MiniCPM-V model relies on trust_remote_code=True, so I don't think we can add this model for now with gguf support. We don't want to have code in transformers that relies on modeling files that are on the hub. I will think about extending trust_remote_code=True to gguf support, so that the author of the model can add it himself !

[010kim]

Hi @010kim, thanks for the interest ! MiniCPM-V model relies on trust_remote_code=True, so I don't think we can add this model for now with gguf support. We don't want to have code in transformers that relies on modeling files that are on the hub. I will think about extending trust_remote_code=True to gguf support, so that the author of the model can add it himself !

@SunMarc Thank you so much for your response. It also makes sense the author should work on it. What about Cohere? Can I take it?

[jungnerd]

Hi @SunMarc 👋🏻
May I work with CLIP model if nobody is working on it?

[SunMarc]

Hey @jungnerd ! The model you choose needs to be in the conversion script from hf to gguf. See the script here

[g-prz]

Hey @SunMarc 🙋‍♂️
I'd like to try my chance to contribute to this issue, can I take Falcon? 🦅

[VladOS95-cyber]

Hi @SunMarc, I take bloom if nobody is working on it