Finetuning with GPU

[Vidit-Ostwal]

Hi Everyone!

First of all thanks for open-sourcing this project.

I had a query regarding the resource needed to finetune a Gemma-3 Model?

Till now I have tried GPU's on these platform

1. Google Colab T4-GPU (Not enough Memory)
2. Google Colab v2-8 TPU (Slow - Colab runtime gets disconnected)
3. Kaggle GPU P-100 (Not enough Memory)
4. Kaggle GPU T4*2 GPU (Currently running into issues because of using multiple GPU - will investigate this further)

I would like to know what training time are we estimating for this model to train for a single epoch (full-finetune) and what base configuration you will suggest me to use.

Thanks for the project!

[Vidit-Ostwal]

Also any other cloud services any one knows about where GPU are rented at a lower rate, then aws, gcp, azure.
Will help a lot, thanks.

[ariG23498]

We are aware of the large VRAM requirements, and would love to have a script to compute that for us. Would you be interested in building that script?

Having said that we are working on LoRA for the next release, to bring the VRAM requirements under control.

[Vidit-Ostwal]

We are aware of the large VRAM requirements, and would love to have a script to compute that for us. Would you be interested in building that script?

Having said that we are working on LoRA for the next release, to bring the VRAM requirements under control.

Yes @ariG23498, would love to contribute.
I do have surface level understanding around how to calculate this.
Still will love to learn more about this.
It will be great if you could share some resource, which can help me understanding this area better.
Thanks

[ariG23498]

Something like this might be a good starting point.

[Vidit-Ostwal]

Great, I would be in touch If I have any doubts.
Thanks @ariG23498

[sergiopaniego]

An interesting idea is to explore how different attn_implementation options affect resource usage. In this case, the "eager" implementation is recommended, but you can experiment with other available options, which typically require less VRAM.

To modify this setting, change the attn_implementation parameter when loading the model:

model = Gemma3ForConditionalGeneration.from_pretrained(
        cfg.model_id,
        torch_dtype=cfg.dtype,
        device_map="cpu",
        attn_implementation="eager",
    )

[Vidit-Ostwal]

An interesting idea is to explore how different attn_implementation options affect resource usage. In this case, the "eager" implementation is recommended, but you can experiment with other available options, which typically require less VRAM.

To modify this setting, change the attn_implementation parameter when loading the model:

model = Gemma3ForConditionalGeneration.from_pretrained(
cfg.model_id,
torch_dtype=cfg.dtype,
device_map="cpu",
attn_implementation="eager",
)

Will try with other attn_implementation,
Any thoughts on using unsloth.ai () for finetuning?
Have tried it as well, but their documentation is not helpful.

[ariG23498]

Hey @Vidit-Ostwal how are things on the benchmarking the memory front?

Let us know if you need any help. If you create a PR (even draft it) please tag this issue so that we are better positioned to review it.

Any thoughts on using unsloth.ai () for finetuning?

We currently are focusing on vanilla torch iterations and no patching of dynamic kernels. But if you want to test that out and add it to the repo, we are more than happy to help.

[Vidit-Ostwal]

Hey @Vidit-Ostwal how are things on the benchmarking the memory front?

Let us know if you need any help. If you create a PR (even draft it) please tag this issue so that we are better positioned to review it.

Any thoughts on using unsloth.ai () for finetuning?

We currently are focusing on vanilla torch iterations and no patching of dynamic kernels. But if you want to test that out and add it to the repo, we are more than happy to help.

I am on it @ariG23498 , I am trying to understand how the 'nanovlm' was made and how the above mentioned function made all the calculations.

I will add a draft PR by EOD, which will help to collaborate in a better way.

[Vidit-Ostwal]

Hey @Vidit-Ostwal how are things on the benchmarking the memory front?

Let us know if you need any help. If you create a PR (even draft it) please tag this issue so that we are better positioned to review it.

Any thoughts on using unsloth.ai () for finetuning?

We currently are focusing on vanilla torch iterations and no patching of dynamic kernels. But if you want to test that out and add it to the repo, we are more than happy to help.

@ariG23498, I am currently facing one error! Let me know how I can resolve this.

For making the VRAM calculation for the training part, I am first trying to train it with the QLORA configuration, in this PR(#13).
I am on the Kaggle and using 2*T4 GPU for the final training,

To help this with the distributed training I thought to use the accelerator library.
But I am currently facing this issues, when I am trying to prepare with accelerator

qlora_model, optimizer, train_dataloader = accelerator.prepare(
    qlora_model, optimizer, train_dataloader
)

TypeError: device() received an invalid combination of arguments - got (NoneType), but expected one of:
 * (torch.device device)
      didn't match because some of the arguments have invalid types: (!NoneType!)
 * (str type, int index = -1)

I think there is some issue with the file made after accelerate config file, but yup I am still trying to debug how to steady perform a fine-tuning on distributed GPU.

[rozeappletree]

@Vidit-Ostwal can you share your accelerate code? Hunch is we need to pass device type explicitly somewhere.

[Vidit-Ostwal]

@Vidit-Ostwal can you share your accelerate code? Hunch is we need to pass device type explicitly somewhere.

This is the code I am using at this moment

from accelerate import Accelerator
from transformers import AutoProcessor
from peft import get_peft_model, LoraConfig, prepare_model_for_kbit_training, PeftType
from transformers import BitsAndBytesConfig

cfg = Configuration()
processor = AutoProcessor.from_pretrained(cfg.model_id)
train_dataloader = get_dataloader(processor)

logger.info("Getting model & turning only attention parameters to trainable")

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = Gemma3ForConditionalGeneration.from_pretrained(
    cfg.model_id,
    torch_dtype=cfg.dtype,
    device_map="auto",  # Let accelerate handle this later
    attn_implementation="eager",
    quantization_config=bnb_config
)

model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    inference_mode=False,
    r=8,
    lora_alpha=32,
    lora_dropout=0.1,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    peft_type=PeftType.LORA,
)

qlora_model = get_peft_model(model, lora_config)
qlora_model.print_trainable_parameters()

# ✅ Use Accelerate
accelerator = Accelerator()
params_to_train = list(filter(lambda x: x.requires_grad, qlora_model.parameters()))
optimizer = torch.optim.AdamW(params_to_train, lr=cfg.learning_rate)

# Prepare model, optimizer, dataloader
qlora_model, optimizer, train_dataloader = accelerator.prepare(
    qlora_model, optimizer, train_dataloader
)

qlora_model.train()

# 🚀 Training Loop
for epoch in range(cfg.epochs):
    for batch in train_dataloader:
        outputs = qlora_model(**batch)
        loss = outputs.loss
        accelerator.backward(loss)
        optimizer.step()
        optimizer.zero_grad()
    accelerator.print(f"Epoch {epoch+1} finished")

logger.info("Train finished")

Still facing the CUDA is out of memory error

[Vidit-Ostwal]

Hi @ariG23498, any thoughts on this!

As per now, I believe that 2*T4 instances are not enough for fine-tuning 3b parameters model.
Let me know if there is any other paid service I can use to work on this further.

This is one of the bottleneck I am currently stuck.

[rozeappletree]

@Vidit-Ostwal wdym by it is not running in P100 due to memory issues?

I was able to run the train.py in main branch without any issues in Kaggle P100.

It executed until line 89

gemma3-object-detection/train.py

Line 89 in 9acb47c

project=cfg.project_name,

And threw error because the attribute was not present in config.py