Checkpoint conversion

[MaxiBoether]

Hey,

I am trying to evaluate a model trained with torchtitan using the lm eval harness. I am using the VLLM backend. Is there any straightforward way to convert a torchtitan model in the pytorch .pt format to, e.g., a huggingface model to be used in VLLM/lm eval harness? Within the torchtune repo, I was able to find some code for VLMs, but (a) that seems to be hardcoded for LLMs, (b) uses a new inference backend instead of e.g. relying on VLLM, and (c) I feel like there might be an easy way to convert torchtitan checkpoints rather than coming up with such an involved solution.

How did you evaluate downstream task accuracy with torchtitan models?

Thank you very much for your help.

[fegin]

When you save PyTorch format, is that full tensor (non-DTensor) and is saved with torch.save()?

[MaxiBoether]

I mean what the checkpoint.md docs of torchtitan call dcp_to_torch. So the .pt file generated from that, which is not - at least natively - supported by hf.

[jaysonfrancis]

You can take take a look at convert_llama_weights_to_hf.py

[tianyu-l]

@MaxiBoether curious how things are going on the eval experience?

In terms of torchtune's current eval solution:

(b) uses a new inference backend instead of e.g. relying on VLLM

It seems to be me that the inference is just running the forward of a model, although parallelisms are not supported.

(a) that seems to be hardcoded for LLMs

This seems the suggested way of supporting lm_eval with a new model, according to https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md#interface

Yeah the alternative is to convert it to HF and then use vLLM. But when the model is large, this solution is not efficient as

1. the checkpoint conversion would take long time
2. HF / vLLM doesn't support all parallelisms torchtitan support

Are they not an issue for you?