-
Notifications
You must be signed in to change notification settings - Fork 461
Open
Labels
Description
Hey,
I am trying to evaluate a model trained with torchtitan using the lm eval harness. I am using the VLLM backend. Is there any straightforward way to convert a torchtitan model in the pytorch .pt format to, e.g., a huggingface model to be used in VLLM/lm eval harness? Within the torchtune repo, I was able to find some code for VLMs, but (a) that seems to be hardcoded for LLMs, (b) uses a new inference backend instead of e.g. relying on VLLM, and (c) I feel like there might be an easy way to convert torchtitan checkpoints rather than coming up with such an involved solution.
How did you evaluate downstream task accuracy with torchtitan models?
Thank you very much for your help.