guidance load models as int8?

[silvacarl2]

how do i call guidance but laod the models as int8 so i can fit them on even an 80Gb GPU?

[andysalerno]

I know this doesn't answer your question about int8, but might still be helpful-

I got guidance working with GPTQ, allowing you to use 4bit quantized models. It was pretty easy, I just took the model-loading code from GPTQ and then wrote a small subclass of guidance.llms._transformers.Transformers that uses it.

Here's the raw code. It's quite hacky, and can certainly be improved, but for experimenting it does the job. I wrote more about it here.

[QuangBK]

To reduce the VRAM, you can use GPTQ-for-LLaMa.
After you load a model with Huggingface Transformer libary, just use this:

llama = guidance.llms.Transformers(model=model, tokenizer=tokenizer, device=0)
guidance.llm = llama

You may check my code to load the wizard-mega-13B-GPTQ. Hope this may help you!