diffusion components have no 'add_weighted_adapter'

[garychan22]

Hi, I have followed the guidance https://huggingface.co/docs/diffusers/using-diffusers/merge_loras here,

pipeline = AutoPipelineForText2Image.from_pretrained(
        source_pipeline_path,
        vae=vae,
        # variant="fp16",
        torch_dtype=torch.float16
    ).to("cuda")
pipeline.to(device, torch.float16)

print("Load unet lora")
pipeline.unet = PeftModel.from_pretrained(pipeline.unet, os.path.join(lora_path, "unet"), adapter_name="unet")
print(type(pipeline.unet))  # output: PeftModel
pipeline.unet.add_weighted_adapter(["unet"], weights=[lora_scale]) 
pipeline.unet.merge_and_unload()
pipeline.unet.to(device, dtype=torch.float16)

When calling pipeline.unet.add_weighted_adapter(["unet"], weights=[lora_scale]) , the following error was encountered

UNet2DConditionModel has no 'add_weighted_adapter'

Not only the unet but also the two text_encoders.

My expectation is to set the lora scale of these components. Thanks!

---

diffusers 0.32.2
peft 0.15.2

[BenjaminBossan]

Hi, the add_weighted_adapter method only exists on PeftModel etc. You are using the transformers integration of PEFT, which does not support the method.

My expectation is to set the lora scale of these components. Thanks!

For setting the LoRA scale, add_weighted_adapter is not the intended method. Instead, check if rescale_adapter_scale is not a better option for you. It should also work with the transformers integration.

[garychan22]

Hi, the add_weighted_adapter method only exists on PeftModel etc. You are using the transformers integration of PEFT, which does not support the method.

My expectation is to set the lora scale of these components. Thanks!

For setting the LoRA scale, add_weighted_adapter is not the intended method. Instead, check if rescale_adapter_scale is not a better option for you. It should also work with the transformers integration.

Thanks, it works for LoRA layers. If I want to rescale the LoKr, how can I achieve that?

[garychan22]

I have rewrite the rescale_adapter_scale function, and it now works for the lokr. Again, thanks for the help!

[BenjaminBossan]

I have rewrite the rescale_adapter_scale function, and it now works for the lokr. Again, thanks for the help!

Great that you got it working. If you have a version of rescale_adapter_scale that works with other layer types than just LoraLayer, please feel free to create a PR and contribute it to PEFT. Other users may find it useful too.