How do you verify that integration is equivalent to original Stable Diffusion?

[gordicaleksa]

Hi there HF folks!

Potentially silly question, but I went through the commit history behind the stable diffusion integration and I'm wondering: how do you make sure that you keep the equivalence with the original code?

Obviously the weights are the same but some of the code around the forward props can introduce differences?

If you can point me to a document on your process in general - curious to learn more.

[patil-suraj]

Hey @gordicaleksa !

To test the equivalence, we first tested the individual models like the unet and vae to see if the forward pass of both the version match upto a certain tolerance (1e-3 or 1e-4).

To test the pipelines, we run the original text2img.py script and then also run the StableDiffusionPipeline with the same inputs and compare if the results match. We have added integration tests that make sure that results match https://github.com/huggingface/diffusers/blob/main/tests/test_modeling_utils.py#L867.

You can also go through this PR #182, where @patrickvonplaten has posted the process of how made sure the generations match.

Let us know if this helps.

[patrickvonplaten]

@gordicaleksa If you run the code of #182 you can see that output images are identical for both PNDM (PLMS) and DDIM

[gordicaleksa]

Hi @patil-suraj!

Thanks for the quick reply, I see, makes sense, cool!

Maybe a small nit: I see you're testing the v1 version but v4 is what most people will end up using.
Could potentially matter. :)

[gordicaleksa]

Thanks @patrickvonplaten!

[patrickvonplaten]

We've tested the v1-3 version :-) But there is no reason that v1-4 would give different results given that it's 1-to-1 the same model architecture

[gordicaleksa]

Hm, I saw v1-1 in the integration tests here.

Yup yup makes sense :)

[patrickvonplaten]

Closing now :-)