FID calculation is five orders of magnitude different from the reference FID implementation

[mbchang]

🐛 Bug description

The documentation states that ignite.metrics.FID is inspired by the reference Pytorch implementation of FID here. However, when I compare ignite's calculated FID score to the reference implementation's, I see that their scores are off by about five orders of magnitude.

Here is a script based on the example in ignite's documentation.

import torch
import torchvision
import tqdm
from ignite.metrics.gan import FID

torch.manual_seed(0)

m = FID()

y_pred, y = torch.rand(100, 3, 299, 299), torch.rand(100, 3, 299, 299)
for i in tqdm.tqdm(range(len(y_pred))):
    torchvision.utils.save_image(y_pred[i], f'pred/{i}.png')
    torchvision.utils.save_image(y[i], f'gt/{i}.png')
    m.update((y_pred[i:i+1], y[i:i+1]))

print('ignite online FID', m.compute())  # 8.98434690701287e-05

m = FID()
m.update((y_pred, y))
print('ignite batch FID', m.compute())  # 8.98434072559458e-05

This snippet of code saves y_pred to a folder called pred and y to a folder called gt. I then installed pytorch-fid from the reference implementation's repo and ran

python -m pytorch_fid pred gt --num-workers 8

and the FID score I got was 5.980631998318767.

I would have expected the FID scores to be the same, or at least within numerical error. However, 5.980631998318767 and 8.98434072559458e-05 are off by 5 orders of magnitude.

Environment

• PyTorch Version (e.g., 1.4): 1.7.0
• Ignite Version (e.g., 0.3.0): 0.4.7
• OS (e.g., Linux): MacOS 10.14.6
• How you installed Ignite (conda, pip, source): pip
• Python version: 3.8.5
• Any other relevant information:

[vfdev-5]

Thanks for reporting @mbchang !
The main difference is related to the InceptionV3 model used to compute features. In ignite case, by default we are using InceptionV3 from torchvision pretrained on ImageNet. I agree that it is not stated clearly and docs should be improved.
In case of fid_pytorch cmd tool, they are using a bit different Inception V3: https://github.com/mseitzer/pytorch-fid/blob/3d604a25516746c3a4a5548c8610e99010b2c819/src/pytorch_fid/inception.py#L184-L192 and
https://github.com/mseitzer/pytorch-fid/blob/3d604a25516746c3a4a5548c8610e99010b2c819/src/pytorch_fid/fid_score.py#L252-L254

If you would like to get almost similar results you can do the following:

import torch
import torchvision
import tqdm
from ignite.metrics.gan import FID

from pytorch_fid.inception import InceptionV3

device = "cpu"
dims = 2048
block_idx = InceptionV3.BLOCK_INDEX_BY_DIM[dims]
model = InceptionV3([block_idx]).to(device)

import torch.nn as nn


class WrapperInceptionV3(nn.Module):
    
    def __init__(self, fid_incv3):
        super().__init__()
        self.fid_incv3 = fid_incv3
    
    @torch.no_grad()
    def forward(self, x):
        y = self.fid_incv3(x)
        y = y[0]
        y = y[:, :, 0, 0]
        return y

wrapper_model = WrapperInceptionV3(model)
wrapper_model.eval();

m = FID(num_features=dims, feature_extractor=wrapper_model)


torch.manual_seed(0)
y_pred, y = torch.rand(100, 3, 299, 299), torch.rand(100, 3, 299, 299)
for i in tqdm.tqdm(range(len(y_pred))):
    torchvision.utils.save_image(y_pred[i], f'pred/{i}.png')
    torchvision.utils.save_image(y[i], f'gt/{i}.png')
    m.update((y_pred[i:i+1], y[i:i+1]))

print('ignite online FID', m.compute())  # 6.109078042628951

and

python -m pytorch_fid pred gt --num-workers 8
> FID:  6.112408053607531

[mbchang]

Ah I see, thank you so much! This works for me.

[vfdev-5]

I'll reopen this issue to fix our docs and also investigate why there's still a minor difference between two implementations.

@mbchang if you have time and would like to help with that it would be awesome!

[louis-she]

I have had a look on this, the difference of the 2 implementations are coming from 2 reasons:

1. the input is not exactly the same

the code provided is not strict enough. the ignite version called torchvision.utils.save_image to convert the random 0 ~ 1 float tensor to png, and the FID official using torchvision.transform.ToTensor to convert that png to 0 ~ 1 float number.

There will be a slight different from these transforms, which cause the float input to ignite version is not the same as the float input to FID official model. We can using the following code to avoid this.

import torch
import torchvision
import tqdm
from ignite.metrics.gan import FID
from torchvision import transforms
from PIL import Image
import numpy as np

from pytorch_fid.inception import InceptionV3

device = "cuda"
dims = 2048
block_idx = InceptionV3.BLOCK_INDEX_BY_DIM[dims]
model = InceptionV3([block_idx]).to(device)

import torch.nn as nn


class WrapperInceptionV3(nn.Module):

    def __init__(self, fid_incv3):
        super().__init__()
        self.fid_incv3 = fid_incv3

    @torch.no_grad()
    def forward(self, x):
        y = self.fid_incv3(x)
        y = y[0]
        y = y[:, :, 0, 0]
        return y

wrapper_model = WrapperInceptionV3(model)
wrapper_model.eval();

m = FID(num_features=dims, feature_extractor=wrapper_model, device=device)

torch.manual_seed(0)

y_pred, y = [[Image.fromarray(np.random.randint(0, 255, (299, 299, 3), dtype=np.uint8)) for _ in range(2)] for _ in range(2)]

transform = transforms.ToTensor()
y_pred_normed, y_normed = torch.stack([transform(y_pred[k]) for k in range(2)]), torch.stack([transform(y[k]) for k in range(2)])

for i in tqdm.tqdm(range(2)):
    y_pred[i].save(f'pred/{i}.png', format="png")
    y[i].save(f'gt/{i}.png', format="png")
    m.update((y_pred_normed[i:i+1], y_normed[i:i+1]))

print('ignite online FID', m.compute())  # 6.109078042628951

2. Batch is not the same

the ignite version inference with batch size 1 and official version using batch size 2. it surprised me a lot that batch size 1 and batch size 2 will produce different results. here is the evidence:

you can see that with input batch and batch[0:1], the result of the first element is slightly different.

and i confirmed that the model(batch[0:1])'s output is the same as ignite model output.

[vfdev-5]

Thanks a lot @louis-she for this detailed analysis !

[vfdev-5]

@louis-she looks like pytorch-fid results depend on the batch-size:

!python -m pytorch_fid pred gt --num-workers 8 --batch-size 10

100%|███████████████████████████████████████████| 10/10 [00:00<00:00, 13.09it/s]
100%|███████████████████████████████████████████| 10/10 [00:00<00:00, 13.83it/s]
FID:  6.223729414041802

!python -m pytorch_fid pred gt --num-workers 8 --batch-size 20

100%|█████████████████████████████████████████████| 5/5 [00:00<00:00,  6.03it/s]
100%|█████████████████████████████████████████████| 5/5 [00:00<00:00,  6.37it/s]
FID:  6.223768560377792

Following your great catch on different input data due to float->uint8 conversion and back, here is the code I used to compute FID score. Even taking the same batch sizes, ignite results are slightly different:

import torch
import torchvision
import tqdm
from ignite.metrics.gan import FID

from PIL import Image

from pytorch_fid.inception import InceptionV3

device = "cpu"
dims = 2048
block_idx = InceptionV3.BLOCK_INDEX_BY_DIM[dims]
model = InceptionV3([block_idx]).to(device)

import torch.nn as nn


class WrapperInceptionV3(nn.Module):
    
    def __init__(self, fid_incv3):
        super().__init__()
        self.fid_incv3 = fid_incv3
    
    @torch.no_grad()
    def forward(self, x):
        y = self.fid_incv3(x)
        y = y[0]
        y = y[:, :, 0, 0]
        return y

wrapper_model = WrapperInceptionV3(model)
wrapper_model.eval();

m = FID(num_features=dims, feature_extractor=wrapper_model)

le = 100
torch.manual_seed(0)
y_pred = torch.randint(0, 255, size=(le, 3, 299, 299), dtype=torch.uint8)
y = torch.randint(0, 255, size=(le, 3, 299, 299), dtype=torch.uint8)

bs = 10
n = (len(y_pred) + bs - 1) // bs

for i in tqdm.tqdm(range(n)):
    
    b_y_pred = y_pred[i * bs: (i + 1) * bs] / 255.0
    b_y = y[i * bs: (i + 1) * bs] / 255.0    
    m.update((b_y_pred, b_y))


print('ignite online FID', m.compute())  # 6.223657349649425, bs=10


m = FID(num_features=dims, feature_extractor=wrapper_model)
bs = 20
n = (len(y_pred) + bs - 1) // bs

for i in tqdm.tqdm(range(n)):
    b_y_pred = y_pred[i * bs: (i + 1) * bs] / 255.0
    b_y = y[i * bs: (i + 1) * bs] / 255.0    
    m.update((b_y_pred, b_y))

print(i * bs, (i + 1) * bs)    
print('ignite online FID', m.compute())  # 6.223657349649425, bs = 20

# Save the data
for i in tqdm.tqdm(range(len(y))):
    # Save data for PyTorch-FID
    # Should be RGB 256 bits
    # https://github.com/mseitzer/pytorch-fid/blob/3d604a25516746c3a4a5548c8610e99010b2c819/src/pytorch_fid/fid_score.py#L84
    pil_y_rgb = Image.fromarray(y[i, ...].permute(1, 2, 0).numpy())
    pil_y_rgb.save(f'gt/{i}.png')
    pil_y_pred_rgb = Image.fromarray(y_pred[i, ...].permute(1, 2, 0).numpy())
    pil_y_pred_rgb.save(f'pred/{i}.png')

However, model's forward pass give consistent results :

device = "cpu"
block_idx = InceptionV3.BLOCK_INDEX_BY_DIM[dims]
model = InceptionV3([block_idx]).to(device)

model.eval()
with torch.no_grad():
    pfid_out = model(b_y_pred)[0]

print(pfid_out.shape, pfid_out.mean())
print(pfid_out[0, :10, 0, 0])

model.eval()
with torch.no_grad():
    pfid_out = model(b_y_pred[:5])[0]

print(pfid_out.shape, pfid_out.mean())
print(pfid_out[0, :10, 0, 0])

> 
torch.Size([10, 2048, 1, 1]) tensor(0.1865)
tensor([0.2680, 0.0000, 0.1904, 0.0124, 0.0264, 0.0327, 0.0000, 3.0221, 0.4712,
        0.0907])
torch.Size([5, 2048, 1, 1]) tensor(0.1841)
tensor([0.2680, 0.0000, 0.1904, 0.0124, 0.0264, 0.0327, 0.0000, 3.0221, 0.4712,
        0.0907])

[louis-she]

@vfdev-5 looks like pytorch-fid results depend on the batch-size

After some tests, the batch-size issue is caused by device, if you give device='cuda' to the code you provided, it will get different results with batch size 10 and 100:

...
device = "cuda"
...
m = FID(num_features=dims, feature_extractor=wrapper_model, device=device)
...
m = FID(num_features=dims, feature_extractor=wrapper_model, device=device)
...

The official fid automatically enables cuda by default, so the results of it with different batch size will not be the same. If we set the device as cpu, the batch-size issue will also be gone.

So the batch size issue is clear( not solved yet, but I believe it should belongs to another issue ).

Now if we enable cpu on both side, the results is still slightly different, i'll look into this.

[louis-she]

I confirmed that the main difference is comming from the batch size with device CUDA. If fixed that, there is still a very minor difference which should be comming from the implementation of computing covariance matrix.

The official implementation use np.cov to get it but ignite online version is computing manually.

Also the ignite implementation is mainly based on torch.Tensor but official implementation is based on np.ndarray. Some tiny numberical difference coming from there too.

Anyway i think we are good to go cause the difference now is at ~ 1e-7, compare to the original: 1e-2

To sum this up:

The main difference is comming from the differenct batch size on CUDA inconsistent behavior of PyTorch(maybe, we will have to look into this more). The implementation difference only contribute very little difference that can be ignore i think.

@vfdev-5