Bug with transforms.Resize when used with transforms.ConvertImageDtype

[numpee]

🐛 Describe the bug

Recent releases of Torchvision and the documentations that support it seem to suggest that we can use io.read_image + transforms.ConvertImageDtype instead of the traditional PIL.Image.read_fn + transforms.ToTensor. However, I have found that there are two issues:

1. io.read_image + transforms.ConvertImageDtype do not actually return the same tensor values as PIL + transforms.ToTensor, even though they are supposed to provide the same functionality.
2. While io.read_image + transforms.ConvertImageDtype itself is significantly faster than using PIL, combining it with the transforms.Resize operation - specifically when upsampling - makes the operation much slower than the PIL alternative.

To add onto point 2, the two sets of functions I mention return the same type of tensor: torch.float. However, applying transforms.Resize on the tensor generated by io.read_image + transforms.ConvertImageDtype is much slower than applying the same resize operation on the output of PIL read + transforms.ToTensor. I can't really understand why this happens, since both calls to Resize are on tensors of type torch.FloatTensor. Also, this only occurs when upsampling.

Please refer to my post on the Pytorch Forum here for the full analysis.

Versions

Collecting environment information...
PyTorch version: 1.10.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.7 LTS (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.23

Python version: 3.9.4 (default, Apr 9 2021, 01:15:05) [GCC 5.4.0 20160609] (64-bit runtime)
Python platform: Linux-4.15.0-142-generic-x86_64-with-glibc2.23
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 2080 Ti
GPU 1: NVIDIA GeForce RTX 2080 Ti
GPU 2: NVIDIA GeForce RTX 2080 Ti
GPU 3: NVIDIA GeForce RTX 2080 Ti

Nvidia driver version: 465.19.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.1
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.3
[pip3] torch==1.10.0+cu113
[pip3] torchaudio==0.10.0+cu113
[pip3] torchvision==0.11.1+cu113
[conda] Could not collect

cc @vfdev-5 @datumbox

[vfdev-5]

@numpee thanks for the report, I'll try to repro your benchmarks and report back.
Transforms on tensor are known to be slower if executed on uint8. For example, here is by benchmark of various transforms PIL vs Tensor by dtype: https://gist.github.com/vfdev-5/229b0a3a7f392d46cae40502e8139fc0

[vfdev-5]

@numpee first of all, I should have say that PIL image reader and torchvision.io.read_file do not produce the same results.
For example

And where your issue is pertinent is that (thanks for raising it!) upsampling resize on float is surprisingly slow:

Torch version: 1.11.0.dev20211104+cu111
Torchvision version: 0.12.0a0
PIL version: 8.3.2
[----------------------- Resize Upsampling ------------------------]
                         |  Transform on PIL  |  Transform on Tensor
1 threads: ---------------------------------------------------------
      torch.uint8 / RGB  |       419.8        |         1085.6      
      torch.float32 / F  |       184.4        |          675.7    

6 threads: ---------------------------------------------------------
      torch.uint8 / RGB  |       437.1        |         308.9       
      torch.float32 / F  |       184.1        |         685.2       

However IMO, the upsampling from 64 -> 224 is not a usual use-case for neural networks training etc ? Any thoughts on how pertinent to work on that ?

cc @fmassa

On the other hand resize is rather fast for downsampling op (500 -> 256)

Torch version: 1.11.0.dev20211104+cu111
Torchvision version: 0.12.0a0
PIL version: 8.3.2
[----------------------------- Resize -----------------------------]
                         |  Transform on PIL  |  Transform on Tensor
1 threads: ---------------------------------------------------------
      torch.uint8 / RGB  |       1732.3       |         1851.5      
      torch.float32 / F  |        900.1       |          875.7      

[numpee]

@vfdev-5 Thanks for the reply!

As you mentioned, upsampling from 64 --> 224 is not a usual case for training, but it was part of the baseline method (not sure why) which is why I was trying to use it as well.

A few follow up questions I have (not necessarily issues, but just out of curiosity):

1. When using transforms.Resize on the output of PIL + transforms.ToTensor, does the Resize function make a different call internally than when the input is from torchvision.io.read_image + transforms.ConvertImageDtype(torch.float)? Because if both cases call the same function internally, I'm not sure why there would be a speed difference in Resize, given that both cases have the same input type (torch.floatTensor)
2. Do you know the reason why PIL reader and torchvision.io.read_image output different tensors given the same image? Is one considered more "accurate" than the other?

[vfdev-5]

Concerning the question 1, the difference is the following:
a) "PIL + transforms.ToTensor" creates a contiguous channel first tensor
b) "torchvision.io.read_image" creates a non-contiguous tensor channel first tensor (or contiguous tensor channel last tensor if unsqueeze(0))

torchvision's Resize should call internally torch.nn.functional.interpolate which performance depends on contiguous/non-contiguous input:

Torch version: 1.11.0.dev20211104+cu111
Torchvision version: 0.12.0a0
PIL version: 8.3.2
[------ Benchmark reader+transformation ------]
                                      |   Time 
1 threads: ------------------------------------
      ContigTensor->DType->Resize     |   898.1
      NonContigTensor->DType->Resize  |  1302.2

Times are in microseconds (us).

As for question 2: torchvision.io.read_image gives the same pixels as opencv with BGR->RGB conversion applied:

import cv2
import numpy as np
import torchvision.io as io

def cv_loader(path):
    return cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2RGB)

def torch_loader(path):
    mode = io.ImageReadMode.RGB
    return io.read_image(path, mode)

t_img = torch_loader(p.as_posix())
np_t_img = t_img.permute(1, 2, 0).contiguous().cpu().numpy()
np_img2 = cv_loader(p.as_posix())

diff = (np_t_img - np_img2)
print("MSE, Abs Max/Min diff: ", (diff ** 2).mean(), np.abs(diff).max(), np.abs(diff).min())
> MSE, Abs Max/Min diff:  0.0 0 0

[vfdev-5]

If I fix locally the issue with with non-contiguous output from read_image the results are the following:

Num threads: 6
Torch version: 1.11.0.dev20211104+cu111
Torchvision version: 0.12.0a0 (decode_jpg, decode_png modified)
PIL version: 8.3.2
[----- Benchmark reader+transformation ------]
                                     |   Time 
6 threads: -----------------------------------
      PIL->Resize->Tensor->Norm      |  1090.3
      PIL->Tensor->Resize->Norm      |   715.0
      PTH->DType->Resize->Norm       |   602.8
      JIT: PTH->DType->Resize->Norm  |   513.4

Times are in microseconds (us).

Here is the code used for the benchmarking: https://gist.github.com/vfdev-5/8c26a109d7718035162a6d5d138b5499

To compare with currently non-contiguous output from read_image:

Num threads: 6
Torch version: 1.11.0.dev20211104+cu111
Torchvision version: 0.12.0a0
PIL version: 8.3.2
[----- Benchmark reader+transformation ------]
                                     |   Time 
6 threads: -----------------------------------
      PIL->Resize->Tensor->Norm      |  1085.2
      PIL->Tensor->Resize->Norm      |   715.4
      PTH->DType->Resize->Norm       |  1327.7
      JIT: PTH->DType->Resize->Norm  |  1228.5

Times are in microseconds (us).

[numpee]

@vfdev-5 thanks for the explanation and the updates. The updated version seems to be a great performance boost (for upsampling)!
Also, did not know torchvision.io.read_image == cv2.imread != PIL read.

Once again, thanks for addressing this 👍

[vfdev-5]

@numpee well, the fix I proposed is not correct (see #4901) as we now lose on reading and decoding:

Num threads: 6
Torch version: 1.11.0.dev20211104+cu111
Torchvision version: 0.12.0a0
PIL version: 8.3.2
[ Benchmark reader+transformation ]
                   |   Time
6 threads: ----------------
      PIL->Tensor  |  349.1
      PTH contig->DType   |  243.5
      PTH nonconig->DType   |  206.4

Times are in microseconds (us).

We have to support the output format in the transforms themselves...

[numpee]

I see - I'll have to keep an eye out for updates in the future 😄

[fmassa]

@vfdev-5 as a temporary workaround, you could probably patch https://github.com/pytorch/pytorch/blob/6011c35a7934a1528c2db02c40752aa3822ccfa6/aten/src/ATen/native/cpu/UpSampleKernel.cpp#L805-L812 (and other functions therein) so that if num_channels < 4 you dispatch to the generic kernels.

Some benchmarks would be necessary, but it would probably be good enough for this use-case I believe

[vfdev-5]

Some info and benchmarks on this issue:

1. Upsampling and downsampling are slow if using the tensor coming from read_image:

1 threads: ------------------------------------------------------------
      read_image tensor -> Resize (32->224)               |  1168.9
      read_image tensor + contiguous -> Resize (32->224)  |   764.9
      CF contig tensor -> Resize (32->224)                |   757.3
      CL contig tensor -> Resize (32->224)                |   786.6
6 threads: ------------------------------------------------------------
      read_image tensor -> Resize (32->224)               |   986.8
      read_image tensor + contiguous -> Resize (32->224)  |   208.9
      CF contig tensor -> Resize (32->224)                |   200.0
      CL contig tensor -> Resize (32->224)                |   790.6

Times are in microseconds (us).

• "read_image tensor -> Resize (32->224)" measures upsampling resize on the tensor returned by read_image. This tensor is 3D (3, 32, 32) and could be seen as channels last.
• "read_image tensor + contiguous -> Resize (32->224)" measures upsampling resize on the tensor returned by read_image and on which we applied contiguous op. This tensor is 3D (3, 32, 32) and could be seen as channels first.
• "CF contig tensor -> Resize (32->224)" and "CL contig tensor -> Resize (32->224)" measure upsampling resize on channels first (CF) and channels last (CL) contiguous 4D tensors, (1, 3, 32, 32).

Why "read_image tensor -> Resize (32->224)" is much slower than "CL contig tensor -> Resize (32->224)" for 1 thread ?
This due to the following reason. Tensor returned by read_image has memory format equivalent to channels last but has 3D. In Resize we call unsqueeze(dim=0) to make it 4D. Thus input to torch.nn.functional.interpolate is 4D channels last tensor, but output constructed from input suggested format here is channels first contiguous.

auto t = at::rand({1, 3, 32, 32}).contiguous(at::MemoryFormat::ChannelsLast);
std::cout << t.sizes()
> [1, 3, 32, 32]
std::cout << t.suggest_memory_format()
> ChannelsLast
std::cout << (t.is_contiguous(at::MemoryFormat::ChannelsLast) ? "true" : "false");
> true
auto t0 = t[0];
auto t1 = t0.unsqueeze(0);
std::cout << t0.sizes()
> [3, 32, 32]
std::cout << t1.sizes()
> [1, 3, 32, 32]
std::cout << (t1.is_contiguous(at::MemoryFormat::ChannelsLast) ? "true" : "false");
> true
std::cout << t1.suggest_memory_format()
> Contiguous

As output is contiguous channels first but input is channels last, there are 2 places where algorithm takes time:

• auto output = output_.contiguous(channels_last_memory_format);, https://github.com/pytorch/pytorch/blob/e3bcf64ff84f8e96839e39056b3b90d1bd1f8bbe/aten/src/ATen/native/cpu/UpSampleKernel.cpp#L362
• output_.copy_(output); : https://github.com/pytorch/pytorch/blob/e3bcf64ff84f8e96839e39056b3b90d1bd1f8bbe/aten/src/ATen/native/cpu/UpSampleKernel.cpp#L511-L513

Similar benchmark results are for downsampling 500 -> 224

1 threads: -------------------------------------------------------------
      read_image tensor -> Resize (500->224)                   |  1162.3
      read_image tensor + contiguous -> Resize (500->224)      |   780.1
      CF contig tensor -> Resize (500->224)                    |   774.1
      CL contig tensor -> Resize (500->224)                    |   790.0
6 threads: -------------------------------------------------------------
      read_image tensor -> Resize (500->224)                   |   958.2
      read_image tensor + contiguous -> Resize (500->224)      |   210.8
      CF contig tensor -> Resize (500->224)                    |   203.7
      CL contig tensor -> Resize (500->224)                    |   790.4

Times are in microseconds (us).