[feature request] rgb2lab / rgb2hsv / rgb2gray and other color space conversions (maybe upstream from kornia? or colorsys python core module?)

[vadimkantorov]

It would be nice to have them available in the standard library (both on CPU/GPU) rather than rolling your own. Kornia seems to have implemented it: https://kornia.readthedocs.io/en/latest/color.html, skimage as well: https://scikit-image.org/docs/dev/api/skimage.color.html, python core as well: https://docs.python.org/3/library/colorsys.html, but I think it's basic enough to merit inclusion of similar functions in core.

(my usecase: SLIC superpixel extraction which requires rgb2lab and selective search algorithm which requires rgb2hsv and rgb2lab)

cc @vfdev-5

[datumbox]

Thanks @vadimkantorov. This is useful feedback as we are currently in discussions to improve our transforms.

[vadimkantorov]

Maybe this is not even related to transforms per se, since useful also in non-augmentation contexts as well.

For transforms, I really hope, there is more native batched transforms, clear separation between function and random-sampling its parameters, and also that there is an easy way to do inverse transforms (where possible). Also getting padding-masks for direct and inverse transforms is useful

[datumbox]

@vadimkantorov The tag transforms includes to anything under the transform module, including augmentations, functional and class transforms. Similar requests have been raised in the past (see #3224 for summary), so that's something we are looking into.

[fmassa]

@vadimkantorov I think adding a rgb2lab would be nice in principle, but the current issue I see with it is that there is no way for the library to know if the input tensor that you are passing is indeed in rgb colorspace. So you can silently get wrong results if you are not careful.
As of now, torchvision assumes that the tensor is either in RGB or grayscale space, and don't try to make any further assumptions otherwise. This works fine because RGB and grayscale have different number of channels, but things break if we start supporting HSV / LAB / etc colorspaces.

Wondering if you think this is a problem per se, and that this is up to the user to make sure that the input tensors are in RGB colorspace?

[vadimkantorov]

I think it's not a problem per se, as we almost do not encode semantics into the tensor type system in PyTorch, in the goal of being flexible and letting users quickly build on things. Same for NumPy.

This problem persists with OpenCV returning BGR images, but it's up to users to remember to convert if this is needed.

In my own practice, I use suffix notation to bring extra attention to semantics, e.g. I use names like mean_rgb, std_rgb in my input normalization functions, especially if I do image reading with OpenCV.

If these functions stay in extra namespace and are not discussed in every other tutorial, the use will be limited to those looking for them, so it should not confuse the users of the most common usecases.

[oke-aditya]

If I understand the problem correctly. In torchvision it is assumed that the colorspace is RGB .
I think if we provide conversions from RGB to other colorspaces and vice versa it would work.

We need to add documentation saying that all the transforms work on RGB Colorspace only and we do not support any other.

This is something we faced earlier in bbox conversions, and we followed xyxy assumption, provided a utility to convert to xyxy and from xyxy.

I guess a similar solution would work here?

Would love to hear from @fmassa and @vadimkantorov 😃

[vadimkantorov]

I think many transforms actually don't care about the input color space and support everything.

Yes, it would be good to add notes to the docs about supported color spaces. But even if it's said somewhere in the docs that the transforms were designed for RGB. And then there're manual color space conversion utils on the side, it would be sufficient and not cause confusion, as long as the image loaders load in RGB by default.

[NicolasHug]

I'm curious about the use-case for this, @vadimkantorov could you describe a bit more why this would be useful to you?

I assume that passing a LAB image into a pretrained model would lead to invalid predictions, so I guess this would only be for new training experiments?

[vadimkantorov]

You are right. Of course it would be invalid to put in LAB image into a RGB-trained model. So yes, this would only be for new models or new functionality.

My usecase is this: reimplement SLIC superpixels extraction in PyTorch, this requires rgb2lab

[NicolasHug]

Thanks for the feedback!

If we consider that interpolation works the same in RGB and in Lab, it's true that most transforms don't make a strong assumption w.r.t. the color space, except maybe stuff like ColorJitter or ToGrayScale.

But as noted above, there aren't just transforms: there are also operators and models, and at the moment literally everything in torchvision is built with RGB/Grayscale in mind.

On top of that, for consistency and coherence of the library, I think it's best not to introduce things for which we don't have an explicit need within the library.

It's still valuable to support new/alternative use-cases though, but IMHO this would be better done in a somewhat separate way, something that we don't officially support but that power-users could still find and use, at their own risk. Typically such conversion snippet could be part of an FAQ entry with a strong disclaimer regarding its compatibility/support... but we don't have an FAQ. Maybe it could be part of a gallery example as well? Or even just a snippet here below that people could copy/paste?

[vadimkantorov]

Well, one also does not have explicit need for drawing bounding boxes within the library (and yet it is was merged in). I think this argument is not very valid. At the end, torchvision is supposed to help users with standardized ways of doing very frequent chores in comptuer vision tasks. Reference implementation of color space transformation is one of these (and it's available in OpenCV since its beginning), and it'd be useful to have one working with GPU tensors in core - like in Kornia (as opposed to OpenCV / skimage).

I think putting it into torchvision.color or torchvision.utils.color would make sense and not confuse anyone. The fact that the models don't support all possible color spaces is obvious, but this should not prevent adding these conversion reference implementations. Putting them into FAQ is also a not -bad option, but I don't see why having these copy-pastes in various codebases woudld be better than providing a stadnard tested implementation. Moreover, conversion APIs have stabilized over the dozens of years in various libraries, so probably a good APIs can just be copied from some other libraries and would not be subject of frequent change. A colorsys module also exists in Python core standard library, but it's not working with arrays, forcing one to write loops.

[NicolasHug]

Well, one also does not have explicit need for drawing bounding boxes within the library (and yet it is was merged in). I think this argument is not very valid

That's why it's a visualization util, not part of the core set of transforms / operators. And the drawing of bounding boxes and masks perfectly integrates with the rest of the library's ecosystem in terms of expected inputs and outputs. It's not an outlier, which a Lab conversion transform would be.

I think putting it into torchvision.color or torchvision.utils.color would make sense and not confuse anyone

Unfortunately, we already have the rgb <-> grayscale conversions in the transforms module.

The fact that the models don't support all possible color spaces is obvious

It's obvious now, because we only support RGB. The second we start having other color spaces conversions, users will assume that the models and transforms will "magically" work with whatever tensor you pass. I'm afraid that this isn't an assumption we can make, and as library authors we try hard to prevent users from doing wrong things.