Ops to convert `masks` to `boxes`

[oke-aditya]

🚀 Feature

A simple torchvision.ops to convert Segmentation masks to bounding boxes.

Motivation

This has a few use-cases.

1. This makes it easier to use semantic segmentation datasets for object detection.
   The pipeline can be easier. Also the bounding boxes are represented as xyxy in torchvision.ops as a convention.
   So probably convert masks to xyxy format.

2. The other use case is to make it easier in comparing performance of segmentation model vs detection model.
   Let's Say that the detection model performs well for segmentation dataset. Then it would be better to go ahead with detection models as it is faster in real-time use-cases than to train a segmentation model.

New Pipeline

from torchvision.ops import masks_to_boxes, box_convert

class SegmentationToDetectionDataset(Dataset):
    def __getitem__(self, idx):
          boxes_xyxy = masks_to_boxes(segmentation_masks)

         # Now for any change of boxes to COCO Format.
          boxes_xywh = box_convert(boxes_xyxy, in_fmt="xyxy", out_fmt="xywh")
          return boxes_xywh

Pitch

Port the masks_to_boxes function from mDeTR.

masks_to_boxes was also used in DeTR.

Alternatives

The above function assumes masks of shape (N, H, W) -> num_masks, Height, Width. A floating tensor.
IIRC, we used a boolean tensor in draw_segmentation_masks (After Nicolas refactored). So perhaps we should be using boolean tensor? Though I see no particular use case of this util being only valid for instance segmentation.

Additional context

I can port this, we perhaps need a few tests to ensure it works fine.
Especially test for float16 overflow.

cc @datumbox @NicolasHug

[NicolasHug]

I think this could be useful, at least as a plotting utils. I'm not sure I fully understand the use of such util within a model though?

So perhaps we should be using boolean tensor?

Yes the input should be boolean: models have different representations for floating masks as illustrated in the new example, so we can't have a unified approach for floating masks.

[oke-aditya]

I think it should under be ops and not utils as utils contain utilities which do not do any manipulation of any masks / boxes.

They return the passed image tensors back

This is a operation (of course is non-differentiable) hece suits better to torchvision.ops.

The only models which have different representation are instance segmentation models. Is this utility essential for instance segmentation? (I'm not sure)

[datumbox]

Thanks for bringing this up @oke-aditya.

This is a text-book example of utility we want to upstream from DeTR to TorchVision. I think it should be OK placing it in torchvision.ops.boxes along with the other box utilities ported already.

Especially test for float16 overflow.

Yes please, good call. We need to be careful for numeric overflows. See #3383 for addressing a similar issues on other box ops.

Edit: Also since we are porting things from DeTR, let's make sure we give credit by putting a reference to the original code. You can see examples of that already in our code.

[oke-aditya]

What do you think @datumbox about the segmentation masks? Should they be kept float as (num_masks, H, W) or should we change to adopt bool masks?

I would prefer to keep float masks.

[datumbox]

This makes it easier to use semantic segmentation datasets for object detection.

I think as per your original point, you would have to support (num_masks, H, W) if you were to convert a the masks of a specific dataset to boxes for object detection. See how DeTR uses the method on the CocoPanoptic dataset.

I also see value to support for bool masks, as this will make it work seamlessly with Nicolas' draw_segmentation_masks but it's unclear to me if that should happen on the same method or on a separate. I would personally start by porting DeTR's implementation, adding tests, handling overflows and take it from there.

Adding @fmassa in the discussion as he usually have good intuition around segmentation/detection.

[0x00b1]

Has anyone started to work on this? If not, I'm more than happy to port it over.

[datumbox]

I think supporting boolean masks probably requires additional discussion but porting DeTR's masks_to_boxes method can be done now. @0x00b1 happy to review a PR about it. Just tag me. :)

[NicolasHug]

I would really rather not start implementing a method that is only specific to a small subset of models.
We should try to avoid the same scenario where we had to re-write the entire draw_segmentation_masks util: #3824

[datumbox]

This is a general purpose utility unrelated to drawing. As part of the Batteries Included initiative, we are upstreaming some utils that exist on ecosystem projects such as DETR. I think the masks_to_boxes is a meaningful addition as it can be on specific datasets such as the CocoPanoptic dataset which I mentioned earlier.

[NicolasHug]

I understand it's unrelated to drawing. My point is that we should aim for this to be as generic as possible so that we can support all possible torchvision use-cases, instead of a subset of them. I might be missing something but the current proposed API seems specifically targeted towards a subset of models / datasets.

[datumbox]

Could you please elaborate on why this specific operator is not generic enough to be included?

IMO a quick search shows that the generic and useful to be used across different use-cases and as a result it's ended up being copy-pasted over and over in multiple projects across FAIR. This makes them an excellent candidate for up streaming which is the key goal of Batteries Included.

[NicolasHug]

From our example https://pytorch.org/vision/stable/auto_examples/plot_visualization_utils.html, instance segmentation masks and semantic segmentation masks both rely on float values but they are encoded very differently and those float values don't mean the same thing. The "greatest common factor" of these 2 distinct representations is the boolean representation.