MaskRCNN Training on Images with no Annotations

[gatordevin]

Hi all,
I am working on a little MaskRCNN training program and ran into an issue. I know it is common practice to remove any images from the dataset that lack annotations upon initializing the dataset which I am doing. However, I am running a series of transforms using albumentations on my image and my mask. One of these transforms is a random crop and sometimes the resulting mask image no longer contains any instances. I was trying to find a way to pass in an empty tensor of some kind without much success. Would it be common practice just to remove it from the batch, and if so what happens if you had a batch size of 1 or an image that only had one annotation and the chances the random crop came across it are really low. I was able to create an empty tensor and pass it in but then received this error.

RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1] because the unspecified dimension size -1 can be any value and is ambiguous

This is because my box tensor had a shape of 0, 4 which is what I want since there are no instances. I read some of the other issue reports and they talked about creating a background class and just making a small bounding box and having an empty segmentation mask but this seems a little hacky and I was wondering if there would be a better solution for my specific use case.

[vfdev-5]

@gatordevin please take a look at reference example here : https://github.com/pytorch/vision/tree/master/references/detection
and

vision/references/detection/coco_utils.py

Line 244 in 8c28175

dataset = _coco_remove_images_without_annotations(dataset)

[gatordevin]

Okay, so my understanding of this code is that the transforms are applied once to the dataset upon initialization. Then any images that had annotations removed from the transforms are then removed from the dataset. Thats all fine but what if I would like to apply the randomized transforms to the images every batch.

[vfdev-5]

During the training random horizontal flip is applied on the images and targets:

vision/references/detection/train.py

Line 55 in 8c28175

transforms.append(T.RandomHorizontalFlip(0.5))

[gatordevin]

Oh alright, I see that now. In that case, how would I include a random crop transformation in my dataset since it introduces the chance of removing annotations? From what I have seen the images are only removed when the dataset is initialized not after transform meaning if a random crop is applied and annotations are gone this won't be recognized.

[vfdev-5]

If you are using albumentations, as far as I remember they have transformations aware of target's geometry such that for example random crop will reduce image such that target still present. If you'd like to use torchvision's transforms, I'd suggest to use functional part to perform image transforms. As for transforming targets, this should be, for instance, manually implemented. Same for random parameters sampling.
cc @fmassa for other hints on that.

[datumbox]

@gatordevin I hit a potentially similar issue as you did. Would it be possible to update the ticket to include the full error message?

[fmassa]

Thanks for opening this issue!

I believe #3031 could fix this error, which should only happen in some very specific corner cases.

Following @datumbox comment, can you post the full error message so that we know if there is somewhere else that we should look into?