Enhance torchvision's draw_keypoint method

[oke-aditya]

🚀 The feature

Follow up of #4216 . The current utility works fine, but we can make it better :)

Motivation, pitch

I think the following can be added.

• Support multiple colors

There were 2 proposals one by @fmassa and one by me. The problem was handling excess colors.
I think that in fmassa's version we can broadcast to bigger dimensions and make it work?

• Support specifying line color.

In my opinion, we shouldn't complicate this and just allow a single string.
Happy to hear thoughts though.

• Plot the circle above the line or vice versa?

We currently plot the line above the circle. So at places I see that the line comes above the circle, somehow it looks like the line cuts / bisects the circle. I feel we should do the opposite.

• Don't use default white for all colors. Use either a random palette / better default one.

I guess we should extend this functionality to all plotting utils, by default both draw_bounding_boxes and draw_keypoints plot with white color. While segmentation masks uses a fixed color palette.
Either we could use random colors / just have a nice fixed default palette.
(Also see #4658 )

cc @fmassa @datumbox @NicolasHug

[oke-aditya]

@datumbox @NicolasHug @YosuaMichael let's take this down. I'm up for a PR. It's been 2 releases since we have the keypoints API which isn't super good yet.

[shivaang12]

Hi,

I am extensively using this method in my projects. I have few design suggestions if it make sense to you.

Keypoints Color

This method excepts the keypoints in shape (K, N, 2) where K is number of instances. No matter how many instances you have of keypoints it going to draw with same color. I think instead of drawing K instance with same color we can modify the behavior to draw with array of different color.

Connectivity Skeleton

• I think there should be different color options for connectivity lines as mentioned in this PR.
• For projects involving keypoint detection, connectivity skeleton options is not out-of-the-box convenient.

Let me know what are your thoughts about these points. I am happy to contribute if needed.

[oke-aditya]

Yeah so we kind of didn't have consensus on this. But @datumbox @fmassa what do you think about adding above features?

[datumbox]

Thanks for the ping! Apologies for the lack of responses.

I want to be transparent on what's holding back the decisions around this. We are currently a bit swamped with the release, on going projects etc and we haven't had the opportunity to look into the Keypoint side of things. For example, though Transforms V2 is built to support Detection and Segmentation models, there are no current plans to extend it to Keypoints. That unfortunately is a beast of each own and requires us to put some thoughts on how to represent them, transform them and also visualize them. The risk we are running here is agreeing with a proposal without getting the full picture and then locking ourselves in due to the very strict BC-guarantees that TorchVision offers. So the problem is mostly that we lack the bandwidth to review the big picture around keypoints and make the necessary decisions over the direction. Note that some of these topics are tricky, as we discussed previously in the area of Detection, colouring different objects often require additional meta-data that we don't have (such as things vs stuff). I don't have great visibility on Keypoints and it's the things you don't know you don't know that often get you.

So a valid question is, when can we pick this up? I think after releasing Transforms V2 for Image & Video classification, Object Detection, Semantic & Instance Segmentation we would have to look into the Keypoints. There are other areas that might get in front of the queue (for example Depth Perception and Optical Flow). So there is overall lack of clarity on that front.

If @fmassa has an advice here, I would be happy to adopt it. But I also know he is completely swamped and I suspect reviewing this will take time.

Apologies for not being very helpful on this one. 😞

[oke-aditya]

Yeah it's a bit tight. And please don't say sorry 😐

Multiple projects like, transforms, datasets with datapipes, newer models, batteries included phase 1/2/3, multi weight support, newer Operators, adding new tasks like optical flow, stereo.

The torchvision team is really working so hard. Just that this is a bit lower on priority but will get back to it soon!

[shivaang12]

Thank you for the response. I completely understand. I would happy to help in future if you work on it. Also want to appreciate all the good work you folks are putting in the repository.

Thanks again!!