Add gallery example for drawing keypoints #4892

oke-aditya · 2021-11-09T14:37:26Z

Follow up of #4216

~~Work in progress should be done in few hours :)~~

Open for review. Feedbacks are welcome!

facebook-github-bot · 2021-11-09T14:37:32Z

💊 CI failures summary and remediations

As of commit 71584c0 (more details on the Dr. CI page):

2/2 failures introduced in this PR

2 failures not recognized by patterns:

Job	Step	Action
^{binary_libtorchvision_ops_android}	^Build	🔁 rerun
^{binary_win_conda_py3.9_cu113}	^{Build conda packages}	🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

datumbox · 2021-11-09T16:12:43Z

@oke-aditya I'm not sure from where you sourced the image you are using but I recommend finding an alternative picture from the Coco keypoint dataset used to train the keypoint model.

oke-aditya · 2021-11-09T16:13:58Z

This image was sourced from COCO dataset. Is it a concern since it displays a human image?

Edit: The reason for the image was that it is easier to connect and visualize the skeleton on this image.

datumbox · 2021-11-09T16:21:51Z

@oke-aditya It's good you got it from COCO. My concern is that it seems to depict a minor so let's be extra careful and replace it. Do you think that we can reuse the standard Grace Hopper picture we already have? This might be interesting because on your example you might need to filter out keypoints that don't appear on the picture (and thus it's going to be a more realistic example of visualization). Thoughts?

oke-aditya · 2021-11-09T16:31:07Z

Oh! So sorry for depicting a minor! I completely agree we should avoid this image.
Very very thankful to you @datumbox for pointing this out.

My apologies to the minor person depicted (who would have grown up by today 😃 ) who might see this PR.

That's Good idea to add an example where we need to filter out keypoints. Both based on visibility and detection threshold.
But I feel that joining a complete skeleton is particular useful as for pose-detection / other tasks. So I will keep two examples.
First one just joining skeletons and next one helping to filter keypoints well

datumbox · 2021-11-09T16:39:13Z

@oke-aditya Sounds good. Let's select an alternative picture from COCO and do 2 examples as you proposed. Thanks!

oke-aditya

Fixing a small bug.
I'm still unable to find an image where visibility of keypoint isn't 1.000.

torchvision/utils.py

test/test_utils.py

gallery/plot_visualization_utils.py

oke-aditya · 2021-11-09T19:17:27Z

Absolutely baffled still that every run with keypoint_rcnn_resnet50 with both pretrained=True and False, the visibility was always 1.00 for all the keypoints and instances.

Tried over 15 images from coco, of not just persons, dogs, cats, surfboards. :(

Is there really a case where visibility won't be 1 ? Can someone provide an image for such case.

datumbox · 2021-11-09T19:36:27Z

@oke-aditya I had the same exact discussion with Francisco couple of days ago and he clarified that this is always 1. So I suggest to just ignore this column and move on.

@fmassa could you please confirm that this is the case?

oke-aditya · 2021-11-09T20:17:27Z

gallery/plot_visualization_utils.py

+# As we see the keypoints appear as colored circles over the image.
+# The coco keypoints for a person are ordered and represent the following list.\
+
+coco_keypoints = [


I think this will be great when people can use metadata from the new keypoint rcnn weights.
Right now we have to explicitly remember this.

Although it isn't used in the code. I made this as a code block so that it is copy-pastable.

Yes agreed. We've already added it on the new API:

vision/torchvision/prototype/models/detection/keypoint_rcnn.py

Line 24 in 9a7dc1a

_common_meta = {"categories": _COCO_PERSON_CATEGORIES, "keypoint_names": _COCO_PERSON_KEYPOINT_NAMES}

vision/torchvision/prototype/models/_meta.py

Lines 1106 to 1109 in 9a7dc1a

_COCO_PERSON_CATEGORIES = ["no person", "person"]

_COCO_PERSON_KEYPOINT_NAMES = [

"nose",

"left_eye",

I guess once the new prototype models and transforms are moved to stable, we would need to revisit the gallery examples.

oke-aditya · 2021-11-09T20:21:22Z

I have added one single example showing plotting keypoints and joining them.
I felt that one detailed and simple example is necessary as keypoint rcnn involves little bit more processing and understanding to connect keypoints.

Let me know if an additional example such as batch of images or plotting multi instance images is needed.
I think it will be nice to have.

datumbox

Overall it looks good to me, thanks @oke-aditya.

Just a couple minor comments. Let me know what you think:

datumbox · 2021-11-10T09:09:03Z

gallery/plot_visualization_utils.py

+# As we see the keypoints appear as colored circles over the image.
+# The coco keypoints for a person are ordered and represent the following list.\
+
+coco_keypoints = [


Yes agreed. We've already added it on the new API:

vision/torchvision/prototype/models/detection/keypoint_rcnn.py

Line 24 in 9a7dc1a

_common_meta = {"categories": _COCO_PERSON_CATEGORIES, "keypoint_names": _COCO_PERSON_KEYPOINT_NAMES}

vision/torchvision/prototype/models/_meta.py

Lines 1106 to 1109 in 9a7dc1a

_COCO_PERSON_CATEGORIES = ["no person", "person"]

_COCO_PERSON_KEYPOINT_NAMES = [

"nose",

"left_eye",

gallery/plot_visualization_utils.py

Co-authored-by: Vasilis Vryniotis <[email protected]>

NicolasHug · 2021-11-10T10:11:18Z

Thanks @oke-aditya , I just gave a brief look to the rendered example and it looks great. The only comment I have is about the first sentence in the example which is

This example illustrates some of the utilities that torchvision offers for visualizing images, bounding boxes, and segmentation masks.

and that could now be updated to

This example illustrates some of the utilities that torchvision offers for visualizing images, bounding boxes, segmentation masks, and keypoints.

Nice work! :)

oke-aditya · 2021-11-10T10:15:04Z

This example illustrates some of the utilities that torchvision offers for visualizing images, bounding boxes, and segmentation masks.

Nice, catch. I will also update the visualization thumbnail to include this image too.

Nice work! :)

Thanks Nicolas :) once we finish supporting multiple colors, line colors, etc.
I will add one small example which shows multiple instances. The utility looks cool 😎 on multiple images.

oke-aditya · 2021-11-10T12:08:15Z

Here is the final gallery. Feel free to suggest changes :)

https://980059-73328905-gh.circle-artifacts.com/0/docs/auto_examples/index.html

cc @NicolasHug @datumbox

github-actions · 2021-11-10T13:07:27Z

Hey @datumbox!

You merged this PR, but no labels were added. The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py

datumbox · 2021-11-10T13:07:51Z

@oke-aditya Great job as always Oke. Thanks for contributing this and for your patience for the reviews on the original PR.

* Start writing gallery example * Remove the child image fix implementation add code * add docs * Apply suggestions from code review Co-authored-by: Vasilis Vryniotis <[email protected]> * address review update thumbnail Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Vasilis Vryniotis <[email protected]>

Summary: * Start writing gallery example * Remove the child image fix implementation add code * add docs * Apply suggestions from code review * address review update thumbnail Reviewed By: datumbox Differential Revision: D32470492 fbshipit-source-id: ae83abd054dccd4c2d423bf0f1c7e1ae04852b8a Co-authored-by: Vasilis Vryniotis <[email protected]> Co-authored-by: Philip Meier <[email protected]> Co-authored-by: Vasilis Vryniotis <[email protected]>

Start writing gallery example

e6a7275

pytorch-probot bot added the ciflow/default label Nov 9, 2021

facebook-github-bot added the cla signed label Nov 9, 2021

Remove the child image fix implementation add code

ca28bbf

oke-aditya commented Nov 9, 2021

View reviewed changes

torchvision/utils.py Show resolved Hide resolved

test/test_utils.py Show resolved Hide resolved

gallery/plot_visualization_utils.py Show resolved Hide resolved

Merge branch 'main' of https://github.com/pytorch/vision into kpt_gall

94a59b5

add docs

b3baf73

oke-aditya commented Nov 9, 2021

View reviewed changes

oke-aditya changed the title ~~[WIP] Add gallery example for drawing keypoints~~ Add gallery example for drawing keypoints Nov 9, 2021

oke-aditya marked this pull request as ready for review November 9, 2021 20:19

datumbox reviewed Nov 10, 2021

View reviewed changes

pmeier and others added 2 commits November 10, 2021 10:38

Merge branch 'main' into kpt_gall

67395c0

Apply suggestions from code review

d965ac9

Co-authored-by: Vasilis Vryniotis <[email protected]>

oke-aditya added 2 commits November 10, 2021 17:21

address review update thumbnail

0370108

Merge branch 'kpt_gall' of github.com:oke-aditya/vision into kpt_gall

47c3d9f

oke-aditya requested a review from datumbox November 10, 2021 12:08

Merge branch 'main' into kpt_gall

71584c0

datumbox merged commit 30669a5 into pytorch:main Nov 10, 2021

oke-aditya deleted the kpt_gall branch November 10, 2021 13:07

datumbox added enhancement module: documentation labels Nov 10, 2021

	_COCO_PERSON_CATEGORIES = ["no person", "person"]
	_COCO_PERSON_KEYPOINT_NAMES = [
	"nose",
	"left_eye",

Add gallery example for drawing keypoints #4892

Add gallery example for drawing keypoints #4892

Uh oh!

Conversation

oke-aditya commented Nov 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Nov 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

2 failures not recognized by patterns:

Uh oh!

datumbox commented Nov 9, 2021

Uh oh!

oke-aditya commented Nov 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datumbox commented Nov 9, 2021

Uh oh!

oke-aditya commented Nov 9, 2021

Uh oh!

datumbox commented Nov 9, 2021

Uh oh!

oke-aditya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oke-aditya commented Nov 9, 2021

Uh oh!

datumbox commented Nov 9, 2021

Uh oh!

oke-aditya Nov 9, 2021

Choose a reason for hiding this comment

Uh oh!

oke-aditya Nov 9, 2021

Choose a reason for hiding this comment

Uh oh!

datumbox Nov 10, 2021

Choose a reason for hiding this comment

Uh oh!

oke-aditya Nov 10, 2021

Choose a reason for hiding this comment

Uh oh!

oke-aditya commented Nov 9, 2021

Uh oh!

datumbox left a comment

Choose a reason for hiding this comment

Uh oh!

datumbox Nov 10, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

NicolasHug commented Nov 10, 2021

Uh oh!

oke-aditya commented Nov 10, 2021

Uh oh!

oke-aditya commented Nov 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 10, 2021

Uh oh!

datumbox commented Nov 10, 2021

Uh oh!

Uh oh!

oke-aditya commented Nov 9, 2021 •

edited

Loading

facebook-github-bot commented Nov 9, 2021 •

edited

Loading

oke-aditya commented Nov 9, 2021 •

edited

Loading

oke-aditya commented Nov 10, 2021 •

edited

Loading