Expand tests for prototype datasets

[pmeier]

This PR adds tests and fixes error of all missing prototype datasets.

Blocked by #5186.

cc @pmeier @bjuncek

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit d16b9f6 (more details on the Dr. CI page):

---

• 1/2 failures introduced in this PR
• 1/2 broken upstream at merge base bf073e7 since Jan 14

---

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

unittest_linux_cpu_py3.7 (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

/root/project/torchvision/io/video.py:406: Runt...log: [mov,mp4,m4a,3gp,3g2,mj2] moov atom not found

test/test_image.py::test_decode_png[L-ImageReadMode.GRAY-palette_pytorch.png]
test/test_image.py::test_decode_png[RGB-ImageReadMode.RGB-palette_pytorch.png]
  /root/project/env/lib/python3.7/site-packages/PIL/Image.py:946: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
    "Palette images with Transparency expressed in bytes should be "

test/test_io.py::TestVideo::test_probe_video_from_memory
  /root/project/torchvision/io/_video_opt.py:423: UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /opt/conda/conda-bld/pytorch_1642552271656/work/torch/csrc/utils/tensor_new.cpp:998.)
    video_data = torch.frombuffer(video_data, dtype=torch.uint8)

test/test_io.py::TestVideo::test_read_video_timestamps_corrupted_file
  /root/project/torchvision/io/video.py:406: RuntimeWarning: Failed to open container for /tmp/tmpov3rjnou.mp4; Caught error: [Errno 1094995529] Invalid data found when processing input: '/tmp/tmpov3rjnou.mp4'; last error log: [mov,mp4,m4a,3gp,3g2,mj2] moov atom not found
    warnings.warn(msg, RuntimeWarning)

test/test_models.py::test_memory_efficient_densenet[densenet121]
test/test_models.py::test_memory_efficient_densenet[densenet169]
test/test_models.py::test_memory_efficient_densenet[densenet201]
test/test_models.py::test_memory_efficient_densenet[densenet161]
  /root/project/env/lib/python3.7/site-packages/torch/autocast_mode.py:162: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
    warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')

test/test_models.py::test_inception_v3_eval

---

🚧 1 ongoing upstream failure:

These were probably caused by upstream breakages that are not fixed yet.

• unittest_prototype since Jan 14 (adf8466)
  • 🔁 rerun

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[NicolasHug]

Thanks @pmeier , full disclosure I didn't give this an in-depth look. I'll have more feedback on this once I implement a few of those prototype datasets on my own (WIP :) )

[NicolasHug]

Would the default pytest id generation be enough here?

[pmeier]

Unfortunately not. You would get something like

test/test_prototype_builtin_datasets.py::TestCommon::test_smoke[dataset_mock0-config0]
test/test_prototype_builtin_datasets.py::TestCommon::test_smoke[dataset_mock1-config1]
test/test_prototype_builtin_datasets.py::TestCommon::test_smoke[dataset_mock2-config2]
...

The problem is that by default, pytest uses the name of the parameter as well as a running index as id. So if we want to have somewhat expressive test names, we need to set the id ourselves. The same example as above but with the custom id from this PR looks like

test/test_prototype_builtin_datasets.py::TestCommon::test_smoke[mnist-train]
test/test_prototype_builtin_datasets.py::TestCommon::test_smoke[mnist-test]
test/test_prototype_builtin_datasets.py::TestCommon::test_smoke[fashionmnist-train]
...

[pmeier]

@ejguan The test failure most likely stems from this:

vision/torchvision/prototype/datasets/_builtin/cub200.py

Lines 171 to 178 in f923aeb

	images_dp, split_dp, image_files_dp, bounding_boxes_dp = Demultiplexer(
	archive_dp, 4, self._2011_classify_archive, drop_none=True, buffer_size=INFINITE_BUFFER_SIZE
	)
	image_files_dp = CSVParser(image_files_dp, dialect="cub200")
	image_files_map = dict(
	(image_id, rel_posix_path.rsplit("/", maxsplit=1)[1]) for image_id, rel_posix_path in image_files_dp
	)

I need this mapping twice and so I thought I can simply create it once. IIUC, the problem comes from the fact that this starts iteration of the Demultiplexer while creating the datapipe and thus it has some file handles in buffer when traverse tries to pickle it.

What is recommended pattern here?

[ejguan]

@ejguan The test failure most likely stems from this:

vision/torchvision/prototype/datasets/_builtin/cub200.py

Lines 171 to 178 in f923aeb

	images_dp, split_dp, image_files_dp, bounding_boxes_dp = Demultiplexer(
	archive_dp, 4, self._2011_classify_archive, drop_none=True, buffer_size=INFINITE_BUFFER_SIZE
	)
	image_files_dp = CSVParser(image_files_dp, dialect="cub200")
	image_files_map = dict(
	(image_id, rel_posix_path.rsplit("/", maxsplit=1)[1]) for image_id, rel_posix_path in image_files_dp
	)

I need this mapping twice and so I thought I can simply create it once. IIUC, the problem comes from the fact that this starts iteration of the Demultiplexer while creating the datapipe and thus it has some file handles in buffer when traverse tries to pickle it.

What is recommended pattern here?

Yeah. The problem is Demux has not been depleted (full iteration) to clean up the iterator. There are two ways

@ejguan The test failure most likely stems from this:

vision/torchvision/prototype/datasets/_builtin/cub200.py

Lines 171 to 178 in f923aeb

	images_dp, split_dp, image_files_dp, bounding_boxes_dp = Demultiplexer(
	archive_dp, 4, self._2011_classify_archive, drop_none=True, buffer_size=INFINITE_BUFFER_SIZE
	)
	image_files_dp = CSVParser(image_files_dp, dialect="cub200")
	image_files_map = dict(
	(image_id, rel_posix_path.rsplit("/", maxsplit=1)[1]) for image_id, rel_posix_path in image_files_dp
	)

I need this mapping twice and so I thought I can simply create it once. IIUC, the problem comes from the fact that this starts iteration of the Demultiplexer while creating the datapipe and thus it has some file handles in buffer when traverse tries to pickle it.

What is recommended pattern here?

Thanks for adding such use case. The problem is iterator object and buffer attached to Demultiplexer object is not cleaned up if demux has not been depleted (full iteration of all children). This can be solved by adding special serialize/deserialize functions for demux to prevent serialization of theses objects.
cc: @NivekT

[pmeier]

@NicolasHug I added the ability add marks like pytest.mark.xfail to datasets through the parametrization decorator. With this in place, we can xfail "cub200" on the failing tests until #5187 (comment) is resolved on the torchdata side.