Skip to content

not all prototype transforms are serializable #6728

Open
@pmeier

Description

@pmeier

This is important for two reasons:

  1. If we stick to torchdata for datasets v2, every object on the datapipe graph has to be serializable for multiprocessing. IIRC, this is only a requirement if forking from the main process is not possible, but that is the case on Windows.
  2. Our classification references have the ability to cache the dataset which includes the transformations:

utils.save_on_master((dataset, traindir), cache_path)

So far we don't have a unified test framework for our prototype transforms and thus it is not easy to add tests for all transforms. Note that we don't need tests for the functional kernels and dispatchers, since they are already tested for JIT scriptability and that protocol is a lot stricter than pickling.

cc @vfdev-5 @datumbox @bjuncek

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions