Documentation Improvements Tracker

[NivekT]

Here are some improvements that we should make to the documentation. Some of these likely should be completed before beta release.

Crucial:

• Add docstrings for the class IterDataPipe and MapDataPipe
  [DataPipe] Add docstrings for IterDataPipe and MapDataPipe, along with small doc changes for consistency pytorch#72618
• Review the categorization of IterDataPipe in torchdata.datapipes.iter.rst
  Categorizing IterDataPipes #219
• Edit first sentence of each DataPipe docstring to be a concise summary of functionality (also include functional name when it exists)
  [DataPipe] Fixing MapDataPipe docstrings pytorch#72476
  [DataPipe] Fixing IterDataPipe docstrings pytorch#72475
  Fixing docstrings for IterDataPipes #209
• Add usage examples to each DataPipe docstring
  [DataPipe] Adding usage examples for IterDataPipes pytorch#73033
  [DataPipe] Adding examples for MapDataPipes with small fixes for others pytorch#73250
  Adding usage examples to all IterDataPipes #249
• Add tutorial (how to use DataPipe, how to write one, how to use it with DataLoader)
  Adding TorchData tutorial #212
• Add domain usage examples (links to files)
  Adding domain specific examples #216
• Decide what utility functions to include
  Adding documentation for utility functions #205
• Link to relevant DataLoader documentation
  Adding documentation for utility functions #205
• Turn on 'gh-pages' in this repo's setting
  It is enabled.
• Clear labelling of prototype vs beta phase
  Change doc version to beta #252
• Add a link under the 'Docs' tab on pytorch.org

Nice-to-have:

• Update issue form for documentation related issues
  Documentation issue form update #215
• Add links to domain usage examples onto individual DataPipe pages (see how TorchVision does this)
• Remove tutorial from README.md and link it to the documentation tutorial
• Make a functional equivalent table in documentation (in a separate page?)

cc: @VitalyFedyunin @ejguan @wenleix @dongreenberg @NivekT

[linminhtoo]

Hello, does this issue also plan to update the PyTorch documentation as well?
it may be slightly confusing to first-time users of datapipes, as the documentation in torchdata and torch can conflict.

For example, in the official PyTorch repo, like, https://github.com/pytorch/pytorch/blob/master/torch/utils/data/datapipes/datapipe.py
the doc-string for how to properly multiprocess with IterDataPipe is:

• incorrect (bcos the worker_init_fn for IterableDataset will not work for IterDataPipe)
• incomplete (bcos it doesn't have the updated documentation about using .sharding_filter() and the new worker_init_fn that does graph_settings.apply_sharding(...))

[linminhtoo]

That being said, I would be happy to contribute, if you could give me pointers on how I can do so.

[NivekT]

Hi @linminhtoo,

Thanks for looking into this!

Hello, does this issue also plan to update the PyTorch documentation as well? it may be slightly confusing to first-time users of datapipes, as the documentation in torchdata and torch can conflict.

This issue does cover our documentation within torch as well.

For example, in the official PyTorch repo, like, https://github.com/pytorch/pytorch/blob/master/torch/utils/data/datapipes/datapipe.py the doc-string for how to properly multiprocess with IterDataPipe is:

• incorrect (bcos the worker_init_fn for IterableDataset will not work for IterDataPipe)

Can you elaborate on that point?

• incomplete (bcos it doesn't have the updated documentation about using .sharding_filter() and the new worker_init_fn that does graph_settings.apply_sharding(...))

We are planning to land this PR, after which, users should not need to pass in additional worker_init_fn for the purpose of sharding.

• [DataLoader] DataLoader now automatically apply sharding to DataPipes pytorch#78631