Skip to content

Commit 558af17

Browse files
committed
Categorizing IterDataPipes
ghstack-source-id: 62ade43 Pull Request resolved: #219
1 parent dfa2676 commit 558af17

File tree

1 file changed

+126
-41
lines changed

1 file changed

+126
-41
lines changed

docs/source/torchdata.datapipes.iter.rst

Lines changed: 126 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -17,16 +17,31 @@ This is an updated version of ``IterableDataset`` in ``torch``.
1717

1818
We have three types of Iterable DataPipes:
1919

20-
1. Load - help you interact with the file systems or online databases (e.g. FileOpener, GDriveReader)
20+
1. IO - DESCRIPTION
2121

22-
2. Transform - transform elements within DataPipes (e.g. batching, shuffling)
22+
2. Archive - DESCRIPTION
2323

24-
3. Utility - utility functions (e.g. caching, CSV parsing, filtering)
24+
3. Mapping - DESCRIPTION
2525

26-
Load DataPipes
26+
4. Selecting - DESCRIPTION
27+
28+
5. Augmenting - DESCRIPTION
29+
30+
6. Combinatorial - DESCRIPTION
31+
32+
7. Text - DESCRIPTION
33+
34+
8. Grouping - DESCRIPTION
35+
36+
9. Combining/Splitting -DESCRIPTION
37+
38+
10. Others - DESCRIPTION
39+
40+
IO DataPipes
2741
-------------------------
2842

29-
These DataPipes help you interact with the file systems or online databases (e.g. FileOpener, GDriveReader).
43+
These DataPipes help interacting with the file systems or online databases (e.g. downloading, opening,
44+
saving files, and listing the files in directories).
3045

3146
.. autosummary::
3247
:nosignatures:
@@ -42,11 +57,28 @@ These DataPipes help you interact with the file systems or online databases (e.g
4257
HttpReader
4358
IoPathFileLister
4459
IoPathFileOpener
60+
IoPathSaver
4561
OnlineReader
4662
ParquetDataFrameLoader
63+
Saver
64+
65+
Archive DataPipes
66+
-------------------------
67+
68+
These DataPipes help opening and decompressing archive files.
4769

70+
.. autosummary::
71+
:nosignatures:
72+
:toctree: generated/
73+
:template: datapipe.rst
4874

49-
Transform DataPipes
75+
Extractor
76+
RarArchiveLoader
77+
TarArchiveReader
78+
XzFileReader
79+
ZipArchiveReader
80+
81+
Mapping DataPipes
5082
-------------------------
5183

5284
These DataPipes transform elements within DataPipes (e.g. batching, shuffling).
@@ -56,59 +88,112 @@ These DataPipes transform elements within DataPipes (e.g. batching, shuffling).
5688
:toctree: generated/
5789
:template: datapipe.rst
5890

59-
Batcher
60-
BucketBatcher
61-
Shuffler
91+
FlatMapper
92+
Mapper
6293

63-
Utility DataPipes
94+
Selecting DataPipes
6495
-------------------------
6596

66-
These DataPipes provide utility functions (e.g. caching, CSV parsing, filtering).
97+
These DataPipes helps you select specific samples.
6798

6899
.. autosummary::
69100
:nosignatures:
70101
:toctree: generated/
71102
:template: datapipe.rst
72103

73-
CSVDictParser
74-
CSVParser
75-
Collator
76-
Concater
77-
Cycler
78-
DataFrameMaker
79-
Demultiplexer
80-
EndOnDiskCacheHolder
81-
Enumerator
82-
Extractor
83104
Filter
84-
FlatMapper
85-
Forker
86-
Grouper
87-
HashChecker
88105
Header
89-
InMemoryCacheHolder
106+
107+
Augmenting DataPipes
108+
-----------------------------
109+
These DataPipes help to augment your samples.
110+
111+
.. autosummary::
112+
:nosignatures:
113+
:toctree: generated/
114+
:template: datapipe.rst
115+
116+
Cycler
117+
Enumerator
90118
IndexAdder
91-
IoPathSaver
92-
IterKeyZipper
93-
IterableWrapper
119+
120+
Combinatorial DataPipes
121+
-----------------------------
122+
These DataPipes help to perform combinatorial operations.
123+
124+
.. autosummary::
125+
:nosignatures:
126+
:toctree: generated/
127+
:template: datapipe.rst
128+
129+
Sampler
130+
Shuffler
131+
132+
Text DataPipes
133+
-----------------------------
134+
These DataPipes help you parse and read text files.
135+
136+
.. autosummary::
137+
:nosignatures:
138+
:toctree: generated/
139+
:template: datapipe.rst
140+
141+
CSVDictParser
142+
CSVParser
94143
JsonParser
95144
LineReader
96-
MapKeyZipper
97-
Mapper
98-
Multiplexer
99-
OnDiskCacheHolder
100145
ParagraphAggregator
101-
RarArchiveLoader
102146
RoutedDecoder
103147
Rows2Columnar
104-
SampleMultiplexer
105-
Sampler
106-
Saver
107-
ShardingFilter
108148
StreamReader
109-
TarArchiveReader
149+
150+
Grouping DataPipes
151+
-----------------------------
152+
These DataPipes have you group samples within a DataPipe.
153+
154+
.. autosummary::
155+
:nosignatures:
156+
:toctree: generated/
157+
:template: datapipe.rst
158+
159+
Batcher
160+
BucketBatcher
161+
Collator
162+
Grouper
110163
UnBatcher
164+
165+
Combining/Spliting DataPipes
166+
-----------------------------
167+
These tend to involve multiple DataPipes and help combining them or spliting one to many.
168+
169+
.. autosummary::
170+
:nosignatures:
171+
:toctree: generated/
172+
:template: datapipe.rst
173+
174+
Concater
175+
Demultiplexer
176+
Forker
177+
IterKeyZipper
178+
MapKeyZipper
179+
Multiplexer
180+
SampleMultiplexer
111181
UnZipper
112-
XzFileReader
113-
ZipArchiveReader
114182
Zipper
183+
184+
Other DataPipes
185+
-------------------------
186+
A miscellaneous set of DataPipes with different functionalities.
187+
188+
.. autosummary::
189+
:nosignatures:
190+
:toctree: generated/
191+
:template: datapipe.rst
192+
193+
DataFrameMaker
194+
EndOnDiskCacheHolder
195+
HashChecker
196+
InMemoryCacheHolder
197+
IterableWrapper
198+
OnDiskCacheHolder
199+
ShardingFilter

0 commit comments

Comments
 (0)