@@ -17,16 +17,31 @@ This is an updated version of ``IterableDataset`` in ``torch``.
17
17
18
18
We have three types of Iterable DataPipes:
19
19
20
- 1. Load - help you interact with the file systems or online databases (e.g. FileOpener, GDriveReader)
20
+ 1. IO - DESCRIPTION
21
21
22
- 2. Transform - transform elements within DataPipes (e.g. batching, shuffling)
22
+ 2. Archive - DESCRIPTION
23
23
24
- 3. Utility - utility functions (e.g. caching, CSV parsing, filtering)
24
+ 3. Mapping - DESCRIPTION
25
25
26
- Load DataPipes
26
+ 4. Selecting - DESCRIPTION
27
+
28
+ 5. Augmenting - DESCRIPTION
29
+
30
+ 6. Combinatorial - DESCRIPTION
31
+
32
+ 7. Text - DESCRIPTION
33
+
34
+ 8. Grouping - DESCRIPTION
35
+
36
+ 9. Combining/Splitting -DESCRIPTION
37
+
38
+ 10. Others - DESCRIPTION
39
+
40
+ IO DataPipes
27
41
-------------------------
28
42
29
- These DataPipes help you interact with the file systems or online databases (e.g. FileOpener, GDriveReader).
43
+ These DataPipes help interacting with the file systems or online databases (e.g. downloading, opening,
44
+ saving files, and listing the files in directories).
30
45
31
46
.. autosummary ::
32
47
:nosignatures:
@@ -42,11 +57,28 @@ These DataPipes help you interact with the file systems or online databases (e.g
42
57
HttpReader
43
58
IoPathFileLister
44
59
IoPathFileOpener
60
+ IoPathSaver
45
61
OnlineReader
46
62
ParquetDataFrameLoader
63
+ Saver
64
+
65
+ Archive DataPipes
66
+ -------------------------
67
+
68
+ These DataPipes help opening and decompressing archive files.
47
69
70
+ .. autosummary ::
71
+ :nosignatures:
72
+ :toctree: generated/
73
+ :template: datapipe.rst
48
74
49
- Transform DataPipes
75
+ Extractor
76
+ RarArchiveLoader
77
+ TarArchiveReader
78
+ XzFileReader
79
+ ZipArchiveReader
80
+
81
+ Mapping DataPipes
50
82
-------------------------
51
83
52
84
These DataPipes transform elements within DataPipes (e.g. batching, shuffling).
@@ -56,59 +88,112 @@ These DataPipes transform elements within DataPipes (e.g. batching, shuffling).
56
88
:toctree: generated/
57
89
:template: datapipe.rst
58
90
59
- Batcher
60
- BucketBatcher
61
- Shuffler
91
+ FlatMapper
92
+ Mapper
62
93
63
- Utility DataPipes
94
+ Selecting DataPipes
64
95
-------------------------
65
96
66
- These DataPipes provide utility functions (e.g. caching, CSV parsing, filtering) .
97
+ These DataPipes helps you select specific samples .
67
98
68
99
.. autosummary ::
69
100
:nosignatures:
70
101
:toctree: generated/
71
102
:template: datapipe.rst
72
103
73
- CSVDictParser
74
- CSVParser
75
- Collator
76
- Concater
77
- Cycler
78
- DataFrameMaker
79
- Demultiplexer
80
- EndOnDiskCacheHolder
81
- Enumerator
82
- Extractor
83
104
Filter
84
- FlatMapper
85
- Forker
86
- Grouper
87
- HashChecker
88
105
Header
89
- InMemoryCacheHolder
106
+
107
+ Augmenting DataPipes
108
+ -----------------------------
109
+ These DataPipes help to augment your samples.
110
+
111
+ .. autosummary ::
112
+ :nosignatures:
113
+ :toctree: generated/
114
+ :template: datapipe.rst
115
+
116
+ Cycler
117
+ Enumerator
90
118
IndexAdder
91
- IoPathSaver
92
- IterKeyZipper
93
- IterableWrapper
119
+
120
+ Combinatorial DataPipes
121
+ -----------------------------
122
+ These DataPipes help to perform combinatorial operations.
123
+
124
+ .. autosummary ::
125
+ :nosignatures:
126
+ :toctree: generated/
127
+ :template: datapipe.rst
128
+
129
+ Sampler
130
+ Shuffler
131
+
132
+ Text DataPipes
133
+ -----------------------------
134
+ These DataPipes help you parse and read text files.
135
+
136
+ .. autosummary ::
137
+ :nosignatures:
138
+ :toctree: generated/
139
+ :template: datapipe.rst
140
+
141
+ CSVDictParser
142
+ CSVParser
94
143
JsonParser
95
144
LineReader
96
- MapKeyZipper
97
- Mapper
98
- Multiplexer
99
- OnDiskCacheHolder
100
145
ParagraphAggregator
101
- RarArchiveLoader
102
146
RoutedDecoder
103
147
Rows2Columnar
104
- SampleMultiplexer
105
- Sampler
106
- Saver
107
- ShardingFilter
108
148
StreamReader
109
- TarArchiveReader
149
+
150
+ Grouping DataPipes
151
+ -----------------------------
152
+ These DataPipes have you group samples within a DataPipe.
153
+
154
+ .. autosummary ::
155
+ :nosignatures:
156
+ :toctree: generated/
157
+ :template: datapipe.rst
158
+
159
+ Batcher
160
+ BucketBatcher
161
+ Collator
162
+ Grouper
110
163
UnBatcher
164
+
165
+ Combining/Spliting DataPipes
166
+ -----------------------------
167
+ These tend to involve multiple DataPipes and help combining them or spliting one to many.
168
+
169
+ .. autosummary ::
170
+ :nosignatures:
171
+ :toctree: generated/
172
+ :template: datapipe.rst
173
+
174
+ Concater
175
+ Demultiplexer
176
+ Forker
177
+ IterKeyZipper
178
+ MapKeyZipper
179
+ Multiplexer
180
+ SampleMultiplexer
111
181
UnZipper
112
- XzFileReader
113
- ZipArchiveReader
114
182
Zipper
183
+
184
+ Other DataPipes
185
+ -------------------------
186
+ A miscellaneous set of DataPipes with different functionalities.
187
+
188
+ .. autosummary ::
189
+ :nosignatures:
190
+ :toctree: generated/
191
+ :template: datapipe.rst
192
+
193
+ DataFrameMaker
194
+ EndOnDiskCacheHolder
195
+ HashChecker
196
+ InMemoryCacheHolder
197
+ IterableWrapper
198
+ OnDiskCacheHolder
199
+ ShardingFilter
0 commit comments