Skip to content

Commit 411673e

Browse files
burgerkingeaterburgerkingeater
committed
add avro tutorial
1 parent e2bb58f commit 411673e

File tree

3 files changed

+363
-0
lines changed

3 files changed

+363
-0
lines changed

‎docs/tutorials/avro.ipynb

Lines changed: 362 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,362 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"id": "Tce3stUlHN0L"
7+
},
8+
"source": [
9+
"##### Copyright 2020 The TensorFlow IO Authors."
10+
]
11+
},
12+
{
13+
"cell_type": "code",
14+
"execution_count": null,
15+
"metadata": {
16+
"cellView": "form",
17+
"id": "tuOe1ymfHZPu"
18+
},
19+
"outputs": [],
20+
"source": [
21+
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
22+
"# you may not use this file except in compliance with the License.\n",
23+
"# You may obtain a copy of the License at\n",
24+
"#\n",
25+
"# https://www.apache.org/licenses/LICENSE-2.0\n",
26+
"#\n",
27+
"# Unless required by applicable law or agreed to in writing, software\n",
28+
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
29+
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
30+
"# See the License for the specific language governing permissions and\n",
31+
"# limitations under the License."
32+
]
33+
},
34+
{
35+
"cell_type": "markdown",
36+
"metadata": {
37+
"id": "qFdPvlXBOdUN"
38+
},
39+
"source": [
40+
"# Avro Dataset API"
41+
]
42+
},
43+
{
44+
"cell_type": "markdown",
45+
"metadata": {
46+
"id": "MfBg1C5NB3X0"
47+
},
48+
"source": [
49+
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
50+
" <td>\n",
51+
" <a target=\"_blank\" href=\"https://www.tensorflow.org/io/tutorials/avro\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
52+
" </td>\n",
53+
" <td>\n",
54+
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/io/blob/master/docs/tutorials/avro.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
55+
" </td>\n",
56+
" <td>\n",
57+
" <a target=\"_blank\" href=\"https://github.com/tensorflow/io/blob/master/docs/tutorials/avro.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
58+
" </td>\n",
59+
" <td>\n",
60+
" <a href=\"https://storage.googleapis.com/tensorflow_docs/io/docs/tutorials/avro.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
61+
" </td>\n",
62+
"</table>"
63+
]
64+
},
65+
{
66+
"cell_type": "markdown",
67+
"metadata": {
68+
"id": "xHxb-dlhMIzW"
69+
},
70+
"source": [
71+
"## Overview\n",
72+
"\n",
73+
"The objective of Avro Dataset API is to load Avro formatted data natively into TensorFlow. \n"
74+
]
75+
},
76+
{
77+
"cell_type": "markdown",
78+
"metadata": {
79+
"id": "upgCc3gXybsA"
80+
},
81+
"source": [
82+
"### Install required Packages"
83+
]
84+
},
85+
{
86+
"cell_type": "code",
87+
"execution_count": null,
88+
"metadata": {
89+
"id": "uUDYyMZRfkX4"
90+
},
91+
"outputs": [],
92+
"source": [
93+
"!pip install tensorflow-io"
94+
]
95+
},
96+
{
97+
"cell_type": "markdown",
98+
"metadata": {
99+
"id": "J0ZKhA6s0Pjp"
100+
},
101+
"source": [
102+
"## Usage"
103+
]
104+
},
105+
{
106+
"cell_type": "markdown",
107+
"metadata": {
108+
"id": "IGnbXuVnSo8T"
109+
},
110+
"source": [
111+
"Download a sample Avro file:"
112+
]
113+
},
114+
{
115+
"cell_type": "code",
116+
"execution_count": null,
117+
"metadata": {
118+
"id": "Tu01THzWcE-J"
119+
},
120+
"outputs": [],
121+
"source": [
122+
"!curl -OL https://github.com/tensorflow/io/raw/master/docs/tutorials/avro/mnist.avro\n",
123+
"!ls -l mnist.avro"
124+
]
125+
},
126+
{
127+
"cell_type": "markdown",
128+
"metadata": {
129+
"id": "IGnbXuVnSo8T"
130+
},
131+
"source": [
132+
"Download the schema file corresponding for the sample Avro data:"
133+
]
134+
},
135+
{
136+
"cell_type": "code",
137+
"execution_count": null,
138+
"metadata": {
139+
"id": "Tu01THzWcE-J"
140+
},
141+
"outputs": [],
142+
"source": [
143+
"!curl -OL https://github.com/tensorflow/io/raw/master/docs/tutorials/avro/mnist.avsc\n",
144+
"!ls -l mnist.avsc"
145+
]
146+
},
147+
{
148+
"cell_type": "code",
149+
"execution_count": null,
150+
"metadata": {
151+
"id": "nS3eTBvjt-O5"
152+
},
153+
"outputs": [],
154+
"source": [
155+
"import tensorflow as tf\n",
156+
"import tensorflow_io as tfio\n",
157+
"\n",
158+
"features = {\n",
159+
" 'features[*]': tfio.experimental.columnar.VarLenFeatureWithRank(dtype=tf.int32),\n",
160+
" 'label': tf.io.FixedLenFeature(shape=[], dtype=tf.int32, default_value=-100),\n",
161+
" 'dataType': tf.io.FixedLenFeature(shape=[], dtype=tf.string)\n",
162+
"}\n",
163+
"\n",
164+
"schema = tf.io.gfile.GFile('mnist.avsc').read()\n",
165+
"\n",
166+
"dataset = tfio.experimental.columnar.make_avro_record_dataset(file_pattern=['mnist.avro'],\n",
167+
" reader_schema=schema,\n",
168+
" features=features,\n",
169+
" shuffle=False,\n",
170+
" batch_size=3,\n",
171+
" num_epochs=1)\n",
172+
"\n",
173+
"for _ in dataset:\n",
174+
" print(_['features[*]'])\n",
175+
" print(_['label'])\n",
176+
" print(_['dataType'])\n",
177+
" print(\"--------------------\")\n"
178+
]
179+
},
180+
{
181+
"cell_type": "markdown",
182+
"metadata": {
183+
"id": "z9GCyPWNuOm7"
184+
},
185+
"source": [
186+
"In the above example, The avro file `mnist.avro` has 3 fields for each record: `features`, which is an array of int, `label`, an int or null, and `dataType`, an enum. Here is what `mnist.avro` looks like\n"
187+
]
188+
},
189+
{
190+
"cell_type": "code",
191+
"execution_count": null,
192+
"metadata": {
193+
"id": "nS3eTBvjt-O5"
194+
},
195+
"outputs": [],
196+
"source": [
197+
"record 1:\n",
198+
"{\n",
199+
"\"features\" : [0, 0, 0, 1, 4]\n",
200+
"\"label\" : null\n",
201+
"\"dataType\" : \"TRAINING\"\n",
202+
"}\n",
203+
"record 2:\n",
204+
"{\n",
205+
"\"features\" : [0, 0]\n",
206+
"\"label\" : {\"int\" : 2}\n",
207+
"\"dataType\" : \"TRAINING\"\n",
208+
"}\n",
209+
"record 3:\n",
210+
"{\n",
211+
"\"features\" : [0]\n",
212+
"\"label\" : {\"int\" : 3}\n",
213+
"\"dataType\" : \"VALIDATION\"\n",
214+
"}\n",
215+
"record 4:\n",
216+
"{\n",
217+
"\"features\" : [1]\n",
218+
"\"label\" : {\"int\" : 4}\n",
219+
"\"dataType\" : \"VALIDATION\"\n",
220+
"}\n"
221+
]
222+
},
223+
{
224+
"cell_type": "markdown",
225+
"metadata": {
226+
"id": "z9GCyPWNuOm7"
227+
},
228+
"source": [
229+
"And the schema of `mnist.avro` which is represented by `mnist.avsc` looks like following \n"
230+
]
231+
},
232+
{
233+
"cell_type": "code",
234+
"execution_count": null,
235+
"metadata": {
236+
"id": "nS3eTBvjt-O5"
237+
},
238+
"outputs": [],
239+
"source": [
240+
"{ \n",
241+
" \"type\" : \"record\"\n",
242+
" \"name\" : \"ImageDataset\"\n",
243+
" \"fields\" : [ {\n",
244+
" \"name\" : \"features\",\n",
245+
" \"type\" : {\n",
246+
" \"type\" : \"array\",\n",
247+
" \"items\" : \"int\"\n",
248+
" }\n",
249+
" }, {\n",
250+
" \"name\" : \"label\",\n",
251+
" \"type\" : [\"int\", \"null\"] //label can be int or null\n",
252+
" }, {\n",
253+
" \"name\" : \"dataType\",\n",
254+
" \"type\" : {\n",
255+
" \"type\" : \"enum\",\n",
256+
" \"name\" : \"dataTypes\",\n",
257+
" \"symbols\" : [\"TRAINING\", \"VALIDATION\"]\n",
258+
" }\n",
259+
" } ]\n",
260+
"}"
261+
]
262+
},
263+
{
264+
"cell_type": "markdown",
265+
"metadata": {
266+
"id": "IF_kYz_o2DH4"
267+
},
268+
"source": [
269+
"The above example converts `mnist.avro` into tensorflow dataset. Each element of the dataset is a dictionary whose key is the feature name, value is the converted sparse or dense tensor. \n",
270+
"E.g, it converts `features`, `label`, `dataType` field to a VarLenFeature(SparseTensor), FixedLenFeature(DenseTensor), and FixedLenFeature(DenseTensor) respectively. Since batch_size is 3, it coerce 3 records from `mnist.avro` into one element in the result dataset.\n",
271+
"For the first record in `mnist.avro` whose label is null, avro reader replaces it with the specified default value(-100).\n",
272+
"In this example, there're 4 records in total in `mnist.avro`. Since batch size is 3, the result dataset contains 3 elements, last of which's batch size is 1. However user is also able to drop the last batch if the size is smaller than batch size by enabling `drop_final_batch`. E.g: \n"
273+
]
274+
},
275+
{
276+
"cell_type": "code",
277+
"execution_count": null,
278+
"metadata": {
279+
"id": "nS3eTBvjt-O5"
280+
},
281+
"outputs": [],
282+
"source": [
283+
"dataset = tfio.experimental.columnar.make_avro_record_dataset(file_pattern=['mnist.avro'],\n",
284+
" reader_schema=schema,\n",
285+
" features=features,\n",
286+
" shuffle=False,\n",
287+
" batch_size=3,\n",
288+
" drop_final_batch=True,\n",
289+
" num_epochs=1)\n",
290+
"\n",
291+
"for _ in dataset:\n ",
292+
" print(_)\n"
293+
]
294+
},
295+
{
296+
"cell_type": "markdown",
297+
"metadata": {
298+
"id": "IF_kYz_o2DH4"
299+
},
300+
"source": [
301+
"Please refer to <a target=\"_blank\" href=\"https://www.tensorflow.org/io/api_docs/python/tfio/experimental/columnar/make_avro_record_dataset\">API doc</a> for the detailed usage of `make_avro_record_dataset.`\n"
302+
]
303+
},
304+
{
305+
"cell_type": "markdown",
306+
"metadata": {
307+
"id": "IF_kYz_o2DH4"
308+
},
309+
"source": [
310+
"The avro dataset can parse and coerce any avro data into TensorFlow tensors, including records in records, maps, arrays, branches, and enumerations. The parsing information is passed into the avro dataset implementation as a map where \n",
311+
"keys encode how to parse the data \n",
312+
"values encode on how to coerce the data into TensorFlow tensors – deciding the primitive type (e.g. bool, int, long, float, double, string) as well as the tensor type (e.g. sparse or dense). A listing of TensorFlow's parser types (see Table 1) and the coercion of primitive types (Table 2) is provided. \n",
313+
"\n",
314+
"Table 1:\n",
315+
"\n",
316+
"TensorFlow Parser Types|TensorFlow Tensors|Explanation\n",
317+
"----|----|------\n",
318+
"tf.FixedLenFeature([], tf.int32)|dense tensor|Parse a fixed length feature; that is all rows have the same constant number of elements, e.g. just one element or an array that has always the same number of elements for each row \n",
319+
"tf.SparseFeature(index_key=['key_1st_index', 'key_2nd_index'], value_key='key_value', dtype=tf.int64, size=[20, 50]) |sparse tensor|Parse a sparse feature where each row has a variable length list of indices and values. The 'index_key' identifies the indices. The 'value_key' identifies the value. The 'dtype' is the data type. The 'size' is the expected maximum index value for each index entry\n",
320+
"tfio.experimental.columnar.VarLenFeatureWithRank([],tf.int64) |sparse tensor|Parse a variable length feature; that means each data row can have a variable number of elements, e.g. the 1st row has 5 elements, the 2nd row has 7 elements\n",
321+
"\n",
322+
"Table 2 the supported conversion from Avro types to TensorFlow's types:\n",
323+
"\n",
324+
"Avro Primitive Type|TensorFlow Primitive Type\n",
325+
"----|----\n",
326+
"boolean: a binary value|tf.bool\n",
327+
"bytes: a sequence of 8-bit unsigned bytes|tf.string\n",
328+
"double: double precision 64-bit IEEE floating point number|tf.float64\n",
329+
"enum: enumeration type|tf.string using the symbol name\n",
330+
"float: single precision 32-bit IEEE floating point number|tf.float32\n",
331+
"int: 32-bit signed integer|tf.int32\n",
332+
"long: 64-bit signed integer|tf.int64\n",
333+
"null: no value|uses default value\n",
334+
"string: unicode character sequence|tf.string\n"
335+
]
336+
},
337+
{
338+
"cell_type": "markdown",
339+
"metadata": {
340+
"id": "IF_kYz_o2DH4"
341+
},
342+
"source": [
343+
"A comprehensive set of examples of Avro dataset API is provided within <a target=\"_blank\" href=\"https://github.com/tensorflow/io/blob/master/tests/test_parse_avro_eager.py#L580r\">the tests.</a>\n"
344+
]
345+
}
346+
],
347+
"metadata": {
348+
"colab": {
349+
"collapsed_sections": [
350+
"Tce3stUlHN0L"
351+
],
352+
"name": "avro.ipynb",
353+
"toc_visible": true
354+
},
355+
"kernelspec": {
356+
"display_name": "Python 3",
357+
"name": "python3"
358+
}
359+
},
360+
"nbformat": 4,
361+
"nbformat_minor": 0
362+
}

‎docs/tutorials/avro/mnist.avro

369 Bytes
Binary file not shown.

‎docs/tutorials/avro/mnist.avsc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"name": "ImageDataset", "type": "record", "fields": [{"name": "features", "type": {"type": "array", "items": "int"}}, {"name": "label", "type": ["int", "null"]}, {"name": "dataType", "type": {"type": "enum", "name": "dataTypes", "symbols": ["TRAINING", "VALIDATION"]}}]}

0 commit comments

Comments
 (0)