Skip to content

Commit 7f5a3ba

Browse files
burgerkingeaterburgerkingeater
committed
add avro tutorial
1 parent e2bb58f commit 7f5a3ba

File tree

5 files changed

+263
-0
lines changed

5 files changed

+263
-0
lines changed

‎docs/tutorials/avro.ipynb

Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"id": "Tce3stUlHN0L"
7+
},
8+
"source": [
9+
"##### Copyright 2020 The TensorFlow IO Authors."
10+
]
11+
},
12+
{
13+
"cell_type": "code",
14+
"execution_count": null,
15+
"metadata": {
16+
"cellView": "form",
17+
"id": "tuOe1ymfHZPu"
18+
},
19+
"outputs": [],
20+
"source": [
21+
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
22+
"# you may not use this file except in compliance with the License.\n",
23+
"# You may obtain a copy of the License at\n",
24+
"#\n",
25+
"# https://www.apache.org/licenses/LICENSE-2.0\n",
26+
"#\n",
27+
"# Unless required by applicable law or agreed to in writing, software\n",
28+
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
29+
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
30+
"# See the License for the specific language governing permissions and\n",
31+
"# limitations under the License."
32+
]
33+
},
34+
{
35+
"cell_type": "markdown",
36+
"metadata": {
37+
"id": "qFdPvlXBOdUN"
38+
},
39+
"source": [
40+
"# Avro Dataset API"
41+
]
42+
},
43+
{
44+
"cell_type": "markdown",
45+
"metadata": {
46+
"id": "MfBg1C5NB3X0"
47+
},
48+
"source": [
49+
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
50+
" <td>\n",
51+
" <a target=\"_blank\" href=\"https://www.tensorflow.org/io/tutorials/avro\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
52+
" </td>\n",
53+
" <td>\n",
54+
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/io/blob/master/docs/tutorials/avro.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
55+
" </td>\n",
56+
" <td>\n",
57+
" <a target=\"_blank\" href=\"https://github.com/tensorflow/io/blob/master/docs/tutorials/avro.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
58+
" </td>\n",
59+
" <td>\n",
60+
" <a href=\"https://storage.googleapis.com/tensorflow_docs/io/docs/tutorials/avro.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
61+
" </td>\n",
62+
"</table>"
63+
]
64+
},
65+
{
66+
"cell_type": "markdown",
67+
"metadata": {
68+
"id": "xHxb-dlhMIzW"
69+
},
70+
"source": [
71+
"## Overview\n",
72+
"\n",
73+
"The objective of Avro Dataset API is to load Avro formatted data natively into TensorFlow. \n"
74+
]
75+
},
76+
{
77+
"cell_type": "markdown",
78+
"metadata": {
79+
"id": "upgCc3gXybsA"
80+
},
81+
"source": [
82+
"### Install required Packages"
83+
]
84+
},
85+
{
86+
"cell_type": "code",
87+
"execution_count": null,
88+
"metadata": {
89+
"id": "uUDYyMZRfkX4"
90+
},
91+
"outputs": [],
92+
"source": [
93+
"!pip install tensorflow-io"
94+
]
95+
},
96+
{
97+
"cell_type": "markdown",
98+
"metadata": {
99+
"id": "J0ZKhA6s0Pjp"
100+
},
101+
"source": [
102+
"## Usage"
103+
]
104+
},
105+
{
106+
"cell_type": "markdown",
107+
"metadata": {
108+
"id": "IGnbXuVnSo8T"
109+
},
110+
"source": [
111+
"Download a sample Avro file:"
112+
]
113+
},
114+
{
115+
"cell_type": "code",
116+
"execution_count": null,
117+
"metadata": {
118+
"id": "Tu01THzWcE-J"
119+
},
120+
"outputs": [],
121+
"source": [
122+
"!curl -OL https://github.com/tensorflow/io/raw/master/docs/tutorials/avro/mnist.avro\n",
123+
"!ls -l mnist.avro"
124+
]
125+
},
126+
{
127+
"cell_type": "markdown",
128+
"metadata": {
129+
"id": "IGnbXuVnSo8T"
130+
},
131+
"source": [
132+
"Download the schema file corresponding for the sample Avro data:"
133+
]
134+
},
135+
{
136+
"cell_type": "code",
137+
"execution_count": null,
138+
"metadata": {
139+
"id": "Tu01THzWcE-J"
140+
},
141+
"outputs": [],
142+
"source": [
143+
"!curl -OL https://github.com/tensorflow/io/raw/master/docs/tutorials/avro/mnist.avsc\n",
144+
"!ls -l mnist.avsc"
145+
]
146+
},
147+
{
148+
"cell_type": "code",
149+
"execution_count": null,
150+
"metadata": {
151+
"id": "nS3eTBvjt-O5"
152+
},
153+
"outputs": [],
154+
"source": [
155+
"import tensorflow as tf\n",
156+
"import tensorflow_io as tfio\n",
157+
"\n",
158+
"features = {\n",
159+
" 'features[*]': tfio.experimental.columnar.VarLenFeatureWithRank(dtype=tf.int32),\n",
160+
" 'label': tf.io.FixedLenFeature(shape=[], dtype=tf.int32),\n",
161+
" 'dataType': tf.io.FixedLenFeature(shape=[], dtype=tf.string)\n",
162+
"}\n",
163+
"\n",
164+
"schema = tf.io.gfile.GFile('mnist.avsc').read()\n",
165+
"\n",
166+
"dataset = tfio.experimental.columnar.make_avro_record_dataset(file_pattern=['mnist.avro'],\n",
167+
" reader_schema=schema,\n",
168+
" features=features,\n",
169+
" batch_size=2,\n",
170+
" num_epochs=1)\n",
171+
"\n",
172+
"for _ in dataset:\n ",
173+
" print(_)\n"
174+
]
175+
},
176+
{
177+
"cell_type": "markdown",
178+
"metadata": {
179+
"id": "z9GCyPWNuOm7"
180+
},
181+
"source": [
182+
"In the above example, The avro file `mnist.avro` has 3 fields for each record: `features`, which is an array of int, `label`, an int, and `dataType`, an enum. Here is a sample record from `mnist.avro`\n",
183+
"<table class=\"avro-record-sample\" align=\"center\">\n",
184+
" <td> <a target=\"_blank\"><img src=\"https://github.com/tensorflow/io/raw/master/docs/tutorials/avro/one_record_in_mnist.png\">first record in mnist.avro</a>\n",
185+
" </td>\n",
186+
"</table>\n",
187+
"And the schema of `mnist.avro` which is represented by `mnist.avsc` is \n",
188+
"<table class=\"avro-schema\" align=\"center\">\n",
189+
" <td> <a target=\"_blank\"><img src=\"https://github.com/tensorflow/io/raw/master/docs/tutorials/avro/avro_schema.png\" width=\"300\" height=\"300\">the schema of mnist.avro</a>\n",
190+
" </td>\n",
191+
"</table>\n"
192+
]
193+
},
194+
{
195+
"cell_type": "markdown",
196+
"metadata": {
197+
"id": "IF_kYz_o2DH4"
198+
},
199+
"source": [
200+
"The above example converts `features`, `label`, `dataType` field to a VarLenFeature, FixedLenFeature, and FixedLenFeature respectively.\n",
201+
"Please refer to <a target=\"_blank\" href=\"https://www.tensorflow.org/io/api_docs/python/tfio/experimental/columnar/make_avro_record_dataset\">API doc</a> for the detailed usage of `make_avro_record_dataset.`\n"
202+
]
203+
},
204+
{
205+
"cell_type": "markdown",
206+
"metadata": {
207+
"id": "IF_kYz_o2DH4"
208+
},
209+
"source": [
210+
"The avro dataset can parse and coerce any avro data into TensorFlow tensors, including records in records, maps, arrays, branches, and enumerations. The parsing information is passed into the avro dataset implementation as a map where \n",
211+
"keys encode how to parse the data \n",
212+
"values encode on how to coerce the data into TensorFlow tensors – deciding the primitive type (e.g. bool, int, long, float, double, string) as well as the tensor type (e.g. sparse or dense). A listing of TensorFlow's parser types (see Table 1) and the coercion of primitive types (Table 2) is provided. \n",
213+
"\n",
214+
"Table 1:\n",
215+
"\n",
216+
"TensorFlow Parser Types|TensorFlow Tensors|Explanation\n",
217+
"----|----|------\n",
218+
"tf.FixedLenFeature([], tf.int32)|dense tensor|Parse a fixed length feature; that is all rows have the same constant number of elements, e.g. just one element or an array that has always the same number of elements for each row \n",
219+
"tf.SparseFeature(index_key=['key_1st_index', 'key_2nd_index'], value_key='key_value', dtype=tf.int64, size=[20, 50]) |sparse tensor|Parse a sparse feature where each row has a variable length list of indices and values. The 'index_key' identifies the indices. The 'value_key' identifies the value. The 'dtype' is the data type. The 'size' is the expected maximum index value for each index entry\n",
220+
"tf.VarLenFeature([],tf.int64) |sparse tensor|Parse a variable length feature; that means each data row can have a variable number of elements, e.g. the 1st row has 5 elements, the 2nd row has 7 elements\n",
221+
"\n",
222+
"Table 2:\n",
223+
"\n",
224+
"Avro Primitive Type|TensorFlow Primitive Type\n",
225+
"----|----\n",
226+
"boolean: a binary value|tf.bool\n",
227+
"bytes: a sequence of 8-bit unsigned bytes|tf.string\n",
228+
"double: double precision 64-bit IEEE floating point number|tf.float64\n",
229+
"enum: enumeration type|tf.string using the symbol name\n",
230+
"float: single precision 32-bit IEEE floating point number|tf.float32\n",
231+
"int: 32-bit signed integer|tf.int32\n",
232+
"long: 64-bit signed integer|tf.int64\n",
233+
"null: no value|uses default value\n",
234+
"string: unicode character sequence|tf.string\n"
235+
]
236+
},
237+
{
238+
"cell_type": "markdown",
239+
"metadata": {
240+
"id": "IF_kYz_o2DH4"
241+
},
242+
"source": [
243+
"A comprehensive set of examples of Avro dataset API is provided within <a target=\"_blank\" href=\"https://github.com/tensorflow/io/blob/master/tests/test_parse_avro_eager.py#L580r\">the tests</a>\n"
244+
]
245+
}
246+
],
247+
"metadata": {
248+
"colab": {
249+
"collapsed_sections": [
250+
"Tce3stUlHN0L"
251+
],
252+
"name": "avro.ipynb",
253+
"toc_visible": true
254+
},
255+
"kernelspec": {
256+
"display_name": "Python 3",
257+
"name": "python3"
258+
}
259+
},
260+
"nbformat": 4,
261+
"nbformat_minor": 0
262+
}

‎docs/tutorials/avro/avro_schema.png

87.8 KB
Loading

‎docs/tutorials/avro/mnist.avro

178 KB
Binary file not shown.

‎docs/tutorials/avro/mnist.avsc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"name": "ImageDataset", "type": "record", "fields": [{"name": "features", "type": {"type": "array", "items": "int"}}, {"name": "label", "type": "int"}, {"name": "dataType", "type": {"type": "enum", "name": "dataTypes", "symbols": ["TRAINING", "VALIDATION"]}}]}

‎docs/tutorials/avro/one_record_in_mnist.png

258 KB
Loading

0 commit comments

Comments
 (0)