Fine-Tuning Fails With Exception Between Epoch1 and Epoch2

[shashankiyer]

I have been trying to use this code to fine-tune the network to classify images from the Cifar10 dataset. However, I get the following error:

Traceback (most recent call last):
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Nan in summary histogram for: fc6/weights_0
[[{{node fc6/weights_0}} = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](fc6/weights_0/tag, fc6/weights/read)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "finetune.py", line 202, in
keep_prob: 1.})
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Nan in summary histogram for: fc6/weights_0
[[node fc6/weights_0 (defined at finetune.py:137) = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](fc6/weights_0/tag, fc6/weights/read)]]

Caused by op 'fc6/weights_0', defined at:
File "finetune.py", line 137, in
tf.summary.histogram(var.name, var)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/summary/summary.py", line 187, in histogram
tag=tag, values=values, name=scope)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 284, in histogram_summary
"HistogramSummary", tag=tag, values=values, name=name)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/shashankiyer/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Nan in summary histogram for: fc6/weights_0
[[node fc6/weights_0 (defined at finetune.py:137) = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](fc6/weights_0/tag, fc6/weights/read)]]

These are lines in the code that cause this:

//Add gradients to summary
for gradient, var in grads_and_vars:
tf.summary.histogram(var.name + '/gradient', gradient)

//Add the variables we train to the summary
for var in var_list:
tf.summary.histogram(var.name, var)

I am running Tensorflow 1.12.0
Any pointers will be greatly appreciated.

[kratzert]

NaN values are almost always a hint that your learning rate ist to high. Try to decrease ist to e.g. 1e-3 or 1e-4