cuda error: no kernel image is available for execution on the device

[swadhinagrawal]

I am using following specifications:

Ubuntu 18.04LTS
GPU = GeForce MX130
CUDA = 10.0.130 (Thats what nvcc -V says)
Driver = NVIDIA-418.67 (CUDA-10.1)(That's what nvidia-smi says)
But I installed only CUDA 10.0
Python = 3.6.7
But GPU is being used by other codes/programs without any problem.

For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From ../python/depthmotionnet/helpers.py:140: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
WARNING:tensorflow:From /home/zorawar/.local/share/virtualenvs/demon_venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From ../python/depthmotionnet/blocks_original.py:73: conv2d_transpose (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d_transpose instead.
WARNING:tensorflow:From /home/zorawar/.local/share/virtualenvs/demon_venv/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py:1624: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
WARNING:tensorflow:From ../python/depthmotionnet/blocks_original.py:395: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
WARNING:tensorflow:From ../python/depthmotionnet/blocks_original.py:165: calling norm (from tensorflow.python.ops.linalg_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /home/zorawar/.local/share/virtualenvs/demon_venv/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-05-30 14:08:18.209430: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
terminate called after throwing an instance of 'std::runtime_error'
  what():  /home/zorawar/Desktop/demon/lmbspecialops/src/warp2d_cuda.cu:218: cuda error: no kernel image is available for execution on the device

Aborted (core dumped)

[valgur]

Sounds like automatically determining the correct compute capability of your GPU failed when building the library. I would suggest cloning the repo and checking the output of python setup.py build. Can you perhaps share the output of this specifically: python setup.py build 2>/dev/null | grep -P 'CUDA|GPU'?

[swadhinagrawal]

After I Installed cmake-setuptools and doing what you told, I am getting the following result:

/home/zorawar/.local/lib/python3.7/site-packages/setuptools/dist.py:481: UserWarning: The version specified ('A collection of tensorflow ops') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  "details." % self.metadata.version
running build
running build_py
running build_ext
-- Found OpenMP_C: -fopenmp  
-- Found OpenMP_CXX: -fopenmp  
-- Found OpenMP: TRUE   
-- /home/zorawar/anaconda3/lib/python3.7/site-packages/tensorflow/include
-- /home/zorawar/anaconda3/lib/python3.7/site-packages/tensorflow
-- /home/zorawar/anaconda3/lib/python3.7/site-packages/tensorflow/libtensorflow_framework.so
-- TensorFlow compile definitions: _GLIBCXX_USE_CXX11_ABI=1
CMake Warning (dev) at cmake/cuda_gencode_options.cmake:12 (set):
  implicitly converting 'LIST' to 'STRING' type.
Call Stack (most recent call first):
  src/CMakeLists.txt:46 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Autodetected CUDA architecture(s):  5.0
-- CUDA code generation flags: -gencode;arch=compute_50,code=sm_50
-- found test test_FloOp
-- found test test_FlowToDepth2
-- found test test_LeakyRelu
-- found test test_Lz4
-- found test test_Lz4Raw
-- found test test_Median3x3Downsample
-- found test test_PfmOp
-- found test test_PpmOp
-- found test test_ReplaceNonfinite
-- found test test_ScaleInvariantGradient
-- found test test_WebpOp
-- Configuring done
-- Generating done
-- Build files have been written to: /home/zorawar/Desktop/lmbspecialops-master/build/temp.linux-x86_64-3.7
[ 29%] Built target webp
[ 29%] Built target lz4
[ 31%] Linking CXX shared library ../../lib.linux-x86_64-3.7/lmbspecialops.so
[100%] Built target lmbspecialops

zorawar@zorawar:~/Desktop/lmbspecialops-master$ python setup.py build 2>/dev/null | grep -P 'CUDA|GPU'
-- Autodetected CUDA architecture(s):  5.0
-- CUDA code generation flags: -gencode;arch=compute_50,code=sm_50
zorawar@zorawar:~/Desktop/lmbspecialops-master$ 

[swadhinagrawal]

But it is still not working in DeepTAM, I am getting no output for python setup.py build 2>/dev/null | grep -P 'CUDA|GPU'
Also I am getting the same Kernal image error.

[TheCrazyT]

looks like I'm having the same problem:

terminate called after throwing an instance of 'std::runtime_error'
  what():  /kaggle/demon_v2/lmbspecialops/src/rotation_format.cu:90: cuda error: no kernel image is available for execution on the device

Edit:
Well my error now just went away after recompiling (was changing a file but reverted my change).
I do not know why it was gone, maybe it can happen randomly?