tools/infer_simple.py fail

[oreilc16]

I downloaded a caffe2 docker file from this link: https://github.com/caffe2/caffe2/blob/master/docker/ubuntu-16.04-cuda8-cudnn6-all-options/Dockerfile

then ran the following to build

# Use the latest Caffe2 master
sed -i -e 's/ --branch v0.8.1//g' Dockerfile
docker build -t caffe2:cuda8-cudnn6-all-options .

I then cloned https://github.com/facebookresearch/Detectron/ and built a Detectron image:

cd $DETECTRON/docker
docker build -t detectron:c2-cuda8-cudnn6 .

I ran this docker using the following commands with my devices added:

nvidia-docker run --rm -it --device=/dev/nvidiactl --device=/dev/nvidia-uvm --device=/dev/nvidia0  detectron:c2-cuda8-cudnn6

Inside my docker I ran this (from GETTING_STARTED.md) to verify I could run inference on a directory of image files :

python2 tools/infer_simple.py \
    --cfg configs/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml \
    --output-dir /tmp/detectron-visualizations \
    --image-ext jpg \
    --wts https://s3-us-west-2.amazonaws.com/detectron/35861858/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml.02_32_51.SgT4y1cO/output/train/coco_2014_train:coco_2014_valminusminival/generalized_rcnn/model_final.pkl \
    demo

The fail I got was this:

E0125 13:59:11.822402    20 init_intrinsics_check.cc:54] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
E0125 13:59:11.822424    20 init_intrinsics_check.cc:54] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
E0125 13:59:11.822428    20 init_intrinsics_check.cc:54] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
INFO io.py:  67: Downloading remote file https://s3-us-west-2.amazonaws.com/detectron/ImageNetPretrained/MSRA/R-101.pkl to /tmp/detectron-download-cache/ImageNetPretrained/MSRA/R-101.pkl
  [------------------------------------------------------------] 0.0% of 170.2MB  [------------------------------------------------------------] 0.0% of 170.2MB  [---------------------------------------------
...
...
...
I0125 14:02:33.693230    20 net_dag.cc:61] Number of parallel execution chains 63 Number of operators = 402
I0125 14:02:33.720326    20 net_dag_utils.cc:118] Operator graph pruning prior to chain compute took: 0.000215514 secs
I0125 14:02:33.720594    20 net_dag.cc:61] Number of parallel execution chains 30 Number of operators = 358
I0125 14:02:33.722873    20 net_dag_utils.cc:118] Operator graph pruning prior to chain compute took: 1.6725e-05 secs
I0125 14:02:33.722909    20 net_dag.cc:61] Number of parallel execution chains 5 Number of operators = 18
INFO infer_simple.py: 111: Processing demo/17790319373_bd19b24cfc_k.jpg -> /tmp/detectron-visualizations/17790319373_bd19b24cfc_k.jpg.pdf
terminate called after throwing an instance of 'caffe2::EnforceNotMet'
  what():  [enforce fail at context_gpu.h:170] . Encountered CUDA error: invalid device function Error from operator: 
input: "gpu_0/res2_0_branch2c_bn" input: "gpu_0/res2_0_branch1_bn" output: "gpu_0/res2_0_branch2c_bn" name: "" type: "Sum" device_option { device_type: 1 cuda_gpu_id: 0 }
*** Aborted at 1516888954 (unix time) try "date -d @1516888954" if you are using GNU date ***
PC: @     0x7f8fbb2c0428 gsignal
*** SIGABRT (@0x14) received by PID 20 (TID 0x7f8f656a6700) from PID 20; stack trace: ***
    @     0x7f8fbb2c04b0 (unknown)
    @     0x7f8fbb2c0428 gsignal
    @     0x7f8fbb2c202a abort
    @     0x7f8fb523784d __gnu_cxx::__verbose_terminate_handler()
    @     0x7f8fb52356b6 (unknown)
    @     0x7f8fb5235701 std::terminate()
    @     0x7f8fb5260d38 (unknown)
    @     0x7f8fbb65c6ba start_thread
    @     0x7f8fbb39241d clone
    @                0x0 (unknown)
Aborted (core dumped)

Any help would be much appreciated thank you

[sweetod]

I got this problem when testing my images on Mask R-CNN with Bells & Whistles X-152-32x8d-FPN-IN5k. On some other models it worked well.

[rbgirshick]

@oreilc16: these types of errors are difficult to reproduce. Can you provide the following information which may help us:

• Operating system: ?
• Compiler version: ?
• CUDA version: ?
• cuDNN version: ?
• NVIDIA driver version: ?
• GPU models (for all devices if they are not all the same): ?
• Anything else that seems relevant: ?

@sweetod: can you check the CUDA error carefully and say if it's the same and not different (e.g., out of memory or something else). Thanks.

[sweetod]

@rbgirshick : Sorry for my vague expression, here is my error:

terminate called after throwing an instance of 'caffe2::EnforceNotMet'
what(): [enforce fail at context_gpu.cu:343] error == cudaSuccess. 2 vs 0. Error at: /home/JohnsmithMBP/caffe2/caffe2/core/context_gpu.cu:343: out of memory Error from operator:
input: "gpu_0/mask_fcn_logits" output: "gpu_0/mask_fcn_probs" name: "" type: "Sigmoid" device_option { device_type: 1 cuda_gpu_id: 0 }
*** Aborted at 1516998135 (unix time) try "date -d @1516998135" if you are using GNU date ***
PC: @ 0x7fd63b847428 gsignal
*** SIGABRT (@0x3e9000008aa) received by PID 2218 (TID 0x7fd5e0564700) from PID 2218; stack trace: ***
@ 0x7fd63b8474b0 (unknown)
@ 0x7fd63b847428 gsignal
@ 0x7fd63b84902a abort
@ 0x7fd63562684d __gnu_cxx::__verbose_terminate_handler()
@ 0x7fd6356246b6 (unknown)
@ 0x7fd635624701 std::terminate()
@ 0x7fd63564fd38 (unknown)
@ 0x7fd63bbe36ba start_thread
@ 0x7fd63b91941d clone
@ 0x0 (unknown)
Aborted (core dumped)

and here is my environment:

• gcc (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609

• Driver Version: 384.111

• CUDA Version 8.0.61

• cuDNN 6.0

Thanks!

[rbgirshick]

@sweetod: this is indeed a different error. The bells and whistles model currently requires a 16GB GPU, but we plan to improve inference memory efficiency in the future.

[sweetod]

@rbgirshick: Indeed I'm using a Tesla K80(24GB), and since I resize image from 16001200 to 800600 it worked. Thanks for your patience anyway!

[rbgirshick]

Closing because no further details were provided.

[oreilc16]

@rbgirshick: I am based in Ireland, so I didn't see your reply as it was Friday night here. I can now provide the details requested:

Operating system: Ubuntu 16.04 LTS
Compiler version:
CUDA version: V8.0.61
cuDNN version: 6.0.21
NVIDIA driver version: NVIDIA-SMI 384.111
GPU models (for all devices if they are not all the same): Quadro M1000M/PCIe/SSE2 2GB

When checking my cuDNN version I noticed I could not run the following command in my docker

cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2

This is because my cuda folder is called cuda-8.0. To fix this I included -v

nvidia-docker run -it --device=/dev/nvidiactl --device=/dev/nvidia-uvm --device=/dev/nvidia0 -v /usr/local/cuda-8.0/include/:/usr/local/cuda/include/ detectron:c2-cuda8-cudnn6

Now when I run infer_simple.py I get an out of memory error:

RuntimeError: [enforce fail at context_gpu.cu:343] error == cudaSuccess. 2 vs 0. Error at: /caffe2/caffe2/core/context_gpu.cu:343: out of memory 

I think I need a new GPU with more than 2GB :)

Hope my cuda cuda-8.0 conflict helps others though.

Thank you

[rbgirshick]

@oreilc16: thanks for the follow up! Yes, unfortunately 2GB isn't enough memory to do much with these models.

[shesung]

I ran into similar problem.

Operating system: CentOS 3.10.0-327.22.2.el7.x86_64
Compiler version: gcc 4.8.5
CUDA version: V8.0.61
cuDNN version: 6.0.21
NVIDIA driver version: 384.69
GeForce GTX 1080 8GB

python2 tools/infer_simple.py --cfg configs/12_2017_baselines/e2e_mask_rcnn_R-50-FPN_2x.yaml --output-dir /tmp/detectron-visualizations --image-ext jpg --wts https://s3-us-west-2.amazonaws.com/detectron/35859007/12_2017_baselines/e2e_mask_rcnn_R-50-FPN_2x.yaml.01_49_07.By8nQcCH/output/train/coco_2014_train:coco_2014_valminusminival/generalized_rcnn/model_final.pkl demo

RuntimeError: [enforce fail at conv_transpose_op_cudnn.cc:266] status == CUDNN_STATUS_SUCCESS. 3 vs 0. , Error at: /home/jack/caffe2/caffe2/operators/conv_transpose_op_cudnn.cc:266: CUDNN_STATUS_BAD_PARAM Error from operator:
input: "gpu_0/_[mask]_fcn4" input: "gpu_0/conv5_mask_w" input: "gpu_0/conv5_mask_b" output: "gpu_0/conv5_mask" name: "" type: "ConvTranspose" arg { name: "kernel" i: 2 } arg { name: "exhaustive_search" i: 0 } arg { name: "pad" i: 0 } arg { name: "order" s: "NCHW" } arg { name: "stride" i: 2 } device_option { device_type: 1 cuda_gpu_id: 0 } engine: "CUDNN"