Running out of memory on a 4GB card

[Omegastick]

I'm trying to run Faster-RCNN on a Nvidia GTX 1050Ti, but I'm running out of memory. Nvidia-smi says that about 170MB are already in use, but does Faster-RCNN really use 3.8GB of VRAM to process an image?

I tried Mask-RCNN too (the model in the getting started tutorial) and got about 4 images in (5 if I closed my browser) before it crashed.

Is this a bug or does it really just need more than 4GB of memory?

INFO infer_simple.py: 111: Processing demo/18124840932_e42b3e377c_k.jpg -> /home/px046/prog/Detectron/output/18124840932_e42b3e377c_k.jpg.pdf
terminate called after throwing an instance of 'caffe2::EnforceNotMet'
  what():  [enforce fail at blob.h:94] IsType<T>(). wrong type for the Blob instance. Blob contains nullptr (uninitialized) while caller expects caffe2::Tensor<caffe2::CUDAContext> .
Offending Blob name: gpu_0/conv_rpn_w.
Error from operator: 
input: "gpu_0/res4_5_sum" input: "gpu_0/conv_rpn_w" input: "gpu_0/conv_rpn_b" output: "gpu_0/conv_rpn" name: "" type: "Conv" arg { name: "kernel" i: 3 } arg { name: "exhaustive_search" i: 0 } arg { name: "pad" i: 1 } arg { name: "order" s: "NCHW" } arg { name: "stride" i: 1 } device_option { device_type: 1 cuda_gpu_id: 0 } engine: "CUDNN"
*** Aborted at 1516787658 (unix time) try "date -d @1516787658" if you are using GNU date ***
PC: @     0x7f08de455428 gsignal
*** SIGABRT (@0x3e800000932) received by PID 2354 (TID 0x7f087cda9700) from PID 2354; stack trace: ***
    @     0x7f08de4554b0 (unknown)
    @     0x7f08de455428 gsignal
    @     0x7f08de45702a abort
    @     0x7f08d187db39 __gnu_cxx::__verbose_terminate_handler()
    @     0x7f08d187c1fb __cxxabiv1::__terminate()
    @     0x7f08d187c234 std::terminate()
    @     0x7f08d1897c8a execute_native_thread_routine_compat
    @     0x7f08def016ba start_thread
    @     0x7f08de52741d clone
    @                0x0 (unknown)
Aborted (core dumped)

[ir413]

Hi @Omegastick, the memory requirements of the Faster R-CNN algorithm vary depending on a number of factors including the backbone network architecture and test image scales used.

For example, you can run Faster R-CNN with the default ResNet-50 config using:

python2 tools/infer_simple.py \
  --cfg configs/12_2017_baselines/e2e_faster_rcnn_R-50-FPN_2x.yaml \
  --output-dir /tmp/detectron-visualizations \ 
  --image-ext jpg \
  --wts https://s3-us-west-2.amazonaws.com/detectron/35857389/12_2017_baselines/e2e_faster_rcnn_R-50-FPN_2x.yaml.01_37_22.KSeq0b5q/output/train/coco_2014_train%3Acoco_2014_valminusminival/generalized_rcnn/model_final.pkl \
  demo

which should not require more than 3GB to run on the demo images.

[rbgirshick]

One additional note: the current implementation uses memory optimizations during training, but not during inference. In the case of inference, it is possible to substantially reduce memory usage since intermediate activations are not needed once they are consumed. We will consider adding inference-only memory optimization in the future.

[mattdingmeng]

@Omegastick Tested on my machine, both Faster RCNN- resnet 101 and Mask RCNN- resnet 101 use around 4GB GPU memory.

[Omegastick]

@ir413 Thanks, the model you linked works great (running at 2.5GB VRAM usage) on my machine.

[samhodge]

It would be cool if inference didnt need a GPU at all.

[x-codingman]

how can l run the mask-rcnn with a 2G memory GPU? can anyone help me ?

[pacowong]

Is this problem due to the implementation of Caffe 2 or Detectron? Which files in Detectron should I look at in order to solve this problem?

[gadcam]

@rbgirshick

In the case of inference, it is possible to substantially reduce memory usage since intermediate activations are not needed once they are consumed. We will consider adding inference-only memory optimization in the future.

Is there already something implemented in PyTorch/Caffe2 ? If yes where do we need to dig ?

[rbgirshick]

@gadcam This has been on my todo list for a long time, but unfortunately its priority has been decreasing instead of increasing :/. I think that caffe2.python.memonger.release_blobs_when_used (https://github.com/pytorch/pytorch/blob/master/caffe2/python/memonger.py#L229) should implement most of what we need. However there are some non-trivial issues that need to be addressed:

• For some networks (e.g. Mask R-CNN) multiple nets are used at inference time and therefore not all activations can be freed by reasoning over only one graph (because they may be needed by another graph, e.g., the mask head net).
• This function requires the use of a caching memory manager, which we have not tested, so there could be issues with simply turning that on.