ROCM version is slow

[boniek83]

🐛 Bug

Inference speed is abysmal - way slower than cpu. It works as expected on NVidia. I'm not really sure whether this is torch vision or torch itself at fault.

To Reproduce

Steps to reproduce the behavior:

1. Dockerfile: https://gist.github.com/boniek83/0bbf858a24961816557c522007a11c11
2. Lab: https://github.com/timesler/facenet-pytorch/blob/master/examples/infer.ipynb
3. Data for lab: https://github.com/timesler/facenet-pytorch/tree/master/data/test_images

Expected behavior

Performance should be competitive.

Environment

• PyTorch / torchvision Version (e.g., 1.0 / 0.4.0): 1.6.0 / 0.7.0
• OS (e.g., Linux): CentOS 7.8
• How you installed PyTorch / torchvision (conda, pip, source): source
• Build command you used (if compiling from source): see Dockerfile above
• Python version: 3.6
• GPU models and configuration: Radeon VII
• Any other relevant information: ROCK 3.7 during runtime

cc @iotamudelta @ashishfarmer

[ashishfarmer]

The long time you are seeing here is because of MIOpen initialization overhead. If you run the same example in the same docker multiple times, the times for subsequent runs would look more reasonable. You may set the env variable export MIOPEN_FIND_MODE=2 in your docker and that will help reduce the initialization time by a bit, however the first run will still be significantly slower.
We do have improvements for initialization time coming up in MIOpen in near future

[fmassa]

@ashishfarmer do you think we should close this issue (given that it's initialization time that is being worked on), or should we keep this open?

[boniek83]

Help me understand this please. What is the point of running the same example in the same docker again? I want to do inference as a service with unknown, most likely not repeated input. Is initialization performed every time per request?

[sunway513]

Hello @boniek83 , thanks for the additional information on your use cases.
The approach @ashishfarmer suggested would help for training use cases. Using MIOPEN_FIND_MODE=2 can certainly help reduce the issue; however, even with that option you'll expect some "warm up" overheads on each and every new input configs.
We have optimizations under development, which would help for your use cases. We'll keep this thread updated once we have more to update.
cc @daniellowell

[boniek83]

Lab: https://gist.github.com/boniek83/6cf8fdffa4ac6eef683da3bdc7ab1314
Use the same docker image and input images as in OP.
Can you explain to me why resnet model inference in lab above behaves as I expect it to behave? First request is way slower (probably due to mentioned initialization) but subsequent ones even with different input images are fast. Not so much with mtcnn where every request is painfully slow on gpu. Why there is no similar pattern?

[vfdev-5]

@boniek83 maybe wrapping with with torch.no_grad() context manager can be helpful too.

with torch.no_grad():
    for x, y in loader:
        x = x.to(device)
        start = time.time()
        x_aligned = resnet(x)
        print(time.time() - start)

[boniek83]

No dice. Example lab with output cells:
https://gist.github.com/boniek83/7278ddad6f604434681a2f63ae3a77ee