Remote nvidia-docker occasionally fails

[jimfleming]

Here's the error I get:

nvidia-docker | 2016/06/27 19:24:29 Error: Get http://plugin/docker/cli?vol=nvidia_driver&dev=: EOF

And the bash script:

export NV_HOST="ssh://username@$(docker-machine ip $MACHINE_NAME):"
eval $(ssh-agent -s)
ssh-add ~/.docker/machine/machines/$MACHINE_NAME/id_rsa
nvidia-docker run "$DOCKER_IMAGE" "$DOCKER_ARG"

FWIW, this works correctly ~80% of the time. Any tips on troubleshooting this further or possible causes would be super helpful.

[flx42]

I tried to repro the problem using the instructions here, but I couldn't.

Which version of nvidia-docker do you have? And when you say 80% of the time, do you mean it works 80% of the time when accessing the same machine?
Or 80% of the machines you provision have this problem?

[jimfleming]

Thanks for the reply. Here's the info:

Which version of nvidia-docker do you have?

nvidia-docker version: 1.0.0rc3

And when you say 80% of the time, do you mean it works 80% of the time when accessing the same machine?
Or 80% of the machines you provision have this problem?

~80% of the machines I've provisioned.

[flx42]

So far no luck reproducing the nvidia-docker issue, but I do get docker-machine hanging on

Installing Docker...

quite reliably :)

[jimfleming]

Hmm, I haven't run into that myself. In the last week I've started 92 remote instances on EC2, 16 of those failed with the above error. None have failed from hanging.

Is there any debug output from nvidia-docker that would be helpful to capture the next time I encounter the issue?

[flx42]

@3XX0 Did you ever see anything like that? Any idea to troubleshoot the issue?

[3XX0]

Can you try with Go 1.6? You need to change the first line here and rebuild from source:
https://github.com/NVIDIA/nvidia-docker/blob/master/tools/Dockerfile.build#L1

[jimfleming]

Sure, I'll give it a try.

[3XX0]

Did it fix the issue?

[flx42]

Ping @jimfleming

[jimfleming]

Sorry, I did not get around to rebuilding with a newer Go. On the upside, I don't seem to be seeing the issue anymore. (I've rebuilt the AMI for other reasons which may have had an effect.)

[amacneil]

I am seeing this issue a small percentage of the time on our CI servers. Note that I'm not using nvidia-docker wrapper, only the docker CLI arguments (fetched in a previous step from the http api):

docker run -t --cpuset-cpus=2-3,5,10-11,13 --volume-driver=nvidia-docker --volume=nvidia_driver_352.63:/usr/local/nvidia:ro --device=/dev/nvidiactl --device=/dev/nvidia-uvm --device=/dev/nvidia0 --device=/dev/nvidia1 --device=/dev/nvidia2 --device=/dev/nvidia3 image foo.sh
Error response from daemon: create nvidia_driver_352.63: Post http://%2Fvar%2Flib%2Fnvidia-docker%2Fnvidia-docker.sock/VolumeDriver.Create: http: ContentLength=44 with Body length 0

Is the recommended fix still to rebuild with Go 1.6? I'm ok with waiting until the next version of nvidia-docker is released if that is likely to fix it too. Let me know if you would like to attempt any debugging steps as this seems to be happening at least once per day for us.

[flx42]

@amacneil we don't know actually, we weren't able to reproduce the problem.
If you can test with Go 1.6 and can confirm it removes the bug, we will gladly bump the Go version before the 1.0.0 release.