Skip to content
This repository was archived by the owner on Jan 22, 2024. It is now read-only.
This repository was archived by the owner on Jan 22, 2024. It is now read-only.

docker: Error response from daemon: cannot set both Count and DeviceIDs on device request. #1026

@barpaum

Description

@barpaum

1. Issue or feature description

docker: Error response from daemon: cannot set both Count and DeviceIDs on device request.
ERRO[0000] error waiting for container: context canceled

2. Steps to reproduce the issue

  1. Fresh installed nvidia-docker
  2. Execute "docker run --gpus device=0,1 nvidia/cuda:10.0-base nvidia-smi"
  3. Above descripted error is shown.
    Voilà

By contrast,
both "docker run --gpus device=0 nvidia/cuda:10.0-base nvidia-smi"
and "docker run --gpus device=1 nvidia/cuda:10.0-base nvidia-smi"
works.

3. Information to attach (optional if deemed irrelevant)

  • Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info

-- WARNING, the following logs are for debugging purposes only --

I0725 02:22:11.036926 3845 nvc.c:281] initializing library context (version=1.0.2, build=ff40da533db929bf515aca59ba4c701a65a35e6b)
I0725 02:22:11.037026 3845 nvc.c:255] using root /
I0725 02:22:11.037041 3845 nvc.c:256] using ldcache /etc/ld.so.cache
I0725 02:22:11.037053 3845 nvc.c:257] using unprivileged user 65534:65534
I0725 02:22:11.039907 3846 nvc.c:191] loading kernel module nvidia
I0725 02:22:11.040350 3846 nvc.c:203] loading kernel module nvidia_uvm
I0725 02:22:11.040580 3846 nvc.c:211] loading kernel module nvidia_modeset
I0725 02:22:11.041129 3847 driver.c:133] starting driver service
I0725 02:22:11.682411 3845 nvc_info.c:434] requesting driver information with ''
I0725 02:22:11.683050 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/tls/libnvidia-tls.so.410.104
I0725 02:22:11.683150 3845 nvc_info.c:150] skipping /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.410.104
I0725 02:22:11.683233 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.410.104
I0725 02:22:11.683353 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.410.104
I0725 02:22:11.683432 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.410.104
I0725 02:22:11.683537 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ifr.so.410.104
I0725 02:22:11.683648 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.410.104
I0725 02:22:11.683717 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.410.104
I0725 02:22:11.683791 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.410.104
I0725 02:22:11.683894 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.410.104
I0725 02:22:11.683971 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.410.104
I0725 02:22:11.684081 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.410.104
I0725 02:22:11.684159 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.410.104
I0725 02:22:11.684241 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.410.104
I0725 02:22:11.684354 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.410.104
I0725 02:22:11.684747 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.410.104
I0725 02:22:11.684988 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.410.104
I0725 02:22:11.685067 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.410.104
I0725 02:22:11.685145 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.410.104
I0725 02:22:11.685220 3845 nvc_info.c:148] selecting /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.410.104
I0725 02:22:11.685344 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/tls/libnvidia-tls.so.410.104
I0725 02:22:11.685428 3845 nvc_info.c:150] skipping /usr/lib/i386-linux-gnu/libnvidia-tls.so.410.104
I0725 02:22:11.685503 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-ptxjitcompiler.so.410.104
I0725 02:22:11.685618 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-opencl.so.410.104
I0725 02:22:11.685693 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-ml.so.410.104
I0725 02:22:11.685805 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-ifr.so.410.104
I0725 02:22:11.685915 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-glsi.so.410.104
I0725 02:22:11.685992 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-glcore.so.410.104
I0725 02:22:11.686086 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-fbc.so.410.104
I0725 02:22:11.686198 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-fatbinaryloader.so.410.104
I0725 02:22:11.686277 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-encode.so.410.104
I0725 02:22:11.686385 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-eglcore.so.410.104
I0725 02:22:11.686465 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvidia-compiler.so.410.104
I0725 02:22:11.686549 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libnvcuvid.so.410.104
I0725 02:22:11.686680 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libcuda.so.410.104
I0725 02:22:11.686805 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libGLX_nvidia.so.410.104
I0725 02:22:11.686887 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libGLESv2_nvidia.so.410.104
I0725 02:22:11.686961 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libGLESv1_CM_nvidia.so.410.104
I0725 02:22:11.687035 3845 nvc_info.c:148] selecting /usr/lib/i386-linux-gnu/libEGL_nvidia.so.410.104
W0725 02:22:11.687080 3845 nvc_info.c:299] missing library libvdpau_nvidia.so
W0725 02:22:11.687094 3845 nvc_info.c:299] missing library libnvidia-opticalflow.so
W0725 02:22:11.687102 3845 nvc_info.c:303] missing compat32 library libnvidia-cfg.so
W0725 02:22:11.687110 3845 nvc_info.c:303] missing compat32 library libvdpau_nvidia.so
W0725 02:22:11.687119 3845 nvc_info.c:303] missing compat32 library libnvidia-opticalflow.so
I0725 02:22:11.687535 3845 nvc_info.c:229] selecting /usr/bin/nvidia-smi
I0725 02:22:11.687580 3845 nvc_info.c:229] selecting /usr/bin/nvidia-debugdump
I0725 02:22:11.687619 3845 nvc_info.c:229] selecting /usr/bin/nvidia-persistenced
I0725 02:22:11.687660 3845 nvc_info.c:229] selecting /usr/bin/nvidia-cuda-mps-control
I0725 02:22:11.687696 3845 nvc_info.c:229] selecting /usr/bin/nvidia-cuda-mps-server
I0725 02:22:11.687754 3845 nvc_info.c:366] listing device /dev/nvidiactl
I0725 02:22:11.687771 3845 nvc_info.c:366] listing device /dev/nvidia-uvm
I0725 02:22:11.687782 3845 nvc_info.c:366] listing device /dev/nvidia-uvm-tools
I0725 02:22:11.687793 3845 nvc_info.c:366] listing device /dev/nvidia-modeset
I0725 02:22:11.687848 3845 nvc_info.c:270] listing ipc /run/nvidia-persistenced/socket
W0725 02:22:11.687883 3845 nvc_info.c:274] missing ipc /tmp/nvidia-mps
I0725 02:22:11.687899 3845 nvc_info.c:490] requesting device information with ''
I0725 02:22:11.694747 3845 nvc_info.c:520] listing device /dev/nvidia1 (GPU-190d757f-178d-db3b-860c-2abc792531fe at 00000000:08:00.0)
I0725 02:22:11.701785 3845 nvc_info.c:520] listing device /dev/nvidia0 (GPU-9c130b4b-a4f9-3a27-7ce8-9576be9ff583 at 00000000:0a:00.0)
NVRM version:   410.104
CUDA version:   10.0

Device Index:   0
Device Minor:   1
Model:          GeForce GTX 750 Ti
Brand:          GeForce
GPU UUID:       GPU-190d757f-178d-db3b-860c-2abc792531fe
Bus Location:   00000000:08:00.0
Architecture:   5.0

Device Index:   1
Device Minor:   0
Model:          GeForce GTX 750 Ti
Brand:          GeForce
GPU UUID:       GPU-9c130b4b-a4f9-3a27-7ce8-9576be9ff583
Bus Location:   00000000:0a:00.0
Architecture:   5.0
I0725 02:22:11.701953 3845 nvc.c:318] shutting down library context
I0725 02:22:11.703697 3847 driver.c:192] terminating driver service
I0725 02:22:11.770110 3845 driver.c:233] driver service terminated successfully
  • Kernel version from uname -a
Linux pbox 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Any relevant kernel output lines from dmesg
    So long...
  • Driver information from nvidia-smi -a

==============NVSMI LOG==============

Timestamp                           : Thu Jul 25 10:34:06 2019
Driver Version                      : 410.104
CUDA Version                        : 10.0

Attached GPUs                       : 2
GPU 00000000:08:00.0
    Product Name                    : GeForce GTX 750 Ti
    Product Brand                   : GeForce
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 4000
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-190d757f-178d-db3b-860c-2abc792531fe
    Minor Number                    : 1
    VBIOS Version                   : 82.07.55.00.02
    MultiGPU Board                  : No
    Board ID                        : 0x800
    GPU Part Number                 : N/A
    Inforom Version
        Image Version               : N/A
        OEM Object                  : N/A
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    IBMNPU
        Relaxed Ordering Mode       : N/A
    PCI
        Bus                         : 0x08
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x138010DE
        Bus Id                      : 00000000:08:00.0
        Sub System Id               : 0x105F10DE
        GPU Link Info
            PCIe Generation
                Max                 : 2
                Current             : 2
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : 0 KB/s
        Rx Throughput               : 0 KB/s
    Fan Speed                       : 33 %
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
            HW Thermal Slowdown     : N/A
            HW Power Brake Slowdown : N/A
        Sync Boost                  : Not Active
        SW Thermal Slowdown         : Not Active
        Display Clock Setting       : Not Active
    FB Memory Usage
        Total                       : 4043 MiB
        Used                        : 0 MiB
        Free                        : 4043 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 1 MiB
        Free                        : 255 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Encoder Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    FBC Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
        Aggregate
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending                     : N/A
    Temperature
        GPU Current Temp            : 41 C
        GPU Shutdown Temp           : 101 C
        GPU Slowdown Temp           : 96 C
        GPU Max Operating Temp      : N/A
        Memory Current Temp         : N/A
        Memory Max Operating Temp   : N/A
    Power Readings
        Power Management            : Supported
        Power Draw                  : 1.93 W
        Power Limit                 : 38.50 W
        Default Power Limit         : 38.50 W
        Enforced Power Limit        : 38.50 W
        Min Power Limit             : 30.00 W
        Max Power Limit             : 38.50 W
    Clocks
        Graphics                    : 1019 MHz
        SM                          : 1019 MHz
        Memory                      : 2600 MHz
        Video                       : 917 MHz
    Applications Clocks
        Graphics                    : 1019 MHz
        Memory                      : 2600 MHz
    Default Applications Clocks
        Graphics                    : 1019 MHz
        Memory                      : 2600 MHz
    Max Clocks
        Graphics                    : 1293 MHz
        SM                          : 1293 MHz
        Memory                      : 2600 MHz
        Video                       : 1164 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes                       : None

GPU 00000000:0A:00.0
    Product Name                    : GeForce GTX 750 Ti
    Product Brand                   : GeForce
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 4000
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-9c130b4b-a4f9-3a27-7ce8-9576be9ff583
    Minor Number                    : 0
    VBIOS Version                   : 82.07.55.00.59
    MultiGPU Board                  : No
    Board ID                        : 0xa00
    GPU Part Number                 : N/A
    Inforom Version
        Image Version               : N/A
        OEM Object                  : N/A
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    IBMNPU
        Relaxed Ordering Mode       : N/A
    PCI
        Bus                         : 0x0A
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x138010DE
        Bus Id                      : 00000000:0A:00.0
        Sub System Id               : 0x105F10DE
        GPU Link Info
            PCIe Generation
                Max                 : 2
                Current             : 2
            Link Width
                Max                 : 16x
                Current             : 8x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : 0 KB/s
        Rx Throughput               : 0 KB/s
    Fan Speed                       : 40 %
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
            HW Thermal Slowdown     : N/A
            HW Power Brake Slowdown : N/A
        Sync Boost                  : Not Active
        SW Thermal Slowdown         : Not Active
        Display Clock Setting       : Not Active
    FB Memory Usage
        Total                       : 2002 MiB
        Used                        : 0 MiB
        Free                        : 2002 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 1 MiB
        Free                        : 255 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 1 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Encoder Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    FBC Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
        Aggregate
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending                     : N/A
    Temperature
        GPU Current Temp            : 34 C
        GPU Shutdown Temp           : 101 C
        GPU Slowdown Temp           : 96 C
        GPU Max Operating Temp      : N/A
        Memory Current Temp         : N/A
        Memory Max Operating Temp   : N/A
    Power Readings
        Power Management            : Supported
        Power Draw                  : 2.27 W
        Power Limit                 : 58.50 W
        Default Power Limit         : 58.50 W
        Enforced Power Limit        : 58.50 W
        Min Power Limit             : 30.00 W
        Max Power Limit             : 58.50 W
    Clocks
        Graphics                    : 1110 MHz
        SM                          : 1110 MHz
        Memory                      : 2700 MHz
        Video                       : 999 MHz
    Applications Clocks
        Graphics                    : 1110 MHz
        Memory                      : 2700 MHz
    Default Applications Clocks
        Graphics                    : 1110 MHz
        Memory                      : 2700 MHz
    Max Clocks
        Graphics                    : 1385 MHz
        SM                          : 1385 MHz
        Memory                      : 2700 MHz
        Video                       : 1246 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes                       : None
  • Docker version from docker version
Client: Docker Engine - Community
 Version:           19.03.0
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        aeac949
 Built:             Wed Jul 17 18:15:07 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       aeac949
  Built:            Wed Jul 17 18:13:43 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
  • NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'

==============NVSMI LOG==============

Timestamp                           : Thu Jul 25 10:34:06 2019
Driver Version                      : 410.104
CUDA Version                        : 10.0

Attached GPUs                       : 2
GPU 00000000:08:00.0
    Product Name                    : GeForce GTX 750 Ti
    Product Brand                   : GeForce
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 4000
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-190d757f-178d-db3b-860c-2abc792531fe
    Minor Number                    : 1
    VBIOS Version                   : 82.07.55.00.02
    MultiGPU Board                  : No
    Board ID                        : 0x800
    GPU Part Number                 : N/A
    Inforom Version
        Image Version               : N/A
        OEM Object                  : N/A
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    IBMNPU
        Relaxed Ordering Mode       : N/A
    PCI
        Bus                         : 0x08
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x138010DE
        Bus Id                      : 00000000:08:00.0
        Sub System Id               : 0x105F10DE
        GPU Link Info
            PCIe Generation
                Max                 : 2
                Current             : 2
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : 0 KB/s
        Rx Throughput               : 0 KB/s
    Fan Speed                       : 33 %
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
            HW Thermal Slowdown     : N/A
            HW Power Brake Slowdown : N/A
        Sync Boost                  : Not Active
        SW Thermal Slowdown         : Not Active
        Display Clock Setting       : Not Active
    FB Memory Usage
        Total                       : 4043 MiB
        Used                        : 0 MiB
        Free                        : 4043 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 1 MiB
        Free                        : 255 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Encoder Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    FBC Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
        Aggregate
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending                     : N/A
    Temperature
        GPU Current Temp            : 41 C
        GPU Shutdown Temp           : 101 C
        GPU Slowdown Temp           : 96 C
        GPU Max Operating Temp      : N/A
        Memory Current Temp         : N/A
        Memory Max Operating Temp   : N/A
    Power Readings
        Power Management            : Supported
        Power Draw                  : 1.93 W
        Power Limit                 : 38.50 W
        Default Power Limit         : 38.50 W
        Enforced Power Limit        : 38.50 W
        Min Power Limit             : 30.00 W
        Max Power Limit             : 38.50 W
    Clocks
        Graphics                    : 1019 MHz
        SM                          : 1019 MHz
        Memory                      : 2600 MHz
        Video                       : 917 MHz
    Applications Clocks
        Graphics                    : 1019 MHz
        Memory                      : 2600 MHz
    Default Applications Clocks
        Graphics                    : 1019 MHz
        Memory                      : 2600 MHz
    Max Clocks
        Graphics                    : 1293 MHz
        SM                          : 1293 MHz
        Memory                      : 2600 MHz
        Video                       : 1164 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes                       : None

GPU 00000000:0A:00.0
    Product Name                    : GeForce GTX 750 Ti
    Product Brand                   : GeForce
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 4000
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-9c130b4b-a4f9-3a27-7ce8-9576be9ff583
    Minor Number                    : 0
    VBIOS Version                   : 82.07.55.00.59
    MultiGPU Board                  : No
    Board ID                        : 0xa00
    GPU Part Number                 : N/A
    Inforom Version
        Image Version               : N/A
        OEM Object                  : N/A
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    IBMNPU
        Relaxed Ordering Mode       : N/A
    PCI
        Bus                         : 0x0A
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x138010DE
        Bus Id                      : 00000000:0A:00.0
        Sub System Id               : 0x105F10DE
        GPU Link Info
            PCIe Generation
                Max                 : 2
                Current             : 2
            Link Width
                Max                 : 16x
                Current             : 8x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : 0 KB/s
        Rx Throughput               : 0 KB/s
    Fan Speed                       : 40 %
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
            HW Thermal Slowdown     : N/A
            HW Power Brake Slowdown : N/A
        Sync Boost                  : Not Active
        SW Thermal Slowdown         : Not Active
        Display Clock Setting       : Not Active
    FB Memory Usage
        Total                       : 2002 MiB
        Used                        : 0 MiB
        Free                        : 2002 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 1 MiB
        Free                        : 255 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 1 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Encoder Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    FBC Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
        Aggregate
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending                     : N/A
    Temperature
        GPU Current Temp            : 34 C
        GPU Shutdown Temp           : 101 C
        GPU Slowdown Temp           : 96 C
        GPU Max Operating Temp      : N/A
        Memory Current Temp         : N/A
        Memory Max Operating Temp   : N/A
    Power Readings
        Power Management            : Supported
        Power Draw                  : 2.27 W
        Power Limit                 : 58.50 W
        Default Power Limit         : 58.50 W
        Enforced Power Limit        : 58.50 W
        Min Power Limit             : 30.00 W
        Max Power Limit             : 58.50 W
    Clocks
        Graphics                    : 1110 MHz
        SM                          : 1110 MHz
        Memory                      : 2700 MHz
        Video                       : 999 MHz
    Applications Clocks
        Graphics                    : 1110 MHz
        Memory                      : 2700 MHz
    Default Applications Clocks
        Graphics                    : 1110 MHz
        Memory                      : 2700 MHz
    Max Clocks
        Graphics                    : 1385 MHz
        SM                          : 1385 MHz
        Memory                      : 2700 MHz
        Video                       : 1246 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes                       : None
  • NVIDIA container library version from nvidia-container-cli -V

==============NVSMI LOG==============

Timestamp                           : Thu Jul 25 10:34:06 2019
Driver Version                      : 410.104
CUDA Version                        : 10.0

Attached GPUs                       : 2
GPU 00000000:08:00.0
    Product Name                    : GeForce GTX 750 Ti
    Product Brand                   : GeForce
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 4000
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-190d757f-178d-db3b-860c-2abc792531fe
    Minor Number                    : 1
    VBIOS Version                   : 82.07.55.00.02
    MultiGPU Board                  : No
    Board ID                        : 0x800
    GPU Part Number                 : N/A
    Inforom Version
        Image Version               : N/A
        OEM Object                  : N/A
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    IBMNPU
        Relaxed Ordering Mode       : N/A
    PCI
        Bus                         : 0x08
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x138010DE
        Bus Id                      : 00000000:08:00.0
        Sub System Id               : 0x105F10DE
        GPU Link Info
            PCIe Generation
                Max                 : 2
                Current             : 2
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : 0 KB/s
        Rx Throughput               : 0 KB/s
    Fan Speed                       : 33 %
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
            HW Thermal Slowdown     : N/A
            HW Power Brake Slowdown : N/A
        Sync Boost                  : Not Active
        SW Thermal Slowdown         : Not Active
        Display Clock Setting       : Not Active
    FB Memory Usage
        Total                       : 4043 MiB
        Used                        : 0 MiB
        Free                        : 4043 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 1 MiB
        Free                        : 255 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Encoder Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    FBC Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
        Aggregate
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending                     : N/A
    Temperature
        GPU Current Temp            : 41 C
        GPU Shutdown Temp           : 101 C
        GPU Slowdown Temp           : 96 C
        GPU Max Operating Temp      : N/A
        Memory Current Temp         : N/A
        Memory Max Operating Temp   : N/A
    Power Readings
        Power Management            : Supported
        Power Draw                  : 1.93 W
        Power Limit                 : 38.50 W
        Default Power Limit         : 38.50 W
        Enforced Power Limit        : 38.50 W
        Min Power Limit             : 30.00 W
        Max Power Limit             : 38.50 W
    Clocks
        Graphics                    : 1019 MHz
        SM                          : 1019 MHz
        Memory                      : 2600 MHz
        Video                       : 917 MHz
    Applications Clocks
        Graphics                    : 1019 MHz
        Memory                      : 2600 MHz
    Default Applications Clocks
        Graphics                    : 1019 MHz
        Memory                      : 2600 MHz
    Max Clocks
        Graphics                    : 1293 MHz
        SM                          : 1293 MHz
        Memory                      : 2600 MHz
        Video                       : 1164 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes                       : None

GPU 00000000:0A:00.0
    Product Name                    : GeForce GTX 750 Ti
    Product Brand                   : GeForce
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 4000
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-9c130b4b-a4f9-3a27-7ce8-9576be9ff583
    Minor Number                    : 0
    VBIOS Version                   : 82.07.55.00.59
    MultiGPU Board                  : No
    Board ID                        : 0xa00
    GPU Part Number                 : N/A
    Inforom Version
        Image Version               : N/A
        OEM Object                  : N/A
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    IBMNPU
        Relaxed Ordering Mode       : N/A
    PCI
        Bus                         : 0x0A
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x138010DE
        Bus Id                      : 00000000:0A:00.0
        Sub System Id               : 0x105F10DE
        GPU Link Info
            PCIe Generation
                Max                 : 2
                Current             : 2
            Link Width
                Max                 : 16x
                Current             : 8x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : 0 KB/s
        Rx Throughput               : 0 KB/s
    Fan Speed                       : 40 %
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
            HW Thermal Slowdown     : N/A
            HW Power Brake Slowdown : N/A
        Sync Boost                  : Not Active
        SW Thermal Slowdown         : Not Active
        Display Clock Setting       : Not Active
    FB Memory Usage
        Total                       : 2002 MiB
        Used                        : 0 MiB
        Free                        : 2002 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 1 MiB
        Free                        : 255 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 1 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Encoder Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    FBC Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
        Aggregate
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending                     : N/A
    Temperature
        GPU Current Temp            : 34 C
        GPU Shutdown Temp           : 101 C
        GPU Slowdown Temp           : 96 C
        GPU Max Operating Temp      : N/A
        Memory Current Temp         : N/A
        Memory Max Operating Temp   : N/A
    Power Readings
        Power Management            : Supported
        Power Draw                  : 2.27 W
        Power Limit                 : 58.50 W
        Default Power Limit         : 58.50 W
        Enforced Power Limit        : 58.50 W
        Min Power Limit             : 30.00 W
        Max Power Limit             : 58.50 W
    Clocks
        Graphics                    : 1110 MHz
        SM                          : 1110 MHz
        Memory                      : 2700 MHz
        Video                       : 999 MHz
    Applications Clocks
        Graphics                    : 1110 MHz
        Memory                      : 2700 MHz
    Default Applications Clocks
        Graphics                    : 1110 MHz
        Memory                      : 2700 MHz
    Max Clocks
        Graphics                    : 1385 MHz
        SM                          : 1385 MHz
        Memory                      : 2700 MHz
        Video                       : 1246 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes                       : None
  • NVIDIA container library logs (see troubleshooting)
  • Docker command, image and tag used

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions