In CuArrays v2.0, GPU operation takes hours to run for the first time

[lihebi]

Describe the bug
On CuArrays v2.0.1, Any GPU operation takes hours to finish. More than 3 hours, but finished overnight. Once the first operation finished, it works fine for the rest of this session, as usual. But a new Julia session still suffers.

It works fine on CuArrays v1.7.2.

To Reproduce
The Minimal Working Example (MWE) for this bug:

using CuArrays
cu(randn(10)) .- 1
cu(randn(10)) - cu(randn(10))

Expected behavior
finish instantly, or a few seconds when running for the first time.

Build log
I have a few other packages installed but they should not affect CuArrays.

julia> Pkg.build()
  Building NNlib  `~/.julia/packages/NNlib/FAI3o/deps/build.log`
  Building FFTW  `~/.julia/packages/FFTW/qqcBj/deps/build.log`
  Building GR  `~/.julia/packages/GR/yMV3y/deps/build.log`
  Building Plots  `~/.julia/packages/Plots/cc8wh/deps/build.log`
  Building ImageMagick  `~/.julia/packages/ImageMagick/vMfoS/deps/build.log`
false

Environment details (please complete this section)
Details on Julia:

Julia Version 1.3.1
Commit 2d5741174c (2019-12-30 21:36 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: AMD Ryzen 5 3600 6-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, znver1)

Julia packages:

(v1.3) pkg> status
    Status `~/.julia/environments/v1.3/Project.toml`
  [336ed68f] CSV v0.5.26
  [be33ccc6] CUDAnative v3.0.2
  [3a865a2d] CuArrays v2.0.1
  [a93c6f00] DataFrames v0.20.2
  [864edb3b] DataStructures v0.17.10
  [1d5cc7b8] IntelOpenMP_jll v2018.0.3+0
  [ae8d54c2] Luxor v1.11.0
  [91a5bcdd] Plots v0.29.9
  [f3b207a7] StatsPlots v0.14.3
  [e88e6eb3] Zygote v0.4.12

CUDA: toolkit and driver version
10.2, 440.59, RTX2060Super

[maleadt]

Try running with JULIA_DEBUG=all, interrupting the operation to see a stack trace, etc. It's probably downloading artifacts, but that should only happen once.

[lihebi]

Thanks, looks like it is downloading artifacts. Probably my network is slow for this. I just tried a proxy, and it finished pretty fast. Then I tried a new Julia session, and this time the delay is a few seconds as expected.

Closing this. Not sure why previously a new Julia session did not yield this behavior. Maybe the artifacts cache somehow expired after several hours? I'll monitor this and reopen it if I still have problems.

[maleadt]

Maybe the artifacts cache somehow expired after several hours?

I don't think that happens, although you can GC them. @staticfloat, is the way we use artifacts here (multiple lazy ones, one @artifact at first use) compatible with that mechanism (i.e. won't downloaded CUDA artifacts get removed too soon)?

[lihebi]

Just to confirm that it still works fine after a day. It should be something else happened. I tried to re-install NVIDIA driver several times, maybe that invalidated the artifacts at some point.

[staticfloat]

Just to confirm; Pkg.gc() won't delete lazy artifacts. It treats lazy and non-lazy artifacts the same; it will only delete them once all Artifacts.toml files referring to those artifacts are removed. (And only if they have been unreachable for a certain amount of time, too)