Non-linear performance at larger batch sizes.

[NiklasGustafsson]

Repro file:

Repro.jl.txt

The attached repro file (added with .txt so that GitHub will accept it) contains a hand-edited VGG16 model based on an ONNX model file from the ONNX model gallery.

The code relies on NNlib and CuArrays for the significant operations (pooling and convolution). The original weights file are almost 1GB in size, so I changed the code to generate random numbers for the weights.

I benchmarked the model across a number of different batch sizes.

This is a graph of the results: .

I noticed some strange performance issues.

First, there's an odd spike at N=38:

julia> fullbatch(36:1:40)
N = 36  62.850 ms (2994 allocations: 20.83 MiB)
N = 37  61.790 ms (2994 allocations: 21.40 MiB)
N = 38  161.827 ms (3039 allocations: 21.98 MiB)
N = 39  65.711 ms (2998 allocations: 22.55 MiB)
N = 40  66.615 ms (3001 allocations: 23.12 MiB)

Then, once you get to N=69 or 70 (it differs from run to run), the performance goes from more-or-less scaling linearly (with the N=38 exception) to wildly varying (and poor):

julia> fullbatch(68:1:73)
N = 68  130.874 ms (3020 allocations: 39.20 MiB)
N = 69  130.836 ms (3020 allocations: 39.78 MiB)
N = 70  131.772 ms (3026 allocations: 40.35 MiB)
N = 71  477.422 ms (3139 allocations: 40.93 MiB)
N = 72  471.267 ms (3133 allocations: 41.50 MiB)
N = 73  448.079 ms (3126 allocations: 42.08 MiB)

I assume that this has something to do with automatic memory management in CuArrays.jl, but I'm only a user, so I really don't know how to approach analyzing this problem or even understanding whether it is my code that is too naïve. I do take care to use in-place operations when possible, but that may not be enough.

This is run on Windows 10 with a NVIDIA RTX 2080 SUPER GPU.

[maleadt]

Thanks for the great repro! Much easier to debug issues like this without large dependencies (i.e. Flux).

It's curious that you are seeing any scaling at all, because you are not synchronizing the GPU after computations and (on my system at least) are only measuring set-up overheads (memory copies, kernel launches):

N = 36  4.927 ms (4345 allocations: 194.67 KiB)
N = 37  4.934 ms (4345 allocations: 194.67 KiB)
N = 38  5.017 ms (4345 allocations: 194.67 KiB)
N = 39  5.742 ms (4345 allocations: 194.67 KiB)
N = 40  4.959 ms (4345 allocations: 194.67 KiB)
N = 68  9.282 ms (4370 allocations: 195.48 KiB)
N = 69  9.117 ms (4367 allocations: 194.97 KiB)
N = 70  8.949 ms (4367 allocations: 194.97 KiB)
N = 71  9.625 ms (4370 allocations: 195.48 KiB)
N = 72  9.131 ms (4376 allocations: 204.03 KiB)
N = 73  9.364 ms (4370 allocations: 195.48 KiB)

To synchronize the GPU, we have CUDA-like APIs in CUDAdrv, and @sync in CuArrays.jl:

function fullbatch(range)
  for i in range
    print("N = ", i)
    global c_inputs = rand(Float32, 224, 224, 3, i)
    global gpu = Test.Model(i, cuArr)
    @btime CuArrays.@sync gpu(c_inputs |> cu)

    gpu = nothing
    GC.gc(true)
    CuArrays.reclaim()
  end
end

(Also note the change to GC.gc(true), forcing a full GC collection, and the call to CuArrays.reclaim() to empty the GPU memory pool. Neither are required, but improve benchmark accuracy.)

With these changes, I'm seeing the following timings (restricting the pool to 8GB matching your GPU):

$ CUARRAYS_MEMORY_LIMIT="8000000000" julia
N = 36  180.284 ms (4352 allocations: 194.80 KiB)
N = 37  188.239 ms (4352 allocations: 194.80 KiB)
N = 38  192.789 ms (4352 allocations: 194.80 KiB)
N = 39  201.246 ms (4352 allocations: 194.80 KiB)
N = 40  210.003 ms (4355 allocations: 194.86 KiB)
N = 68  345.037 ms (4383 allocations: 196.64 KiB)
N = 69  350.995 ms (4374 allocations: 195.09 KiB)
N = 70  354.848 ms (4380 allocations: 204.22 KiB)
N = 71  366.471 ms (4380 allocations: 196.13 KiB)
N = 72  365.902 ms (4377 allocations: 195.61 KiB)
N = 73  371.753 ms (4380 allocations: 196.13 KiB)

So pretty linear. Doing some more measurements:

Which version of Julia and CuArrays are you using? I'm using the latest master branch of CuArrays, but that should be identical to v1.6.0, in combination with Julia v1.3.1. There's also two upcoming changes that might improve your situation: if you use Julia 1.4 (i.e. a nightly build), improved GC usage should significantly lower certain overheads. There's also a new memory allocator which you can use by specifying CUARRAYS_MEMORY_POOL=split in your environment, but it has some issues still (for example, it failed once for the example here on N=80).

[NiklasGustafsson]

Thank you. User error, simply put.

I see similar results by converting the output of the model to a CPU-based array, which I guess forces a sync.

I've been using Julia 1.2, and I believe the CuArrays version is 1.2.1. I'm going to upgrade and see if that improves the numbers.

[NiklasGustafsson]

Upgrading to 1.3.1 and CuArrays 1.6.0 does not improve the numbers, but I do see the linear curve now that I'm doing the sync correctly.

Thanks,

Niklas

[maleadt]

User error, simply put.

The timings didn't match mine when not using synchronization, so I wasn't sure.
Don't hesitate to open an issue or contact me if you run into other issues though!