GPU CI maintainance

[DhairyaLGandhi]

No description provided.

[DhairyaLGandhi]

CI failure seems unrelated

[vchuravy]

bors try

[bors]

🔒 Permission denied

Existing reviewers: click here to make vchuravy a reviewer

[DhairyaLGandhi]

bors try

[bors]

try

Build failed

• ci/gitlab/trying

[maleadt]

I'd drop the branch restrictions, that way you get automatic testing for people who push to branches on Flux.jl (and only need Bors for external contributors).

[DhairyaLGandhi]

Thanks @maleadt!

Does this look better?

[maleadt]

Yes. Any particular reason you define your own flux target and don't use the templates (e.g. https://raw.githubusercontent.com/JuliaGPU/gitlab-ci/master/templates/v4/test_v1.2.yml)?

[DhairyaLGandhi]

We used to need it since we had to add CuArrays as a dependency for testing on the GPUs. But since we don't need to do that anymore, we can safely get rid of it.

[MikeInnes]

Is this ready to go?

[DhairyaLGandhi]

I think the code is there, but I am seeing access to undefined reference on Julia v1.1 and v1.2

[maleadt]

@MikeInnes reported something similar yesterday, I'll have a closer look.
EDIT: I misremembered, JuliaGPU/CuArrays.jl#404 (comment) is different.

[DhairyaLGandhi]

Am I doing something incorrect here with regard to the error from the bors job above?

[maleadt]

No, bors just reported that the build failed.

[maleadt]

Interesting, this is a legitimate issue now that we depend on CuArrays from Flux and as a result the CuArrays __init__ functions might not have run before Flux' global code (which performs an allocation in global scope). I'll push a quick fix on a stable branch of CuArrays, but I've fixed the issue in the PR that moves over Flux DNN functionality (including that global allocation) to CuArrays. JuliaGPU/CuArrays.jl#404 (comment) was related after all 😄

@dhairyagandhi96 could you try the CuArrays stable branch? https://github.com/JuliaGPU/CuArrays.jl/tree/stable

[DhairyaLGandhi]

bors try

Thanks!

[DhairyaLGandhi]

ERROR: LoadError: LoadError: LoadError: CUDA error: invalid device context (code #201, ERROR_INVALID_CONTEXT)

Seems related to the initialisation

[maleadt]

Maybe it's easier to include the fix here instead: JuliaGPU/CuArrays.jl@5048e89

[bors]

try

Build failed

• ci/gitlab/trying

[DhairyaLGandhi]

bors try

[bors]

try

Build succeeded

• ci/gitlab/trying

[bors]

try

Build failed

• ci/gitlab/trying

[DhairyaLGandhi]

bors try

[bors]

try

Build failed

• ci/gitlab/trying

[DhairyaLGandhi]

bors try

[bors]

try

Build failed

• ci/gitlab/trying

[DhairyaLGandhi]

bors try

[maleadt]

Why do you need to extend manually? The failures before didn't seem CI infrastructure-related.

[DhairyaLGandhi]

In the default the template does a dev of the project per
https://github.com/JuliaGPU/gitlab-ci/blob/52d59dadbc16b7b5cbb87f6a276cb50f68ecf554/templates/v4/common.yml#L25

With having dependencies on the master in a couple places, it is possible that the exact commit is not picked up (or rather the tag is), hence returning the pullback not defined

Trying this out to see if this if this way allows for better reproducibility.

[maleadt]

With having dependencies on the master in a couple places, it is possible that the exact commit is not picked up (or rather the tag is), hence returning the pullback not defined

There shouldn't be a difference between develop $PATH and --project $PATH instantiate, right? Ah wait, I guess the former ignores any Manifest. But isn't that risky for CI, where regular package installation ignores the Manifest anyhow?

[MikeInnes]

The benefits are that (a) the CI builds are completely reproducible and (b) it makes it substantially easier to do WIP on multiple packages at once (e.g. we can update Flux for Zygote changes and make patches as needed, rather than needing to tag a Zygote release, then doing several patch releases for issues Flux finds).

Of course at release time we need to make sure we're using release versions; the manifest then serves as a clear log of what needs tags and/or compat updates, without us having to keep track of that throughout the release cycle.

[DhairyaLGandhi]

Right. It does mean that any breaking changes in the dependencies can break CI. This comes against the reproducibility of the environment (since the Manifest is ignored)

[DhairyaLGandhi]

bors try

[bors]

try

Already running a review

[maleadt]

That sounds like a good fit for CUDAnative/CuArrays/GPUArrays too. @dhairyagandhi96 the v3 templates used instantiate instead of develop, I can update those so that we can continue using them.

[DhairyaLGandhi]

Okay, sounds good! We can then just use the corresponding targets without rewriting them here.

[bors]

try

Build succeeded

• ci/gitlab/trying

[DhairyaLGandhi]

bors r+

[bors]

Build succeeded

• ci/gitlab/staging