Replace conda with mamba

[crusaderky]

Shave 2 minutes off each workflow
Twin of dask/dask#7227

[crusaderky]

@jrbourbeau my tests are being randomly killed. Is #4581 acting up?

[jrbourbeau]

Hmm that is strange. I don't think it's the cancel workflow. Here is the correcsponding cancel workflow for the the latest commit on this PR and it didn't cancel anything.

Interestingly all the CI runs are existing when they hit the same test distributed/protocol/tests/test_serialize.py::test_profile_nested_sizeof . I'm testing CI over in against the latest main branch to see if this failure is reproducible outside this PR.

[crusaderky]

@jrbourbeau looking more carefully, it's a deterministic failure on distributed/protocol/tests/test_serialize.py::test_profile_nested_sizeof which is always terminated with exit code 127 (command not found?) on Windows 3.8 and 3.9. I'll try reproducing it locally.

[jrbourbeau]

Thanks for looking into things @crusaderky. FWIW here's the test PR I opened up to test CI against main #4586. Note that I didn't encounter the test_profile_nested_sizeof issue in that PR

[crusaderky]

It's ipython/ipython#12197 again.
The packages installed by miniconda vs. miniforge are slightly different in version (can't figure out the reason).
As a result, with miniforge there's an interpreter failure on both 3.8 and 3.9 when test_profile_nested_sizeof raises RecursionError. On miniconda, I could produce a minimal POC only for 3.9. Can't figure out the difference.
I also could not figure out why test_profile_nested_sizeof fails on miniforge 3.9 but works on miniconda 3.9, whereas the POC fails on both.
Finally I could not figure out why dask/dask with miniforge, python=3.8 and ipython works fine on windows.

[crusaderky]

@jrbourbeau ready for review and merge

[jrbourbeau]

Thanks for your work on this @crusaderky!

From the output of mamba list it looks like we're installing more packages from defaults than conda-forge than we do with our current setup in main. Looking at the README for conda-incubator/setup-miniconda it appears using mamba requires us to explicitly include conda-forge in the workflow step. Fortunately, when I tried this out in jrbourbeau@20ffce4, I was also able to remove the ipython workaround (see this CI build)

[crusaderky]

@jrbourbeau I integrated your change

[jrbourbeau]

@jakirkham switching to mamba in CI involves us specifying conda channels in our GitHub action workflow file. Right now we're just specifying conda-forge and defaults, which is exactly the same as our current CI setup in main expect for the Python 3.9 build where we install pytorch and torchvision from the pytorch channel

distributed/continuous_integration/environment-3.9.yaml

Lines 2 to 5 in 8942856

	channels:
	- conda-forge
	- defaults
	- pytorch

Do you know if there's a reason to prefer the pytorch channel here, or is conda-forge sufficient? (xref #4017)

[jrbourbeau]

Thanks @crusaderky! Will merge after CI finishes

[jrbourbeau]

Hmm it looks like we're not installing from conda-forge by default again and are running into the ipython issue

[crusaderky]

Weird. I tried removing the line that sets the channels in the workflow because I spotted that in the latest version it's getting packages from the pytorch channel... but apparently it's something else

[jrbourbeau]

Thanks @crusaderky! This is in

[jakirkham]

Do you know if there's a reason to prefer the pytorch channel here, or is conda-forge sufficient? (xref #4017)

Sorry missed this 😞

At the time pytorch in conda-forge didn't support GPUs. Also torchvision was pretty out-of-date in conda-forge. I don't think either of these things are issues any more. So we could probably change to conda-forge

[crusaderky]

Both pytorch and torchvision are still being pulled from the pytorch channel after this PR anyway