Skip to content

Conversation

@hugovk
Copy link
Member

@hugovk hugovk commented Jan 3, 2024

Fixes #7390.
Closes #7485.

Follow on from discussion starting #7485 (comment).

This PR replaces the native Linux aarch64 built on Travis CI, and instead builds them on GitHub Actions using QEMU emulation.

  • The native builds are fast: 3 jobs of 7+7+5 wheels = 19 wheels in 40m + 40m + 20m = 1h40m total build time, about 40 mins waiting time.

  • Emulated builds are much slower: one job per each of 19 wheels, taking between 1h25m and 2h10m. Waiting time about 2h10m.

Because they're slow, we build each wheel in its own job, to make full use of the 20 parallel jobs available.

The jobs are started in alphabetical order of job ID:

  • We want to start the slowest QEMU first (19 x 1h25m-2h10m) to give them a headstart.
  • Then Linux and macOS native wheels next (4 x 15m-40m).
  • Then Windows wheels (3 x 10-35m).
  • Finally the single sdist (30s).

Of the QEMU wheels, the CPython builds take around 1.5h and PyPy just over 2h. These are started in by order of the matrix variables (see https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#using-a-matrix-strategy). This means the PyPy ones unfortunately come last.

We can work around this, like add another variable to the matrix to make PyPy come first. Or split the config like build-1-QEMU-emulated-wheels-pypy and build-2-QEMU-emulated-wheels-cpython. But I've left that for now -- with 19 QEMU jobs and a clear CI queue, they should all get started at the same time. Maybe something to consider later.

This will increase total time for the release workflow from around 50 minutes to maybe 2h15m, but we no longer rely on Travis CI and the whole build and deploy to PyPI takes place in a single workflow, to make releases easier, more reliable and secure.

@radarhere radarhere merged commit 4233dd7 into python-pillow:main Jan 4, 2024
@hugovk hugovk deleted the qemu branch January 5, 2024 05:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Set up GitHub release token for Travis CI Automate build and release to PyPI

2 participants