Installing cmake on aarch64 using pip via wheel

[odidev]

Summary
Installing cmake on aarch64 linux platform via pip using command "pip3 install cmake" tries to build wheel from source code.

Problem description
Cmake don't have wheel for aarch64 on PyPI repository. So, while installing cmake via pip on aarch64, it builds wheel from source which takes more time to install cmake. Making wheel available for aarch64 will benefit aarch64 users by minimizing installation time.

Expected Output
Pip should be able to download cmake wheel from PyPI repository rather than building it from source code.

@thewtex , I was able to build wheel with command "python setup.py bdist_wheel" and wheel has been created for specific python version in dist folder. I have tried installing in aarch64 linux platform with command "pip install cmake-xxx.whl", it got installed successfully. Please let me know if any help required in building wheel and uploading it to PyPI repository.

[odidev]

Hi @thewtex ,

Installation of cmake is taking time while building from a source in aarch64. It will be efficient and helpful if wheel is added for the same as many packages are dependent on cmake.

[thewtex]

@odidev Since pypa/manylinux#84 is still open, my impression is that pip / manylinux support is a work in progress. This means that even if we create a wheel and upload it, pip will not recognize that it can use it. But, we can try to create it -- users may just need to download the *.whl file manually to install it.

@odidev would you like to try to create a wheel based on the toolchain provided in: https://github.com/pypa/manylinux/blob/manylinux2014/docker/Dockerfile-aarch64 ?

[thewtex]

It is also worth noting that dockcross may enable cross-compiling of the wheels from x86_64 systems, which could help alleviate build time and CI resource issues.

[jcfr]

If there is interest and human resources to make it happen, an approach similar to what was discussed in scikit-build/ninja-python-distributions#22 (comment) could be implemented for aarch64 wheels

In a nutshell, wheels for aarch64 would be hosted as assets in a github release associated with this project and could be installed using something like this:

pip install -f https://github.com/kit-build/cmake-python-distributions/releases/tag/aarch64-python-wheels

[jcfr]

And since we would need wheels for ninja and cmake, the project https://github.com/scikit-build/aarch64-python-wheels could be created ...

[odidev]

@odidev Since pypa/manylinux#84 is still open, my impression is that pip / manylinux support is a work in progress. This means that even if we create a wheel and upload it, pip will not recognize that it can use it. But, we can try to create it -- users may just need to download the *.whl file manually to install it.

@odidev would you like to try to create a wheel based on the toolchain provided in: https://github.com/pypa/manylinux/blob/manylinux2014/docker/Dockerfile-aarch64 ?

@thewtex ,

Thanks for the suggestion. I have created a dockerfile for creating wheel based on suggested toolchain. Please find below dockerfile used to create cmake wheel.

Dockerfile:

FROM quay.io/pypa/manylinux2014_aarch64:latest

RUN yum update -y
&& yum install -y epel-release git wget openssl-devel python3 python3-pip python3-devel
&& yum clean all

RUN mkdir /tmp/dl && cd /tmp/dl
&& wget https://github.com/Kitware/CMake/releases/download/v3.16.5/cmake-3.16.5.tar.gz && tar -xf cmake-3.16.5.tar.gz
&& cd cmake-3.16.5 && ./configure && make && make install

RUN git clone https://github.com/scikit-build/cmake-python-distributions && cd cmake-python-distributions
&& pip3 install -r requirements-dev.txt && python3 setup.py bdist_wheel && ls dist

Wheel(cmake-3.16.3.post1+g0f6336c-cp36-cp36m-linux_aarch64.whl) has been created in dist folder. Can you suggest me on how can we upload it to releases?

[odidev]

It is also worth noting that dockcross may enable cross-compiling of the wheels from x86_64 systems, which could help alleviate build time and CI resource issues.

Thanks for the suggestion. In https://github.com/dockcross/dockcross, we don't have dockerfile for manylinux2014_aarch64. We have only manylinux2014_x64. I think we require that to build wheel through dockcross or I may be wrong, please suggest me on how can we use that for manylinux2014_aarch64? Thanks.

[thewtex]

Please find below dockerfile used to create cmake wheel.

Awesome! Well done @odidev ! 🎉 🥇

In https://github.com/dockcross/dockcross, we don't have dockerfile for manylinux2014_aarch64.

The image to try is dockcross/linux-arm64.

Can you suggest me on how can we upload it to releases?

If cross-compiling works, you could configure a GitHub Action on your fork to build the wheel on push, then push to the GitHub Release. Once ready, create a pull request for this repository.

If cross-compiling does not work, a hosted GitHub Action worker could be possible if you have hardware you can keep available for the cause.

[odidev]

Hi @thewtex ,

Thanks for the suggestion.

I am able to successfully cross-compile using python crossenv and built aarch64 wheel. Checking on github actions for pushing wheel to release.

Dockerfile: Dockerfile_cmake_cross

Travis-CI has added support for arm64. We can build wheel for arm64 and deploy to github releases with tag aarch64-python-wheels.

Sample code to build in Travis:

language: python

matrix:
include:
- os: linux
arch: arm64
env:
- PYTHON_VERSION=3.7.4

script:
- if [ "${TRAVIS_CPU_ARCH}" == "arm64" ]; then
python setup.py bdist_wheel
fi

before_deploy:
- export TRAVIS_TAG=${aarch64-python-wheels:-$(date +'%Y%m%d%H%M%S')-$(git log --format=%h -1)}
- git tag $TRAVIS_TAG

deploy:
provider: releases
api_key:
secure: YOUR_API_KEY_ENCRYPTED
file: "Wheel created in dist folder"
skip_cleanup: true
on:
tags: true

If we can follow this, I think we can upload to git hub releases with tag aarch64-python-wheels.

Can we do this process as through this process it is easy to build and upload to releases?

[thewtex]

@odidev well done! 👏 🔢

Can we do this process as through this process it is easy to build and upload to releases?

Yes -- perhaps you can develop and test the TravisCI configuration and GitHub Action on your folk, then we can enable and merge here when ready.

arm64 CMake packages will very helpful to the community.

[ax3l]

I was setting up CD with cibuildwheel these days and its really simple to set up if you already have a great wheel recipe like yours in place.

You can drive Linux/Windows/OSX builds, the former for ppc64le, aarch64, s390x, i686 and amd64, with the popular open CI platforms in quite compact setups.

I was able to build cmake with it, but since I did not cache the wheel somewhere I am running into timeouts for my actual builds...

[odidev]

@odidev well done! 👏 🔢

Can we do this process as through this process it is easy to build and upload to releases?

Yes -- perhaps you can develop and test the TravisCI configuration and GitHub Action on your folk, then we can enable and merge here when ready.

arm64 CMake packages will very helpful to the community.

Hi @thewtex ,

I have successfully built aarch64 wheel with an aarch64 job. I have modified the travis file as per my requirement to build and deploy the wheel on git release. You can have a look at the same git release at https://github.com/odidev/cmake-python-distributions/releases/tag/aarch64-python-wheels. Using aarch64 job cmake wheel can successfully be deployed.

Travis link: https://github.com/odidev/cmake-python-distributions/blob/master/.travis.yml

Please check the travis file and we had to remove PyPI deployment phase to check deploying aarch64 wheel to git releases with tag.

Please suggest if any more changes are required.