Install Python 2.7 and 3.6 on Centos6 base image through Pyenv by cosmicexplorer · Pull Request #7064 · pantsbuild/pants

cosmicexplorer · 2019-01-12T19:55:14Z

Problem

We have used a python 3 interpreter in CI for a while now, using pyenv to install this in our Dockerfiles instead of relying on CentOS's scl features, which don't allow for multiple pythons at a time. Using pyenv to install Python 2.7 as well instead of relying on the package manager allows us to upgrade or modify the image without screwing up our Python installation.

That work began in this PR and ended up in #6981, #7352, #7418, #7483, and more, but it then led to having to duplicate code across travis_ci and travis_ci_py36 Dockerfiles. Using pyenv to install python in our travis_ci images also involves a significant setup time in bootstrap phases to install our desired pythons. Installing Python in the CentOS 6 base image avoids having to install Python at all in our CI, which saves us minutes in the bootstrap phase.

Solution

Use Pyenv to install Python 2.7 and Python 3.6 in the centos6 Dockerfile.
Remove travis_cy_py36.
Modify the travis_ci Dockerfile to bootstrap pyenv itself if not available.

Result

We get some CI time back in the bootstrap phase.

TODO

merge this PR
publish the new centos6 image to dockerhub at pantsbuild/centos6:latest
merge a followup PR removing the manual pyenv bootstrapping from the travis_ci image.

Eric-Arellano

Thank you Danny for cleaning this up! Looks great.

We’ll need to figure out how to publish this to Docker once it’s merged.

build-support/docker/centos6/Dockerfile

cosmicexplorer · 2019-01-13T17:15:56Z

As the CI run was shown to work on Travis in 05e7b67, I've moved the pyenv changes from travis_ci/Dockerfile to centos6/Dockerfile so that the CentOS6 base image has a python 2 and a python 3 install ready to go. This will need a docker release for pantsbuild/centos6:latest once merged.

build-support/docker/centos6/Dockerfile

cosmicexplorer · 2019-01-14T02:43:45Z

I pushed e641c3e first with changes made to both centos6 and travis_ci dockerfiles to get an idea of whether this change would work once we push the centos6 container, then pushed the final commit for the code which will actually be checked in. I made #7072 to consider caching docker builds in CI avoid this conundrum by allowing us to build both images in travis without any extra CI time.

build-support/docker/centos6/Dockerfile

cosmicexplorer · 2019-01-14T09:06:30Z

Was having trouble with Docker on my local machine not having enough space even after I deleted all of the images locally. Will look at this with fresh eyes tomorrow.

illicitonion

Looks reasonable, but I don't know enough about pyenv to be useful - leaving to other folks who know what they're talking about :) Thanks!

Realized we don't need to be blocked by pantsbuild#7064, although it should certainly be addressed soon after by a followup PR.

build-support/docker/centos6/Dockerfile

cosmicexplorer · 2019-01-15T02:06:00Z

pyenv-installer doesn't actually modify ~/.bashrc -- see #7064 (comment). Also, docker silently drops any changes made to a directory after it is marked as a VOLUME, which would have been fantastic if it had been mentioned in any level of bold text. Both of these fun surprises were added as comments.

cosmicexplorer · 2019-01-15T02:55:02Z

With #6981 squash-merged on top of the current commit, we get this travis log for building the py3 linux pex (from this travis run on my fork), which shows it failing at importing sqlite, which is expected until the changes to centos6/Dockerfile are pushed, and I believe demonstrates that a valid python 3 interpreter is indeed available.

EDIT: After talking to @Eric-Arellano, that appears not to be the case. Will make another squash merge temporarily adding the sqlite-devel package in the travis_ci dockerfile to verify it works with the pants3-in-ci branch.

cosmicexplorer · 2019-01-15T04:42:59Z

There are failures in the branch I've pushed my to own fork which squash merges pants3-in-ci, one for py2 and py3, which I'll get to tomorrow.

### Problem Right now our CI exclusively uses Python 2 under-the-hood. It tries to test Python 3 support by setting`--setup-python-interpreter-constraints` to Python 3, but this only constrains subprocesses like tests and is not the same thing as fully supporting Python 3. We want our CI to test using both Python 2 under-the-hood and Python 3 under-the-hood, because for the next two releases are promising users that either interpreter is fully supported. Daily CI should run primarily with Python 3, and the nightly cron job should check for Python 2 regressions. ### Solution Use the new `./pants3` script added in #6959. #### Installing Python 3 on each shard * OSX: git clone pyenv to consistently install 3.6.8. Necessary because some older images come with an outdated version of pyenv. We don't use Travis's `language` feature because several shards are supposed to be `language: general`. * Linux build engine, temporarily modify `travis_ci/Dockerfile` to download pyenv and install Python 3. This change is being pulled out the `centos6` image in #7064 to avoid rebuilding Py3 every CI run, but it must be published to Docker hub so will be addressed in a followup PR. * Normal Linux shards, use Travis's `language: python` feature #### Add integration test blacklist 23 integration test targets are failing when ran with Py3 under-the-hood, so we restore support for an integration test blacklist. This allows us to track Py3 fixes we make and to collaborate more easily. Further, the contrib tests, linter, and osx rust shards when ran with Py3. So, we run these 3 shards and the blacklist with a Py2 pex but Py3 subprocesses, which is what we currently do in master. #### Expand Cron job We add non-integration test jobs to the nightly cron run to check for any regressions in shards like `Linux build wheels`, now that we run daily with a Py3 PEX and need to check that they still work with a Py2 PEX. #### Default to Python 3 in CI Defaulting to Py3 reflects that we are transitioning towards exclusive Py3 use. This is not user-facing, and only impacts the `.travis.yml` code, so this change is safe to make, whereas we want to wait to rename `./pants3` -> `./pants` until it is a bit more robust. ### FYI - new .travis.yml idiom This required a major rewrite of `.travis.yml` to avoid duplication. The new idiom we use when defining a shard is to have a `base_my_new_shard` which defines the minimum amount of config necessary for that job, and then to have a `py2_my_new_shard` and `py3_my_new_shard` that extend that base along with their corresponding OS + python version config.

add openssl and other required libs to the travis_ci dockerfile

cosmicexplorer · 2019-04-04T04:34:08Z

I have pushed changes which address the recent review comments and also allow the travis_ci Dockerfile to work with or without the centos6 dockerhub push, which breaks the dependency cycle of pushing the centos6 image vs merging this PR. The plan:

Merge this PR.
Push the updated centos6 dockerfile to dockerhub
Followup PR removing the section marked BEGIN CENTOS6 DOCKERFILE CHANGES in the travis_ci dockerfile

Eric-Arellano

Good idea with the plan to make this robust to when we publish to Dockerhub!

build-support/docker/centos6/Dockerfile

Eric-Arellano

Re-reviewed code changes and they all look good.

Some suggestions for the PR title and PR description:

Rename to "Install Python 2.7 and 3.6 on Centos6 base image through Pyenv". Pyenv is more of an implementation detail than the important takeaway.
The cache from #7470 does not apply to the docker images. For the Docker images, we rebuild Python 3 every time.
In solution section, change to Use Pyenv to install Python 2.7 and Python 3.6
Maybe worth mentioning why we use Pyenv to install Python 2.7 rather than the apt installed Python. I think the main motivation is symmetry.

Eric-Arellano · 2019-04-05T21:37:24Z

build-support/docker/travis_ci_py27_ucs2/Dockerfile

@@ -21,10 +21,12 @@ RUN yum install sqlite-devel -y
 ENV PYENV_ROOT "${PYENV_ROOT:-/pyenv-docker-build}"


@cosmicexplorer can you please remove lines 19 and 20: RUN yum install sqlite-devel -y. We don't need them anymore.

### Problem The conclusion of #7064 was, after `pantsbuild/centos6:latest` was updated, to remove the pyenv bootstrapping in `travis_ci/Dockerfile` as it was now being done in the centos6 base image. ### Solution - Remove the marked section. ### Result This will speed up all Docker shards that had Py3 installed by 2-3 minutes. So, Linux build engine Py36 and Linux build wheels abi3. This means overall CI is about 5 minutes faster, and the bootstrap stage 2-3 minutes faster, which is especially significant since that blocks starting other tests. Finally, this means everywhere we run Pants in CI, we have Python 3 installed now (instead of having to install it again!).

…7418) ### Problem We will in #7064 start using Pyenv to install Python 2.7 in the base Centos6 image. This will conflict with our Linux UCS2 shard, which currently uses Pyenv to install Python 2.7 with UCS2 instead of the default UCS4. ### Solution Tweak the Dockerfile to avoid any ambiguity with Centos6 image: * Rename the Python version env var to not conflict by marking it as `UCS2`. * Check if `PYENV_ROOT` is already defined. * Check if `PYENV_ROOT` already exists before git cloning to avoid a git failure. The key line is `${PYENV_BIN} global ${PYTHON_27_VERSION_UCS2}`, which will override whatever `${PYENV_ROOT}/shims/python2.7` used to point to.

### Problem We have used a python 3 interpreter in CI for a while now, using `pyenv` to install this in our Dockerfiles instead of relying on CentOS's `scl` features, which don't allow for multiple pythons at a time. Using `pyenv` to install Python 2.7 as well instead of relying on the package manager allows us to upgrade or modify the image without screwing up our Python installation. That work began in this PR and ended up in #6981, #7352, #7418, #7483, and more, but it then led to having to duplicate code across `travis_ci` and `travis_ci_py36` Dockerfiles. Using `pyenv` to install python in our `travis_ci` images also involves a significant setup time in bootstrap phases to install our desired pythons. Installing Python in the CentOS 6 base image avoids having to install Python at all in our CI, which saves us minutes in the bootstrap phase. ### Solution - Use Pyenv to install Python 2.7 and Python 3.6 in the `centos6` Dockerfile. - Remove `travis_cy_py36`. - Modify the `travis_ci` Dockerfile to bootstrap pyenv itself if not available. ### Result We get some CI time back in the bootstrap phase. ### TODO - [ ] merge this PR - [ ] publish the new `centos6` image to dockerhub at `pantsbuild/centos6:latest` - [ ] merge a followup PR removing the manual pyenv bootstrapping from the `travis_ci` image.

cosmicexplorer requested review from Eric-Arellano, benjyw and illicitonion January 12, 2019 19:55

Eric-Arellano approved these changes Jan 12, 2019

View reviewed changes

build-support/docker/centos6/Dockerfile Show resolved Hide resolved

cosmicexplorer mentioned this pull request Jan 12, 2019

Nicer local docker iteration #7065

Closed

This was referenced Jan 13, 2019

consider caching travis_ci/Dockerfile builds in CI #7072

Closed

merge pants.util.objects.datatype with python 3 dataclasses #7074

Closed

Support python 3.6+ in CI #7078

Closed

benjyw reviewed Jan 13, 2019

View reviewed changes

build-support/docker/centos6/Dockerfile Outdated Show resolved Hide resolved

Eric-Arellano reviewed Jan 13, 2019

View reviewed changes

build-support/docker/centos6/Dockerfile Outdated Show resolved Hide resolved

build-support/docker/centos6/Dockerfile Outdated Show resolved Hide resolved

benjyw reviewed Jan 14, 2019

View reviewed changes

build-support/docker/centos6/Dockerfile Outdated Show resolved Hide resolved

benjyw reviewed Jan 14, 2019

View reviewed changes

build-support/docker/centos6/Dockerfile Outdated Show resolved Hide resolved

benjyw reviewed Jan 14, 2019

View reviewed changes

build-support/docker/centos6/Dockerfile Show resolved Hide resolved

illicitonion reviewed Jan 14, 2019

View reviewed changes

Eric-Arellano mentioned this pull request Jan 14, 2019

Use ./pants3 for majority of CI #6981

Merged

Eric-Arellano reviewed Jan 14, 2019

View reviewed changes

build-support/docker/centos6/Dockerfile Show resolved Hide resolved

build-support/docker/centos6/Dockerfile Outdated Show resolved Hide resolved

build-support/docker/centos6/Dockerfile Outdated Show resolved Hide resolved

cosmicexplorer force-pushed the python3-centos6-dockerfile branch 3 times, most recently from 12c12db to 1be1a25 Compare January 15, 2019 01:42

cosmicexplorer force-pushed the python3-centos6-dockerfile branch 3 times, most recently from 776cb9b to 13f7b4c Compare January 23, 2019 07:54

cosmicexplorer added 12 commits April 3, 2019 20:02

fix docker permissions

990bb6b

remove centos6 base image changes for now

4b1cf69

enable building python as a shared library in the docker image

ff47eb2

try adding a clean to the bootstrap args for the py27 pex shard

c205036

remove travis_ci_py36

5fb981c

add note on -x in py2 bootstrap shard

a18380e

add openssl and other required libs to the travis_ci dockerfile

e9a0b7d

add openssl and other required libs to the travis_ci dockerfile

move everything to centos6 dockerfile too

7ec054a

remove openssl and abbreviate PYENV_BIN

5fad238

remove reference to deleted travis ci image for py3.6

921d80c

some review comments

84da10c

more review comments

a06bfbe

cosmicexplorer force-pushed the python3-centos6-dockerfile branch from afea5f2 to a06bfbe Compare April 4, 2019 03:19

make the travis_ci dockerfile work with or without the centos6 change

9e228c2

Eric-Arellano approved these changes Apr 4, 2019

View reviewed changes

respond to review comments

9989083

cosmicexplorer changed the title ~~install python 3.6.8 on the centos6 base image for ./pants3~~ install pyenv in the centos6 base image Apr 4, 2019

Eric-Arellano approved these changes Apr 4, 2019

View reviewed changes

cosmicexplorer changed the title ~~install pyenv in the centos6 base image~~ Install Python 2.7 and 3.6 on Centos6 base image through Pyenv Apr 4, 2019

cosmicexplorer merged commit 20c497c into pantsbuild:master Apr 5, 2019

cosmicexplorer mentioned this pull request Apr 5, 2019

remove pyenv bootstrapping from travis_ci/Dockerfile #7505

Merged

Eric-Arellano reviewed Apr 5, 2019

View reviewed changes

Eric-Arellano added this to the 1.15.x milestone Apr 6, 2019

Eric-Arellano added the needs-cherrypick [CI] label Apr 6, 2019

stuhood removed needs-cherrypick [CI] labels May 19, 2019

		@@ -21,10 +21,12 @@ RUN yum install sqlite-devel -y
		ENV PYENV_ROOT "${PYENV_ROOT:-/pyenv-docker-build}"

Uh oh!

Conversation

cosmicexplorer commented Jan 12, 2019 • edited by Eric-Arellano Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Result

TODO

Uh oh!

Eric-Arellano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cosmicexplorer commented Jan 13, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cosmicexplorer commented Jan 14, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cosmicexplorer commented Jan 14, 2019

Uh oh!

illicitonion left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cosmicexplorer commented Jan 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cosmicexplorer commented Jan 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cosmicexplorer commented Jan 15, 2019

Uh oh!

cosmicexplorer commented Apr 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eric-Arellano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Eric-Arellano left a comment

Choose a reason for hiding this comment

Uh oh!

Eric-Arellano Apr 5, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cosmicexplorer commented Jan 12, 2019 •

edited by Eric-Arellano

Loading

cosmicexplorer commented Jan 15, 2019 •

edited

Loading

cosmicexplorer commented Jan 15, 2019 •

edited

Loading

cosmicexplorer commented Apr 4, 2019 •

edited

Loading