Use Python 3 in h Docker image

[robertknight]

Transition the QA/production Docker image to use Python 3 to run the h web app and workers.

supervisord still requires Python 2, so to avoid having to install Python 2 alongside Python 3, I have replaced it with a Golang implementation which has all the features we need.

After this hits QA we should do some sanity testing of important parts of the site and check memory/CPU usage. After this hits production we should keep an eye on memory and CPU usage stats.

If we need to roll back for any reason, we can revert these commits and redeploy, or even just re-deploy the previous Docker image from the deployment build page.

Image size stats

Sizes reported by docker inspect hypothesis/hypothesis:dev:

Image size (master): 154MB
Image size (this branch): 167MB

Testing

1. Build the dev docker image using make docker
2. Run the image in a new container using make run-docker
3. Inspect the container and verify that processes are using the expected version of Python

[I] (h-py3)  ~/h/r/h > docker exec -it <ID of container> /bin/sh
~ $ ps aux 
PID   USER     TIME   COMMAND
    1 hypothes   0:00 {supervisord} /usr/bin/python2 /usr/bin/supervisord -c co
   11 hypothes   0:00 python /var/lib/hypothesis/bin/logger
   12 hypothes   0:00 {gunicorn} /usr/bin/python3.6 /usr/bin/gunicorn --name we
   13 hypothes   0:00 {gunicorn} /usr/bin/python3.6 /usr/bin/gunicorn --paste c
   14 hypothes   0:02 python -m h celery worker
   16 hypothes   0:00 nginx: master process nginx
   17 hypothes   0:00 nginx: worker process
   18 hypothes   0:00 nginx: worker process
   24 hypothes   0:01 {gunicorn} /usr/bin/python3.6 /usr/bin/gunicorn --paste c
   29 hypothes   0:02 {gunicorn} /usr/bin/python3.6 /usr/bin/gunicorn --name we
   35 hypothes   0:00 python -m h celery worker
   36 hypothes   0:00 python -m h celery worker
   44 hypothes   0:00 /bin/sh
   48 hypothes   0:00 ps aux
~ $ stat /proc/11/exe # check that "python" is Python 3
  File: '/proc/11/exe' -> '/usr/bin/python3.6'
  Size: 0         	Blocks: 0          IO Block: 1024   symbolic link
Device: a0h/160d	Inode: 1034058     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (  102/hypothesis)   Gid: (  103/hypothesis)
Access: 2018-03-21 08:56:10.000000000
Modify: 2018-03-21 08:56:10.000000000
Change: 2018-03-21 08:56:10.000000000

~ $ 

[codecov]

Codecov Report

Merging #4901 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4901      +/-   ##
==========================================
- Coverage   97.02%   97.02%   -0.01%     
==========================================
  Files         444      444              
  Lines       24643    24672      +29     
  Branches     1320     1330      +10     
==========================================
+ Hits        23910    23938      +28     
- Misses        606      607       +1     
  Partials      127      127

Impacted Files	Coverage Δ
h/activity/bucketing.py	96.42% <0%> (-1.01%)	⬇️
h/search/parser.py	100% <0%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 99a9e9d...bd17b9d. Read the comment docs.

[seanh]

I wonder if we need to wait for a quiet time (e.g. when not so busy on groups work) to merge this...

[hmstepanek]

That's prob not be a bad idea @seanh.

[hmstepanek]

If supervisor requires python2-do we still need to keep python2 here?

[robertknight]

The py2-pip dependency takes care of that. Strictly speaking this was redundant before.

[seanh]

Something to watch out for: the click library (which we use to implement the hypothesis CLI) has complicated problems with Python 3: http://click.pocoo.org/5/python3/. We should make sure that the CLI works correctly with Python 3, including when run via Jenkins.

[robertknight]

I wonder if we need to wait for a quiet time (e.g. when not so busy on groups work) to merge this...

I agree it makes sense to pick a quiet time of day and sprint. I'd prefer to have a more definite timeline than "when we're done with groups work" so we can wrap this up. How about next Tuesday morning (London time) when it'll be the start of the sprint and fairly quiet traffic-wise?

If there is a problem, we can just revert the single commit here and schedule work for another time.

Something to watch out for: the click library (which we use to implement the hypothesis CLI) has complicated problems with Python 3: http://click.pocoo.org/5/python3/.

I'm using Python 3 in my local dev environment and haven't encountered issues with the CLI tools. I've also done some basic testing of critical commands (hypothesis {shell, migrate, search}) in the Docker image.

From reading the docs, the main issues flagged up are to do with CLI tools that read binary data from stdin/write to stdout. AFAIK, none of ours do. Regarding the section at the bottom about I/O encoding, Python is configured to use UTF-8 in the Docker image which also avoids several hazards.

[hmstepanek]

I think the start of the sprint sounds good.

[seanh]

I'd like to wait at least another week from next Tuesday. I'd like some time to do a little Python 3 review of the code myself, but I'll be focusing on the groups stuff in the current sprint until end of Tuesday. The following Tuesday should work.

[robertknight]

The following Tuesday should work.

OK, that's fine with me. Unless there are any objections I will mark Tuesday 10th April as 🐍🐍🐍-day.

[robertknight]

Thanks for the detailed investigation @seanh. I hadn't responded yet because I wanted to focus on other development tasks this week.

I personally don't think it is absolutely necessary to do everything on that list before we switch over, in particular updating all dependencies (which has its own risks) and trying to get pylint clean. However I'm happy to let you noodle a way for a while and see what issues you can find. I'm sure it will be educational 🙂

[robertknight]

This may be useful to resolve the issue with supervisord still requiring Python 2 to be installed: https://github.com/ochinchina/supervisord (supervisord written in Go, and thus compiled to a static binary)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[seanh]

I've deleted the big checklist comments above and moved them to a new issue: #5384