nginx-proxy container broken with newer Docker and cgroups2

[mejo-]

Apparently docker-gen, which is used to dynamically generate the nginx config in nginx-proxy, doesn't work with newer Docker and cgroups2. I hit this bug after upgrading my system to Debian Bullseye (with Docker 20.10.3).

The result is that the nginx config at /etc/nginx/conf.d/default.conf is broken, leading to error no servers inside upstream when nginx is due to be restarted.

The bug is tracked in an upstream nginx-proxy issue (nginx-proxy/nginx-proxy#1548) and there's even two pull requests to fix it in underlying docker-gen: nginx-proxy/docker-gen#335 and nginx-proxy/docker-gen#336.

I was able to locally fix the problem by unpacking docker-gen-linux-amd64-0.7.3-29-g84b3509.tar.gz from https://github.com/steffen-p/docker-gen/releases (compiled docker-gen binaries with PR nginx-proxy/docker-gen#335) to docker/bin/docker-gen and applying the following patch:

diff --git a/docker/Dockerfile.nginx b/docker/Dockerfile.nginx
index 7f8bf99..d82bdeb 100644
--- a/docker/Dockerfile.nginx
+++ b/docker/Dockerfile.nginx
@@ -1,4 +1,5 @@
 FROM jwilder/nginx-proxy
+ADD bin/docker-gen /usr/local/bin/
 RUN { \
       echo 'server_tokens off;'; \
       echo 'client_max_body_size 100m;'; \

The issue will probably be solved by 1) Docker fixing its cgroups2 implementation and/or 2) docker-gen merging one of the two linked PRs. Nevertheless I thought it would be helpful to have the issue documented here as others who use nextcloud-docker-dev might run into it sooner or later.

[juliusknorr]

Thanks for the report and investigation already :) I assume we could just extend the dockerfile to also just download and put the binary into place then.

[mejo-]

Sure, that should be possible easily. In general I don't like the idea to automatically pull random binaries from the internet, but I guess that's nitpicking in this context 😉

I would suggest to wait a few days whether the upstream author (@jwilder) merges the fix and if not go ahead and implement the workaround here?

[mejo-]

The upstream fix (PR #336) has been merged now, so this issue should be fixed. I didn't have time to verify this myself yet, but will close this issue for now and reopen in case it's not fixed 😉

[juliusknorr]

Just tried as I'm also on docker 20.10.x now but it still seems to be broken nginx-proxy/docker-gen#336 (comment)

I didn't have time to investigate yet unfortunately

[juliusknorr]

Waiting for nginx-proxy/nginx-proxy#1585 to become available

[juliusknorr]

Works again with the latest nginxproxy/nginx-proxy release:

docker pull nginxproxy/nginx-proxy:latest
docker-compose build proxy
docker-compose up -d --force-recreate proxy