graceful shutdowns

[biw]

Hello 👋, I'm having a little trouble making sure that my server restarts gracefully. I have my script waiting for SIGTERM but it doesn't seem like they get called, is there a way I can watch for them and pass them to my (node) app? 😄

[garhiggins]

I've been struggling with this also. We forked the start script and trapped the SIGTERM and then send the nginx -s QUIT, but it doesn't seem to help. I believe the problem is that Heroku sends a SIGTERM to all processes (https://devcenter.heroku.com/articles/dynos#shutdown), not just the bash script, so nginx still sees a SIGTERM and does a fast shutdown. We drop a ton of requests every time we deploy :(

[sidonath]

Ouch!

I looked into the source and there's a TODO: #ifndef next to

#define NGX_SHUTDOWN_SIGNAL      QUIT
#define NGX_TERMINATE_SIGNAL     TERM

If we could land a change in Nginx to wrap those two within #ifndef indeed, we could compile a Heroku-friendlier version of Nginx that treats SIGTERM as SIGQUIT. Not sure if that's a good idea, but it would help with this issue.

[mguida22]

👀 Any updates on this?

[beanieboi]

@mguida22 this is outside of the scope of this buildpack. it has to do with the Heroku platform itself and how nginx works. there is nothing this buildpack can do.

[mguida22]

@beanieboi thanks for the reply!

Is there a better place to bring up this issue with the Heroku platform?

[beanieboi]

@mguida22 This is a known issue and since this i a behavior of the platform for a long time, i believe there is no way to fix it without breaking a lot of applications. I think our only chance is to patch nginx to react differently. We do this in the pgbouncer buildpack and it works ok.
https://github.com/heroku/heroku-buildpack-pgbouncer/blob/master/support/patchfiles/1.13.0/0001-Disable-SIGTERM.patch