Use structured logging and send it to Sentry

[jyn514]

Sentry has been setup already following the instructions in https://forge.rust-lang.org/infra/docs/sentry.html. We could add sentry integration to docs.rs, see rust-lang/crates.io#2961 for an example. There are docs at https://docs.sentry.io/platforms/rust/ and https://docs.rs/sentry/0.20.1/sentry/#quickstart.

Note that sentry has a drop guard for flushing messages, so we should consider implementing #1150 first to avoid losing logs.

[jyn514]

Integrating this with rustwide will require waiting until the next sentry release so we get the changes in getsentry/sentry-rust#268.

[syphar]

Integrating this with rustwide will require waiting until the next sentry release so we get the changes in getsentry/sentry-rust#268.

It seems it was released in 0.21.0, so we could give this another go.

Note that sentry has a drop guard for flushing messages, so we should consider implementing #1150 first to avoid losing logs.

so the remaining question is, does sentry have value for us even without graceful shutdown?
It seems like the errors that are interesting for us happen when answering requests, right? So we could do without the shutdown drop-guard, and revisit the topic after the warp migration (whenever that happens)?

Or should we find a quickfix for graceful shutdown first?

[jyn514]

so the remaining question is, does sentry have value for us even without graceful shutdown?

Absolutely yes; we may need to explicitly call flush on a schedule or something, but that's still a ton more alerts than "none at all". We just might lose a few when restarting the server, which is infrequent.

It seems it was released in 0.21.0, so we could give this another go.

I do not have time to tackle this at the moment :( happy for someone else to pick it up