Stream connections dropping on 0.5.35 release

[eliteprox]

Describe the bug

A few orchestrators have mentioned issues with the latest release (0.5.35). I upgraded on Monday and noticed a significant change in stream stability. Large spikes and crashes were common. There were a lot of “connection reset by peer” errors. After extensive troubleshooting, a rollback to 0.5.34 today restored stability and we performed better than any day this week.

I think PR #2628 might be related to these connections dropping prematurely.

To Reproduce
Steps to reproduce the behavior:

While operating orchestrator in production, observe logs for connection resets and monitor per stream metrics. Many are associated with “connection reset by peer” errors.

Expected behavior
After some research it seems the idle timeout may not be the best way to handle latent connections https://stackoverflow.com/questions/29334407/creating-an-idle-timeout-in-go

At first I thought perhaps 8 seconds was just too low, but it seems that separate Read and Write time-out may perform better than Idle timeout. There is also an event handler option. See stackoverflow link above.

Screenshots
None

Desktop (please complete the following information):

• OS: Ubuntu Linux 22.04

Additional context

[leszko]

Interesting. I think that setting ReadTimeout instead of IdleTimeout will not help here, because that's what's in the Golang docs for IdleTimeout:

If IdleTimeout is zero, the value of ReadTimeout is used. If both are zero, there is no timeout.

We could also use the ConnState callback (aka event handler), but I think it could only help if we wanted to have different timeout for read and different for idle. And for now, all we wanted is to close connections which are idle for a long time.

Since we introduced the IdleTimeout just to handle too many connections opened, I think that increasing IdleTimeout to a a few minutes should help. I've sent a PR here. Saying that, it's hard to test this issue, because I think it's not deterministic.

Wdyt? @yondonfu @thomshutt

[yondonfu]

The timeout for a transcode loop run internally by an O times is currently 1 minute so transcode sessions on an O are considered inactive and are cleaned up after 1 minute. So, I think it makes sense for the IdleTimeout for connections to be >= this transcode loop timeout - otherwise, the connection would be considered idle and would be torn down before the transcode session is considered inactive and cleaned up.