meta: Runtime metrics stabilization

[carllerche]

Tracks the stabilization of runtime statistics.

RFC: #3845
PRs: #4043

Roadmap

• Release current implementation as unstable to crates.io (chore: prepare Tokio v1.11.0 #4083)
• Polish docs (remove TODO)
• Add more counters
• Write tokio-metrics providing a higher level api to be consumed that is easier to understand.
• Resolve open questions.
• Validate design by evaluating users' experience reports.
  • @Matthias247: graph

Open questions

• Naming: Should the type be named Metrics, Stats, or PerfCounters.
• Should RuntimeStats::workers() return &[WorkerStats] or an iterator.
• Should there be a feature flag to enable stats explicitly?
  • Should there be a runtime Builder option to enable / disable stats.
• Builder API for more complex configurations, like histograms (rt: instrument task poll times with a histogram #5685).
• inc_budget_forced_yield_count should become a per-worker metric.
• Should some current counters be lowered to internal counters?
  • steal_count
  • steal_operations
  • overflow_count

Additional counters

• Duration between last two polls.
  • There are open questions related to how this should be used.
• Worker queue depth
  • It is unclear how this should be tracked.

[carllerche]

@LucioFranco @Matthias247 any opportunities to try using the metrics?

[Matthias247]

I was on vacation and after that mostly block on other things. But I might be able to try this out this week.

I will however say upfront that I'll expect mostly to report back on the general accessor APIs and how integration into an application will look like. I think that putting poll_count/steal_count/etc on a dashboard will not be super useful for most people, because the numbers in itself have no significant meaning. They don't necessarily indicate that something is right or wrong. The not-yet-implemented timing metrics are more interesting, because they would indicate issues with code in tasks blocking too long. I will nevertheless check and see how the other metrics would look like.

[carllerche]

We should add a counter tracking the number of "false-positive" runtime wakeups. This would be incremented when a worker wakes up without having any work to do.

[e-ivkov]

The not-yet-implemented timing metrics are more interesting, because they would indicate issues with code in tasks blocking too long.

I second this. In our project built on tokio, we implemented a custom macro to track poll times. It would be very useful to have it in tokio.