Implement timeout mechanism for several metrics components

[ocelotl]

collect doesn't return anything, so returning True until we implement timeout/error handling mechanism seemed reasonable to me. What do you think?

Originally posted by @lonewolf3739 in #2401 (comment)

Also, the timeout mechanism is required in several parts of the metrics spec:

1. MeterProvider.shutdown
2. MeterProvider.forceflush
3. Asynchronous callbacks
4. MetricReader.collect
5. MetricReader.shutdown
6. PushMetricExporter.forceflush
7. PushMetricExporter.shutdown

[ocelotl]

Just for the record: https://pypi.org/project/timeout-decorator/

[ocelotl]

Also #385

[aabmass]

Based on the linked issues, the previous decision was to not use the signal based approach for implementing generic timeout mechanism because it won't work outside of the main thread, and it can interfere with applications using the same signal handler already.

For some background,

• The Java methods return a future like object which lets user join() with a certain timeout
• JS doesn't appear to do anything special, just returns a Promise or accepts a callback. There is no way to cancel
• Go simply passes the ctx and tells the implementor to respect the cancellation/timeout
• .NET accepts timeout for shutdown and force flush which the implementor should respect

Based on that, I think it's reasonable to just pass down a timeout duration where possible and document that implementors should respect the timeout. This doesn't actually implement "SHOULD complete or abort within some timeout" but is not terribly opinionated.

It would be great if the mechanism we add for metrics can also be used in the tracing SDK, where timeouts were never implemented previously. The problem is that adding a timeout optional parameter to the SpanExporter, SpanProcessor etc. interface signatures will break existing implementations of those interfaces when we go to call them. We could potentially defensively call with the timeout param and fallback to not passing timeout e.g. here

try:
  self._active_span_processor.shutdown(timeout=timeout)
except TypeError:
  self._active_span_processor.shutdown()

[aabmass]

I think potentially we should try to add *args, **kwargs or config objects to any user implementable interfaces going forward to avoid this kind of issue.

[aabmass]

Looks like I lied about JS. They accept some timeout parameters as config options and apply them: https://github.com/open-telemetry/opentelemetry-js/blob/cfda625e83a164c9e19745392879c32aedcfe76f/packages/opentelemetry-sdk-trace-base/src/BasicTracerProvider.ts#L153

The obvious downside with this approach is you can't change the timeout at different call sites. I think it still satisfies the spec though. And we could add it without the try/except stuff.

[aabmass]

Ok.. new issue is the OTLP exporter accepts a timeout parameter on it's own. The really weird thing is, this is considered a timeout for just RPCs to the OTLP endpoint and we use an exponential backoff that will delay things way past the default of 10s all the way to 63 seconds plus whatever delay we had from the RPCs.

And on top of that.. the timeout for PeriodicExportingMetricReader and BatchSpanProcessor have a export_timeout_millis of 30s (way shorter than the max backoff), but neither of those classes pass the timeout to the exporter

opentelemetry-python/opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py

Line 143 in 29e4bab

export_timeout_millis: float = None,

I'm not even sure the correct behavior at this point