feat: Synchronous Metrics Reader and Exporter

[Jayclifford345]

Is your feature request related to a problem?

Summary

Add a SynchronousExportingMetricReader to complement the existing PeriodicExportingMetricReader, supporting on-demand batch exporting of metrics rather than timer-based collection.

Problem Statement

The current PeriodicExportingMetricReader uses fixed intervals, which doesn't fit all use cases. There are scenarios which require explicit control over when metrics are collected and exported. It also means there is a feature gap between metrics compared to logs and traces, which can handle batch exporting.

Describe the solution you'd like

Solution

SynchronousExportingMetricReader provides:

• On-demand collection via explicit collect() calls
• Efficient batching of metrics before export
• Configurable queue and batch sizes
• No background threads, reducing resource usage

Real-World Use Cases

1. Serverless Functions - Lambda/Azure Functions need to export metrics before completion with no persistent background processes
2. Batch Jobs/ETL - Data pipelines that collect metrics at specific processing stages
3. CLI Tools - Short-lived applications that report metrics before exit
4. Web Applications - Metrics tied to specific HTTP requests rather than time intervals
5. Microservice Health Checks - Services reporting metrics on-demand when probed
6. Testing Environments - Precise control over metric collection during tests
7. Game Development - Collection at frame boundaries or level completions

Example Usage

reader = SynchronousExportingMetricReader(
    exporter,
    max_export_batch_size=5,
    max_queue_size=1000
)
meter_provider = MeterProvider(resource=resource, metric_readers=[reader])
meter = metrics.get_meter("my-meter")
counter = meter.create_counter("my_counter")

# Record and explicitly collect
counter.add(1)
reader.collect()  # Adds to queue and exports when batch size reached

Describe alternatives you've considered

Alternatives Considered

It could be possible to achieve similar behaviour with the PeriodicExportingMetricReader (setting collection interval to infinite and calling collect manually). However, it does not explicitly implement a queue and batching system. This function would bring feature parity between logs and traces.

Additional Context

References

PR with new SynchronousExportingMetricReader #4542

Would you like to implement a fix?

Yes

[aabmass]

However, it does not explicitly implement a queue and batching system. This function would bring feature parity between logs and traces.

Could you explain a little more the use case for queueing and batching? We don't have any additional batching in the spec for PeriodicExportingMetricReader because metrics are effectively already batched via aggregation. If you get 1000 counter.add() calls, it rolls up into one counter. On top of that, when you call MetricReader.collect() it provides a snapshot of all of the metrics in the SDK as a single batch. If you have very few metric streams, I could see a use case for batching though.

[Jayclifford345]

Hi @aabmass,
I completely see your point, this is more than likely an edge case on how I am using the SDK, but here is the use case:
In several of my projects, I lean heavily on Gauges, and I need a way to capture a series of numeric gauge readings in a single synchronous batch (for example, inside a unit test where I want to assert on the exact values emitted during that run).

Today, my options are:

• Lower the PeriodicExportingMetricReader interval so it fires frequently (but then you still get a send on every tick), or
• Use the “manual” workaround of disabling the timer and calling collect() yourself.

I mentioned on the PR that if this puts us in a tough spot, then I am happy to drop the feature request or turn it experimental to see if there is any interest. But I don't want to cause an issue with the specification for trying to implement out with agreed readers we already have. I ended up creating a PR: #4559 to add an example for users that need synchronous gauge collections using the method we discussed on the last SIG call.

I thought it also might be useful in a CICD where you might want to explicitly gauge readings and write them in batches before shutdown.

[aabmass]

Thanks for explaining. I agree OTel metrics don't support this well today since they aggregate to reduce data volume. Would generating a histogram work for you?

• Use SynchronousGauge and manually invoke an observation function in a background thread at a short interval (workaround for not being able to customize ObservableGauge's sampling rate)
• Add a View for the gauge with a histogram aggregation

You wouldn't keep every single measurement along the way, but you could sample more frequently without generating a ton of points. Or is it super important to keep every point?

Also, have you considered emitting Events (logs) with individual measurements?

[Jayclifford345]

Oh, agreed, I think a histogram is a good alternative, but it locks you into that data representation. If you have the option to save the raw gauge measurements, it allows you to transform those results later, but retain the exact metric values generated at the time of collection. I was thinking of a three-stage Kafka processing pipeline, to capture telemetry measurements at each stage; you may want to capture message size, transformation logic .etc. Though these can be stored in a histogram as well. I think with the workaround, I am happy to close out the issue, and then if it pops up, we could revisit.

[Jayclifford345]

I have closed out #4542 as out of scope, as I agree with the above options and workarounds. I think now that we have documented an example of the workaround, this should help future users with similar use cases. I mentioned in the closed PR that if the feature is asked for again in the future I am happy for the PR to be reopened or the code reused in another PR to solve the issue. Thanks for your analysis @aabmass and @lzchen I shall see you at the next Python sig call :)

[aabmass]

Great, thank you again @Jayclifford345