Problems with prometheus on OTel 1.8 #5613
Description
Area
area:exporter:prometheus
Package Version
| Package Name | Version |
|---|---|
| OpenTelemetry.Api | 1.8.0 |
| OpenTelemetry | 1.8.0 |
| OpenTelemetry.Exporter.OpenTelemetryProtocol | 1.8.0 |
| OpenTelemetry.Exporter.Prometheus.AspNetCore | .8.0-rc.1 |
| OpenTelemetry.Exporter.Prometheus.HttpListener | 1.8.0-rc.1 |
| OpenTelemetry | 1.8.0 |
Runtime Version
net8.0
Description
After update to new Otel 1.8 we are expirience issues in our prometheus. In one version it says "Error on ingesting samples that are too old or are too far into the future" and after cross check with new version of prometheus it says "Error on ingesting samples with different value but same timestamp". Additionally we observe that on some random queries to /metrics (we are getting otel_scope_info that is missing timestamp like described here prometheus/prometheus#6554). Like below
If I'm getting this right this info should be returned only when "Accept: application/openmetrics-text" is send but maybe I'm wrong. Additionally this info is random - one query return metrics with this info and another without.
If this behavior is by design please share some documentation.
ps. After downgrade to 1.7 we don't have those errors
Steps to Reproduce
Just use standard OTel and serve prometheus endpoint
Expected Result
Prometheus agent is grabbing data with success
Actual Result
Errors from description
Additional Context
No response