feat: Improve error handling in prometheus exporter #7363

tongoss · 2025-09-15T01:41:57Z

codecov · 2025-09-15T01:44:33Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.9%. Comparing base (a817caa) to head (12f6459).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #7363     +/-   ##
=======================================
+ Coverage   85.7%   85.9%   +0.1%     
=======================================
  Files        289     289             
  Lines      25377   25382      +5     
=======================================
+ Hits       21773   21819     +46     
+ Misses      3223    3191     -32     
+ Partials     381     372      -9

Files with missing lines	Coverage Δ
exporters/prometheus/exporter.go	`90.7% <100.0%> (+7.2%)`	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dashpole

thanks for taking this on!

CHANGELOG.md

exporters/prometheus/exporter.go

exporters/prometheus/exporter_test.go

MrAlias · 2025-09-25T17:08:32Z

@tongoss are you still able to work on this?

tongoss · 2025-09-25T17:24:17Z

@tongoss are you still able to work on this?

@MrAlias , my sincere apologies for the delay. I am able to continue working on this. I will try to get PR ready for review by the end of this week.

tongoss · 2025-09-26T00:56:26Z

@MrAlias , this PR is ready for another review.

dashpole

Great work

## Overview ### Added - Greatly reduce the cost of recording metrics in `go.opentelemetry.io/otel/sdk/metric` using hashing for map keys. (#7175) - Add `WithInstrumentationAttributeSet` option to `go.opentelemetry.io/otel/log`, `go.opentelemetry.io/otel/metric`, and `go.opentelemetry.io/otel/trace` packages. This provides a concurrent-safe and performant alternative to `WithInstrumentationAttributes` by accepting a pre-constructed `attribute.Set`. (#7287) - Add experimental observability for the Prometheus exporter in `go.opentelemetry.io/otel/exporters/prometheus`. Check the `go.opentelemetry.io/otel/exporters/prometheus/internal/x` package documentation for more information. (#7345) - Add experimental observability metrics in `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`. (#7353) - Add temporality selector functions `DeltaTemporalitySelector`, `CumulativeTemporalitySelector`, `LowMemoryTemporalitySelector` to `go.opentelemetry.io/otel/sdk/metric`. (#7434) - Add experimental observability metrics for simple log processor in `go.opentelemetry.io/otel/sdk/log`. (#7548) - Add experimental observability metrics in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`. (#7459) - Add experimental observability metrics in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`. (#7486) - Add experimental observability metrics for simple span processor in `go.opentelemetry.io/otel/sdk/trace`. (#7374) - Add experimental observability metrics in `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#7512) - Add experimental observability metrics for manual reader in `go.opentelemetry.io/otel/sdk/metric`. (#7524) - Add experimental observability metrics for periodic reader in `go.opentelemetry.io/otel/sdk/metric`. (#7571) - Support `OTEL_EXPORTER_OTLP_LOGS_INSECURE` and `OTEL_EXPORTER_OTLP_INSECURE` environmental variables in `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#7608) - Add `Enabled` method to the `Processor` interface in `go.opentelemetry.io/otel/sdk/log`. All `Processor` implementations now include an `Enabled` method. (#7639) - The `go.opentelemetry.io/otel/semconv/v1.38.0` package. The package contains semantic conventions from the `v1.38.0` version of the OpenTelemetry Semantic Conventions. See the [migration documentation](./semconv/v1.38.0/MIGRATION.md) for information on how to upgrade from `go.opentelemetry.io/otel/semconv/v1.37.0.`(#7648) ### Changed - `Distinct` in `go.opentelemetry.io/otel/attribute` is no longer guaranteed to uniquely identify an attribute set. Collisions between `Distinct` values for different Sets are possible with extremely high cardinality (billions of series per instrument), but are highly unlikely. (#7175) - `WithInstrumentationAttributes` in `go.opentelemetry.io/otel/trace` synchronously de-duplicates the passed attributes instead of delegating it to the returned `TracerOption`. (#7266) - `WithInstrumentationAttributes` in `go.opentelemetry.io/otel/meter` synchronously de-duplicates the passed attributes instead of delegating it to the returned `MeterOption`. (#7266) - `WithInstrumentationAttributes` in `go.opentelemetry.io/otel/log` synchronously de-duplicates the passed attributes instead of delegating it to the returned `LoggerOption`. (#7266) - Rename the `OTEL_GO_X_SELF_OBSERVABILITY` environment variable to `OTEL_GO_X_OBSERVABILITY` in `go.opentelemetry.io/otel/sdk/trace`, `go.opentelemetry.io/otel/sdk/log`, and `go.opentelemetry.io/otel/exporters/stdout/stdouttrace`. (#7302) - Improve performance of histogram `Record` in `go.opentelemetry.io/otel/sdk/metric` when min and max are disabled using `NoMinMax`. (#7306) - Improve error handling for dropped data during translation by using `prometheus.NewInvalidMetric` in `go.opentelemetry.io/otel/exporters/prometheus`. ⚠️ **Breaking Change:** Previously, these cases were only logged and scrapes succeeded. Now, when translation would drop data (e.g., invalid label/value), the exporter emits a `NewInvalidMetric`, and Prometheus scrapes **fail with HTTP 500** by default. To preserve the prior behavior (scrapes succeed while errors are logged), configure your Prometheus HTTP handler with: `promhttp.HandlerOpts{ ErrorHandling: promhttp.ContinueOnError }`. (#7363) - Replace fnv hash with xxhash in `go.opentelemetry.io/otel/attribute` for better performance. (#7371) - The default `TranslationStrategy` in `go.opentelemetry.io/exporters/prometheus` is changed from `otlptranslator.NoUTF8EscapingWithSuffixes` to `otlptranslator.UnderscoreEscapingWithSuffixes`. (#7421) - Improve performance of concurrent measurements in `go.opentelemetry.io/otel/sdk/metric`. (#7427) - Include W3C TraceFlags (bits 0–7) in the OTLP `Span.Flags` field in `go.opentelemetry.io/exporters/otlp/otlptrace/otlptracehttp` and `go.opentelemetry.io/exporters/otlp/otlptrace/otlptracegrpc`. (#7438) - The `ErrorType` function in `go.opentelemetry.io/otel/semconv/v1.37.0` now handles custom error types. If an error implements an `ErrorType() string` method, the return value of that method will be used as the error type. (#7442) ### Fixed - Fix `WithInstrumentationAttributes` options in `go.opentelemetry.io/otel/trace`, `go.opentelemetry.io/otel/metric`, and `go.opentelemetry.io/otel/log` to properly merge attributes when passed multiple times instead of replacing them. Attributes with duplicate keys will use the last value passed. (#7300) - The equality of `attribute.Set` when using the `Equal` method is not affected by the user overriding the empty set pointed to by `attribute.EmptySet` in `go.opentelemetry.io/otel/attribute`. (#7357) - Return partial OTLP export errors to the caller in `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`. (#7372) - Return partial OTLP export errors to the caller in `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#7372) - Return partial OTLP export errors to the caller in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`. (#7372) - Return partial OTLP export errors to the caller in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`. (#7372) - Return partial OTLP export errors to the caller in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`. (#7372) - Return partial OTLP export errors to the caller in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`. (#7372) - Fix `AddAttributes`, `SetAttributes`, `SetBody` on `Record` in `go.opentelemetry.io/otel/sdk/log` to not mutate input. (#7403) - Do not double record measurements of `RecordSet` methods in `go.opentelemetry.io/otel/semconv/v1.37.0`. (#7655) - Do not double record measurements of `RecordSet` methods in `go.opentelemetry.io/otel/semconv/v1.36.0`. (#7656) ### Removed - Drop support for [Go 1.23]. (#7274) - Remove the `FilterProcessor` interface in `go.opentelemetry.io/otel/sdk/log`. The `Enabled` method has been added to the `Processor` interface instead. All `Processor` implementations must now implement the `Enabled` method. Custom processors that do not filter records can implement `Enabled` to return `true`. (#7639)

improve error handling in prometheus exporter

098985d

tongoss force-pushed the feature/prometheus-exporter-error-handling branch from 0a224d0 to 098985d Compare September 15, 2025 03:06

tongoss marked this pull request as ready for review September 15, 2025 03:17

tongoss requested review from MrAlias, XSAM, dashpole, dmathieu, flc1125 and pellared as code owners September 15, 2025 03:17

dashpole reviewed Sep 15, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

exporters/prometheus/exporter.go Outdated Show resolved Hide resolved

exporters/prometheus/exporter_test.go Show resolved Hide resolved

exporters/prometheus/exporter_test.go Outdated Show resolved Hide resolved

tongoss marked this pull request as draft September 16, 2025 20:17

MrAlias added this to the v1.39.0 milestone Sep 18, 2025

tongoss and others added 6 commits September 25, 2025 16:51

create error variables

fb98400

Merge branch 'main' into feature/prometheus-exporter-error-handling

404b0b9

fix unit tests and address PR comment

87ff8d0

address PR comments

c6c6ad3

update error variables

63ae765

Merge branch 'main' into feature/prometheus-exporter-error-handling

969051b

tongoss marked this pull request as ready for review September 26, 2025 00:52

dashpole approved these changes Sep 26, 2025

View reviewed changes

MrAlias approved these changes Oct 9, 2025

View reviewed changes

MrAlias added 2 commits October 9, 2025 13:06

Merge branch 'main' into feature/prometheus-exporter-error-handling

729da43

Merge branch 'main' into feature/prometheus-exporter-error-handling

12f6459

MrAlias merged commit 874c4c3 into open-telemetry:main Oct 9, 2025
33 checks passed

MrAlias mentioned this pull request Dec 5, 2025

Release v1.39.0 #7676

Merged

MrAlias mentioned this pull request Dec 8, 2025

The Prometheus exporter fails with an error by default #7686

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Improve error handling in prometheus exporter #7363

feat: Improve error handling in prometheus exporter #7363

Uh oh!

tongoss commented Sep 15, 2025

Uh oh!

codecov bot commented Sep 15, 2025 •

edited

Loading

Uh oh!

dashpole left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MrAlias commented Sep 25, 2025

Uh oh!

tongoss commented Sep 25, 2025

Uh oh!

tongoss commented Sep 26, 2025

Uh oh!

dashpole left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: Improve error handling in prometheus exporter #7363

feat: Improve error handling in prometheus exporter #7363

Uh oh!

Conversation

tongoss commented Sep 15, 2025

Uh oh!

codecov bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dashpole left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MrAlias commented Sep 25, 2025

Uh oh!

tongoss commented Sep 25, 2025

Uh oh!

tongoss commented Sep 26, 2025

Uh oh!

dashpole left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Sep 15, 2025 •

edited

Loading