unit "bytes" not a suffix of metric "process_runtime_dotnet_gc_allocations_size_bytes_total"

[RoyLi-Pvt]

Bug Report

List of all OpenTelemetry NuGet packages and version using:

OpenTelemetry.Exporter.Console" Version="1.8.0"
OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.8.0"
OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.8.0-rc.1"
OpenTelemetry.Extensions.Hosting" Version="1.8.0"
OpenTelemetry.Instrumentation.AspNetCore" Version="1.7.1"
OpenTelemetry.Instrumentation.EntityFrameworkCore" Version="1.0.0-beta.10"
OpenTelemetry.Instrumentation.GrpcNetClient" Version="1.7.0-beta.1"
OpenTelemetry.Instrumentation.Http" Version="1.7.1"
OpenTelemetry.Instrumentation.Process" Version="1.0.0-alpha.6"
OpenTelemetry.Instrumentation.Runtime" Version="1.7.0"

Runtime version:

net8.0

Symptom

Prometheus pull error msg : unit "bytes" not a suffix of metric "process_runtime_dotnet_gc_allocations_size_bytes_total"
Prometheus Build Info:
Version 2.51.1
Rev 855b5ac4b80956874eb1790a04c92327f2f99e38

Init Code:

  Telemetry.WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddRuntimeInstrumentation()
            .AddProcessInstrumentation()
            .AddMeter("Microsoft.AspNetCore.Hosting")
            .AddMeter("Microsoft.AspNetCore.Routing")
            .AddMeter("Microsoft.AspNetCore.Diagnostics")
            .AddMeter("Microsoft.AspNetCore.Server.Kestrel")
            .AddMeter("Microsoft.AspNetCore.Http.Connections")
            .AddMeter("Microsoft.Extensions.Diagnostics.HealthChecks")
            .SetMaxMetricStreams(500)
            .SetMaxMetricPointsPerMetricStream(200)
            .AddPrometheusExporter();
    });

This error occurred after upgrading the version of OpenTelemetry.Exporter.Prometheus.AspNetCore to 1.8.0-rc1 and no longer occurred after downgrading to 1.7.0-rc1.

[xiang17]

This is the same issue as open-telemetry/opentelemetry-dotnet-contrib#1617.

You can configure the Prometheus exporter to disable the _total suffix with this option: AddPrometheusExporter(o => o.DisableTotalNameSuffixForCounters = true). The option is included in releases starting with 1.8.0-beta.1.

[hacst]

@xiang17 Could you (or someone else familiar) clarify whether this is a problem with AddRuntimeInstrumentation of the contrib that should be fixed there, or something to be fixed in the core implementation here? Having the workaround with the parameter is good to know, but it is non-default and the update breaking previously working exports is quite concerning.

All of this becoming only visible when prometheus fails to get metrics for a service after update is not ideal either. Shouldn't the implementation behind AddPrometheusExporter prevent us from (silently) producing output prometheus cannot handle?

[bIgBV]

Agreed, we also ran into this issue after deploying a service change which included the AddRuntimeInstrumentation call. Ideally, if it is a known breaking issue, wouldn't it be better to set the DisableTotalNameSuffixForCounters to true by default?

[kamisoft-fr]

This is the same issue as open-telemetry/opentelemetry-dotnet-contrib#1617.

You can configure the Prometheus exporter to disable the _total suffix with this option: AddPrometheusExporter(o => o.DisableTotalNameSuffixForCounters = true). The option is included in releases starting with 1.8.0-beta.1.

Same problem here with npgsql metrics, this workarround does not work in this case:

it transforms "db_client_commands_bytes_written_total" in "db_client_commands_bytes_written"

so we still get unit "bytes" not a suffix of metric "db_client_commands_bytes_written"