Consider more efficient protobuf implementations

[beorn7]

Since we will have significant breaking changes in 0.10, this is a good time to consider using https://github.com/gogo/protobuf instead of the current stock protobuf implementation. In high frequency scrape scenarios, the resource consumption of the metric serving has proven to be considerable, and https://github.com/gogo/protobuf is an easy way to cut out a lot of resource usage.

[peergynt]

Prometheus 2.0 no longer supports the protobuf exposition format. Maybe protobuf should be removed from the Go client altogether.

[beorn7]

On the other hand, the Go client is not only meant for the Prometheus server (and even less so for a specific version of it). An informed decision will depend on the outcome of OpenMetrics, which is the effort to create a standardized metrics exposition format.

[peergynt]

Fair enough. What do you think about separating the core metrics processing (counters, gauges, histogram, registry...) from the exposition/http side of things?
It might be nice to have a metrics-only module decoupled from protobuf/http.

[beorn7]

In a certain view, this is already the case. promhttp has the HTTP exposition, graphite the push to Graphite etc., while the prometheus package is the core (all HTTP parts still there are deprecated and soon to be removed.) Of course, the core prometheus package contains protobuf stuff. But the Go client was designed with the idea in mind that metrics.proto is the data model. This original idea was later abandoned in the Prometheus project (in a rather uninformed way, IMHO, but that's a different story), but if I understood correctly, OpenMetrics will have a similar idea, i.e. the data model will be expressed an protobuf, while the actual exposition format might/will be quite different on the wire (similar to the current Prometheus text format).

[beorn7]

Relevant discussion: https://groups.google.com/g/prometheus-developers/c/uFWRyqZaQis

[beorn7]

I mostly assigned this to myself as a “default” because I used to be the maintainer of this repo. Therefore, I'm un-assigning this from myself no. New maintainers @bwplotka & @kakkoyun, please deal with this as you see fit.

[kakkoyun]

It seems like gogo/protobuf is not maintained anymore. There's a new kid in town, it could be a nice candidate for replacement https://vitess.io/blog/2021-06-03-a-new-protobuf-generator-for-go/

[bwplotka]

Do we need protobuf at all in v2? Or it is needed for histograms.

We maintain unused protocol https://github.com/prometheus/client_model but with histograms we start to use it more. Still definitely not gogo, but vitess one but might be better choice. 👍🏽

[beorn7]

Yeah, it's complicated…

If we looked purely at the state of Prometheus 2.x and OpenMetrics, then it would appear quite obvious that any future version of client_golang should completely ignore protobuf.

However, the planned shiny new high-res sparse histograms will most likely benefit immensely from a binary format, which might very well be protobuf. (But maybe someone finds a good way of representing the new histograms efficiently in a text format? And even for a binary format, we might settle on something else than protobuf. Both options aren't very likely, IMHO, though…)

But even if we assume the new histograms will work much better with protobuf, it's unlikely that we will just return to a slightly extended version of the old protobuf format. (We are doing exactly that for playing around with the new histograms, but that's really just because it was so easy given that v1 of client_golang still fully supports the old protobuf format.) OpenMetrics already has an (optional) protobuf representation. I believe it is most likely that there will be an OpenMetrics v2 to support the new sparse histogram, and it will have a stronger emphasis on its protobuf side, and that's probably the one we should go for for v2 of client_golang. But that's just guesswork for now. First we have to see how the new sparse histograms will work out and what implication that has for the final format.

[beorn7]

And coming back to the actual point: Once we know protobuf will stay a thing in the Prometheus universe, clearly we shouldn't switch everything over to an unmaintained protobuf implementation. The Vitess one looks quite promising, but needs benchmarking. The choice to be made is much more acute for Prometheus remote-write, as that's definitely protobuf and high throughput and used a lot all over the world, so from the client_golang perspective, we should just wait and see what remote-write settles on, and then we can do exactly the same over here in client_golang land.