Prometheus Input Plugin: Ability to gather metrics from all Kubernetes pods behind a service

[meilke]

A Kubernetes service can represent an interface to one or many pods that would serve the functionality needed (for instance, a web API). Unfortunately, you cannot force communication with any pod in particular and thus, would run into problems collecting Prometheus metrics from all the pods behind a service.

Kubernetes has the ability to create headless services which allow more direct access to the pods behind it without knowing too much about the pods beforehand.

I added functionality to the Prometheus input plugin that allows doing a DNS query on the URLs provided and then querying metrics from all of the associated hosts. Thus, you are able to collect all metrics from all pods behind a Kubernetes service.

Assume, you have the following headless service at my-headless-service.my-namespace:

$ host my-headless-service.my-namespace
my-headless-service.my-namespace has address 10.1.2.11
my-headless-service.my-namespace has address 10.1.4.35

Specifying this:

urls = ["https://other-url/metrics"]
kubernetes_services = ["https://my-headless-service.my-namespace:8080/metrics"]

Will gather metrics from https://10.1.2.11:8080/metrics and https://10.1.4.35:8080/metrics and will even tag them with a label called address.

Required for all PRs:

• Signed CLA.
• Associated README.md updated.
• Has appropriate unit tests.

[danielnelson]

Thank you for the pull request, I have a few ideas for this feature:

[danielnelson]

Use a different value for the tagkey, since Telegraf's default config adds the host tag to all points as the hostname Telegraf is running on. I propose calling it address. We should also rename the config option to go along with this.

[meilke]

Done!

[danielnelson]

Don't add this here, since *.swp are not created by the project but your text editor. (I should actually remove most of this file) Here is how text editor files should be handle IMO:

$ cat ~/.gitconfig
... snip ...
[core]
	excludesfile = /home/dbn/.gitignore
... snip ...
$ cat ~/.gitignore
tags
*.swp
*.pyc
.noseids

[meilke]

Awesome, did not know that, makes perfect sense!

[danielnelson]

You can use URL.Hostname() and URL.Port() for this.

[meilke]

Done!

[danielnelson]

This can be done with URL and its String() function.

[meilke]

Done!

[danielnelson]

I think we should have a separate list of records for dns lookup instead of a boolean, it could be called kubernetes_services since I believe this method is slightly different from standard DNS-SD. I think it would make sense to have these items automatically have the "host" tag.

This will allow combination setups more easily and also I am trying to use less boolean options since they tend to get confusing if you have too many of them.

[meilke]

Done!

[danielnelson]

Don't forget to run go fmt as well so that the tests can pass.