Burrow input: Topic filter applied to partition metrics

[alvarolmedo]

Relevant telegraf.conf:

[[inputs.burrow]]
  ## Burrow API endpoints in format "schema://host:port".
  ## Default is "http://localhost:8000".
  servers = ["http://localhost:8000"]

  ## Filter topics, default is no filtering.
  ## Values can be specified as glob patterns.
  # topics_include = []
  topics_exclude = [ "__consumer_offsets" ]

System info:

Telegraf 1.10.3.
CentOS 7 / RedHat 7

Steps to reproduce:

1. Use a burrow instance to get metrics with telegraf input. Config the address in server config var. The burrow config point to a kafka cluster with activer consumers and producers.

2. Prepare the next two files of configuration

[root@Tools ~]# grep -v "#" /etc/telegraf/telegraf.d/burrow.conf
[[inputs.burrow]]
  servers = ["http://localhost:8000"]

[root@Tools ~]# grep -v "#" /etc/telegraf/telegraf.d/burrow_filter.conf 
[[inputs.burrow]]
  servers = ["http://localhost:8000"]

  topics_exclude = [ "__consumer_offsets" ]

2. Execute telegraf with --test. Observe the burrow_topic, the filter is applied:

[root@Tools ~]# telegraf --test --config /etc/telegraf/telegraf.d/burrow_filter.conf | grep burrow_topic
2019-07-04T11:54:19Z I! Starting Telegraf 1.10.3
> burrow_topic,cluster=local,host=Tools,partition=0,topic=topictest offset=0i 1562241260000000000
> burrow_topic,cluster=local,host=Tools,partition=1,topic=topictest offset=1i 1562241260000000000
> burrow_topic,cluster=local,host=Tools,partition=2,topic=topictest offset=0i 1562241260000000000
[root@Tools ~]# telegraf --test --config /etc/telegraf/telegraf.d/burrow.conf | grep burrow_topic
2019-07-04T11:54:25Z I! Starting Telegraf 1.10.3
> burrow_topic,cluster=local,host=Tools,partition=0,topic=topictest offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=1,topic=topictest offset=1i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=2,topic=topictest offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=0,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=1,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=2,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=3,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=4,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=5,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=6,topic=__consumer_offsets offset=3i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=7,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=8,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=9,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=10,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=11,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=12,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=13,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=14,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=15,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=16,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=17,topic=__consumer_offsets offset=2i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=18,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=19,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=20,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=21,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=22,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=23,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=24,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=25,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=26,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=27,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=28,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=29,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=30,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=31,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=32,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=33,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=34,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=35,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=36,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=37,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=38,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=39,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=40,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=41,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=42,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=43,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=44,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=45,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=46,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=47,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=48,topic=__consumer_offsets offset=0i 1562241266000000000
> burrow_topic,cluster=local,host=Tools,partition=49,topic=__consumer_offsets offset=0i 1562241266000000000

Checking burrow_partition without filter:

[root@Tools ~]# telegraf --test --config /etc/telegraf/telegraf.d/burrow.conf | grep burrow_partition | grep __consumer_offsets | wc -l
2019-07-04T12:01:41Z I! Starting Telegraf 1.10.3
53

Expected behavior:

Checking burrow_partition with filter:

[root@Tools ~]# telegraf --test --config /etc/telegraf/telegraf.d/burrow_filter.conf | grep burrow_partition | grep __consumer_offsets | wc -l
2019-07-04T12:01:41Z I! Starting Telegraf 1.10.3
3

Actual behavior:

Checking burrow_partition with filter:

[root@Tools ~]# telegraf --test --config /etc/telegraf/telegraf.d/burrow_filter.conf | grep burrow_partition | grep __consumer_offsets | wc -l
2019-07-04T12:01:41Z I! Starting Telegraf 1.10.3
53

Additional info:

[danielnelson]

Closed in #6070