Skip to content

Commit d5fc680

Browse files
solssonsolsson
authored
Merge pull request #259 from Yolean/metrics-kafka-minion
Kafka-Minion as alternative to Burrow for consumer lag monitoring
2 parents 8aeff68 + b2eb754 commit d5fc680

17 files changed

+297
-3
lines changed

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ One option is to keep kubernets-kafka as a git submodule and edit the relative p
5959
Have a look at:
6060
* [./prometheus](./prometheus/)
6161
* [./linkedin-burrow](./linkedin-burrow/)
62+
* [./consumers-prometheus](./consumers-prometheus/)
6263
* [or plain JMX](https://github.com/Yolean/kubernetes-kafka/pull/96)
6364
* what's happening in the [monitoring](https://github.com/Yolean/kubernetes-kafka/labels/monitoring) label.
6465
* Note that this repo is intentionally light on [automation](https://github.com/Yolean/kubernetes-kafka/labels/automation). We think every SRE team must build the operational knowledge first.

‎consumers-prometheus/kafka-minion-service.yaml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
apiVersion: v1
2+
kind: Service
3+
metadata:
4+
name: metrics-minion
5+
namespace: kafka
6+
labels: &labels
7+
app: kafka-minion
8+
type: openmetrics
9+
spec:
10+
selector: *labels
11+
ports:
12+
- name: http
13+
port: 8080

‎consumers-prometheus/kafka-minion.yaml

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: metrics-minion
5+
namespace: kafka
6+
labels: &labels
7+
app: kafka-minion
8+
type: openmetrics
9+
spec:
10+
replicas: 1
11+
selector:
12+
matchLabels: *labels
13+
template:
14+
metadata:
15+
labels: *labels
16+
annotations:
17+
prometheus.io/scrape: "true"
18+
prometheus.io/port: "8080"
19+
prometheus.io/path: /metrics
20+
spec:
21+
containers:
22+
- name: kafka-minion
23+
image: solsson/kafka-consumers-prometheus@sha256:a005aa02581fa46884b4a4499f7b4d56e889770c64e7e84d0a9ad195935fa59c
24+
env:
25+
- name: TELEMETRY_HOST
26+
value: 0.0.0.0
27+
- name: TELEMETRY_PORT
28+
value: "8080"
29+
- name: EXPORTER_IGNORE_SYSTEM_TOPICS
30+
value: "true"
31+
- name: EXPORTER_METRICS_PREFIX
32+
value: kafka_minion
33+
- name: LOG_LEVEL
34+
value: info
35+
- name: KAFKA_BROKERS
36+
value: kafka-0.broker:9092, kafka-1.broker:9092, kafka-2.broker:9092
37+
- name: KAFKA_CONSUMER_OFFSETS_TOPIC_NAME
38+
value: __consumer_offsets
39+
ports:
40+
- name: http
41+
containerPort: 8080
42+
readinessProbe:
43+
httpGet:
44+
port: http
45+
path: /readycheck
46+
livenessProbe:
47+
httpGet:
48+
port: http
49+
path: /healthcheck

‎consumers-prometheus/kustomization.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
resources:
2+
- kafka-minion-service.yaml
3+
- kafka-minion.yaml

‎prometheus/50-kafka-jmx-exporter-patch.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# meant to be applied using
1+
# meant to be applied using kustomize, or with pre-1.14 kubectl:
22
# kubectl --namespace kafka patch statefulset kafka --patch "$(cat prometheus/50-kafka-jmx-exporter-patch.yml )"
33
apiVersion: apps/v1
44
kind: StatefulSet

‎prometheus/README.md

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,27 @@
11
# Export metrics to Prometheus
22

3-
Kafka uses JMX to expose metrics, as is already [enabled](https://github.com/Yolean/kubernetes-kafka/pull/96) for broker pods. There's many ways to use JMX. For example [Kafka Manager](../yahoo-kafka-manager/) uses it to display current broker traffic.
3+
JMX is already [enabled](https://github.com/Yolean/kubernetes-kafka/pull/96) for broker pods (TODO extract to kustomization). There's many ways to use JMX. For example [Kafka Manager](../yahoo-kafka-manager/) uses it to display current broker traffic.
44

5-
At Yolean we use Prometheus. This folder adds a sidecar to the broker pods that exports selected JMX metrics over HTTP in Prometheus format. To add a container to an existing pod we must use the `patch`command:
5+
This folder adds a sidecar to the broker pods that exports selected JMX metrics over HTTP in Prometheus format. To add a container to an existing pod we must use the `patch`command:
6+
7+
Using kubectl 1.14+
8+
9+
```
10+
kubectl --namespace kafka apply -k prometheus/
11+
```
12+
13+
Using pre-1.14 kubectl:
614

715
```
816
kubectl --namespace kafka apply -f prometheus/10-metrics-config.yml
917
kubectl --namespace kafka patch statefulset kafka --patch "$(cat prometheus/50-kafka-jmx-exporter-patch.yml )"
1018
```
19+
20+
## Consumer lag monitoring
21+
22+
See [Burrow](../linkedin-burrow)
23+
or [Kafka Minion](../consumers-prometheus/)
24+
25+
## Prometheus Operator
26+
27+
Use the [prometheus-operator](../variants/prometheus-operator/) kustomization.

‎prometheus/kustomization.yaml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
bases:
2+
# This kustomization needs to depend on one with kafka in it, to add the sidecar,
3+
# but it needs to be the kafka from the chosen variant, as ../kafka here would override other kustomizations
4+
#- ../kafka
5+
#- ../variants/scale-1
6+
resources:
7+
- 10-metrics-config.yml
8+
patchesStrategicMerge:
9+
- 50-kafka-jmx-exporter-patch.yml

‎variants/prometheus-operator-example/additional-scrape-configs.yaml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
2+
apiVersion: monitoring.coreos.com/v1
3+
kind: Prometheus
4+
metadata:
5+
name: k8s
6+
namespace: monitoring
7+
spec:
8+
additionalScrapeConfigs:
9+
# github.com/kubernetes-sigs/kustomize/blob/master/examples/kvSourceGoPlugin.md is clearly WIP
10+
name: additional-scrape-configs-5m4c7m6mc9
11+
# See https://github.com/prometheus/prometheus/pull/4131, and upon disagreement see https://github.com/prometheus/prometheus/issues/4484
12+
key: pods-discovery-by-prometheus-io-annotations.yaml

‎variants/prometheus-operator-example/alertmanager-main-scale-1.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: monitoring.coreos.com/v1
2+
kind: Alertmanager
3+
metadata:
4+
name: main
5+
namespace: monitoring
6+
spec:
7+
replicas: 1

‎variants/prometheus-operator-example/k8s-kafka-rbac.yaml

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Allows the "k8s" prometheus from Prometheus Operator contrib to do service discovery iin the kafka namespace
2+
---
3+
apiVersion: rbac.authorization.k8s.io/v1
4+
kind: Role
5+
metadata:
6+
name: prometheus-k8s
7+
namespace: kafka
8+
rules:
9+
- apiGroups:
10+
- ""
11+
resources:
12+
- services
13+
- endpoints
14+
- pods
15+
verbs:
16+
- get
17+
- list
18+
- watch
19+
---
20+
apiVersion: rbac.authorization.k8s.io/v1
21+
kind: RoleBinding
22+
metadata:
23+
name: prometheus-k8s
24+
namespace: kafka
25+
roleRef:
26+
apiGroup: rbac.authorization.k8s.io
27+
kind: Role
28+
name: prometheus-k8s
29+
subjects:
30+
- kind: ServiceAccount
31+
name: prometheus-k8s
32+
namespace: monitoring

0 commit comments

Comments
 (0)