diff --git a/README.md b/README.md index f080d8c8..14cb18f5 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,6 @@ To quote [@arthurk](https://github.com/Yolean/kubernetes-kafka/issues/82#issueco ## Getting started We suggest you `apply -f` manifests in the following order: - * Your choice of storage classes from [./configure](./configure/) * [namespace](./00-namespace.yml) * [./rbac-namespace-default](./rbac-namespace-default/) * [./zookeeper](./zookeeper/) @@ -20,6 +19,8 @@ We suggest you `apply -f` manifests in the following order: That'll give you client "bootstrap" `bootstrap.kafka.svc.cluster.local:9092`. +With the introduction of [-k](#kustomize) support, apply `-f` will report `error: unable to decode "zookeeper/kustomization.yaml": Object 'Kind' is missing in ...`. With kubectl v1.14+ you can avoid that using: `kubectl apply -k variants/scale-3-5/`. + ## Fork Our only dependency is `kubectl`. Not because we dislike Helm or Operators, but because we think plain manifests make it easier to collaborate. @@ -30,21 +31,35 @@ If you begin to rely on this kafka setup we recommend you fork, for example to e With the introduction of [app customization](https://kubectl.docs.kubernetes.io/pages/app_customization/introduction.html) in `kubectl` 1.14 there's an alternative to forks. We as a community can maintain a set of overlays. See the [variants](./variants) folder for different overlays. For example to scale to 1 kafka broker try `kubectl apply -k variants/scale-1/`. +Variants also include examples of how to configure volumes for GKE, AWS and AKS with different storage classes. + +### Quickstart -Currently `apply -k` replaces `apply -f ./zookeeper; apply -f ./kafka`. -The original commands now result in `error: unable to decode "zookeeper/kustomization.yaml": Object 'Kind' is missing in ...` -and though they still seem to work you can get around that with a v1.14+ kubectl using: `kubectl apply -k variants/as-is/`. +``` +kubectl create namespace kafka && \ +kubectl apply -k github.com/Yolean/kubernetes-kafka/variants/dev-small/?ref=storagclass-kustomize +``` + +When all pods are Ready, test with for example `kafkacat -b localhost:9094 -L` over `kubectl -n kafka port-forward kafka-0 9094`. ### Maintaining your own kustomization -`kubectl apply -k` takes a single overlay, meaning that you can't compose different overlays from this repo. -You'll probably want to maintain your own variant. -One option is to keep kubernets-kafka as a git submodule and edit the relative path from an example variant. +Start your variant as a new folder in your choice of version control, with a base `kustomization.yaml` pointing to a tag or revision in this repository: + +``` +bases: +- github.com/Yolean/kubernetes-kafka/rbac-namespace-default/?ref=553f327 +- github.com/Yolean/kubernetes-kafka/kafka/?ref=553f327 +- github.com/Yolean/kubernetes-kafka/zookeeper/?ref=553f327 +``` + +Then pick and chose from patches our [example variants](https://github.com/Yolean/kubernetes-kafka/tree/master/variants) to tailor your Kafka setup. ## Version history | tag | k8s ≥ | highlights | | ------ | ----- | ----------- | +| v6.0.0 | 1.11+ | Kafka 2.2.0 + `apply -k` (kubectl 1.14+) | | v5.1.0 | 1.11+ | Kafka 2.1.1 | | v5.0.3 | 1.11+ | Zookeeper fix [#227](https://github.com/Yolean/kubernetes-kafka/pull/227) + [maxClientCnxns=1](https://github.com/Yolean/kubernetes-kafka/pull/230#issuecomment-445953857) | | v5.0 | 1.11+ | Destabilize because in Docker we want Java 11 [#197](https://github.com/Yolean/kubernetes-kafka/pull/197) [#191](https://github.com/Yolean/kubernetes-kafka/pull/191) | @@ -61,7 +76,7 @@ Have a look at: * [./linkedin-burrow](./linkedin-burrow/) * [or plain JMX](https://github.com/Yolean/kubernetes-kafka/pull/96) * what's happening in the [monitoring](https://github.com/Yolean/kubernetes-kafka/labels/monitoring) label. - * Note that this repo is intentionally light on [automation](https://github.com/Yolean/kubernetes-kafka/labels/automation). We think every SRE team must build the operational knowledge first. + * Note that this repo is intentionally light on [automation](https://github.com/Yolean/kubernetes-kafka/labels/automation). We think every SRE team must build the operational knowledge first. But there is an example of a [Cruise Control](./cruise-control/) setup. ## Outside (out-of-cluster) access @@ -69,13 +84,6 @@ Available for: * [Brokers](./outside-services/) -## Fewer than three nodes? - -For [minikube](https://github.com/kubernetes/minikube/), [youkube](https://github.com/Yolean/youkube) etc: - - * [Scale 1](https://github.com/Yolean/kubernetes-kafka/pull/44) - * [Scale 2](https://github.com/Yolean/kubernetes-kafka/pull/118) - ## Stream... * [Kubernetes events to Kafka](./events-kube/) diff --git a/configure/README.md b/configure/README.md deleted file mode 100644 index 227d40e0..00000000 --- a/configure/README.md +++ /dev/null @@ -1,5 +0,0 @@ -## Sample storage classes - -Volume sizes are configured in the respective statefulset, but for everything else adapt the examples here to your needs. Kafka is said to be optimized for local storage, but this repository has taken a more pragmatic approach, typically using the cloud provider's volumes. - -Kubernetes has lately introduced better support for [local volumes](https://kubernetes.io/docs/concepts/storage/volumes/#local) and for example [GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/local-ssd) has so too, but such options are yet to be explored here. diff --git a/configure/docker-storageclass-broker.yml b/configure/docker-storageclass-broker.yml deleted file mode 100644 index 4a997760..00000000 --- a/configure/docker-storageclass-broker.yml +++ /dev/null @@ -1,6 +0,0 @@ -kind: StorageClass -apiVersion: storage.k8s.io/v1 -metadata: - name: kafka-broker -provisioner: docker.io/hostpath -reclaimPolicy: Retain diff --git a/configure/docker-storageclass-zookeeper.yml b/configure/docker-storageclass-zookeeper.yml deleted file mode 100644 index 038f2c87..00000000 --- a/configure/docker-storageclass-zookeeper.yml +++ /dev/null @@ -1,6 +0,0 @@ -kind: StorageClass -apiVersion: storage.k8s.io/v1 -metadata: - name: kafka-zookeeper -provisioner: docker.io/hostpath -reclaimPolicy: Retain diff --git a/configure/minikube-storageclasses.yml b/configure/minikube-storageclasses.yml deleted file mode 100644 index 045699f1..00000000 --- a/configure/minikube-storageclasses.yml +++ /dev/null @@ -1,21 +0,0 @@ ---- -kind: StorageClass -apiVersion: storage.k8s.io/v1 -metadata: - name: kafka-broker -provisioner: k8s.io/minikube-hostpath -reclaimPolicy: Retain ---- -kind: StorageClass -apiVersion: storage.k8s.io/v1 -metadata: - name: kafka-zookeeper -provisioner: k8s.io/minikube-hostpath -reclaimPolicy: Retain ---- -kind: StorageClass -apiVersion: storage.k8s.io/v1 -metadata: - name: kafka-zookeeper-regional -provisioner: k8s.io/minikube-hostpath -reclaimPolicy: Retain diff --git a/kafka/50kafka.yml b/kafka/50kafka.yml index 73465387..8eb2d711 100644 --- a/kafka/50kafka.yml +++ b/kafka/50kafka.yml @@ -100,7 +100,7 @@ spec: name: data spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: kafka-broker + storageClassName: standard resources: requests: storage: 10Gi diff --git a/rbac-namespace-default/kustomization.yaml b/rbac-namespace-default/kustomization.yaml new file mode 100644 index 00000000..c2155fb0 --- /dev/null +++ b/rbac-namespace-default/kustomization.yaml @@ -0,0 +1,3 @@ +resources: +- node-reader.yml +- pod-labler.yml diff --git a/configure/aks-storageclass-broker-managed.yml b/variants/aks-managed/aks-storageclass-broker-managed.yaml similarity index 100% rename from configure/aks-storageclass-broker-managed.yml rename to variants/aks-managed/aks-storageclass-broker-managed.yaml diff --git a/configure/aks-storageclass-zookeeper-managed.yml b/variants/aks-managed/aks-storageclass-zookeeper-managed.yaml similarity index 100% rename from configure/aks-storageclass-zookeeper-managed.yml rename to variants/aks-managed/aks-storageclass-zookeeper-managed.yaml diff --git a/variants/aks-managed/kustomization.yaml b/variants/aks-managed/kustomization.yaml new file mode 100644 index 00000000..aca693de --- /dev/null +++ b/variants/aks-managed/kustomization.yaml @@ -0,0 +1,7 @@ +bases: +- ../scale-3-5 +resources: +- aks-storageclass-broker-managed.yaml +- aks-storageclass-zookeeper-managed.yaml +patchesStrategicMerge: +- volume-claims.yaml diff --git a/variants/aks-managed/volume-claims.yaml b/variants/aks-managed/volume-claims.yaml new file mode 100644 index 00000000..fb20db89 --- /dev/null +++ b/variants/aks-managed/volume-claims.yaml @@ -0,0 +1,32 @@ +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: kafka + namespace: kafka +spec: + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: [ "ReadWriteOnce" ] + storageClassName: kafka-broker + resources: + requests: + storage: 10Gi +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: pzoo + namespace: kafka +spec: + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: [ "ReadWriteOnce" ] + storageClassName: kafka-zookeeper + resources: + requests: + storage: 1Gi diff --git a/configure/aws-storageclass-broker-gp2.yml b/variants/aws-gp2/aws-storageclass-broker-gp2.yaml similarity index 100% rename from configure/aws-storageclass-broker-gp2.yml rename to variants/aws-gp2/aws-storageclass-broker-gp2.yaml diff --git a/configure/aws-storageclass-zookeeper-gp2.yml b/variants/aws-gp2/aws-storageclass-zookeeper-gp2.yaml similarity index 100% rename from configure/aws-storageclass-zookeeper-gp2.yml rename to variants/aws-gp2/aws-storageclass-zookeeper-gp2.yaml diff --git a/variants/aws-gp2/kustomization.yaml b/variants/aws-gp2/kustomization.yaml new file mode 100644 index 00000000..ea9f8aac --- /dev/null +++ b/variants/aws-gp2/kustomization.yaml @@ -0,0 +1,7 @@ +bases: +- ../scale-3-5 +resources: +- aws-storageclass-broker-gp2.yaml +- aws-storageclass-zookeeper-gp2.yaml +patchesStrategicMerge: +- volume-claims.yaml diff --git a/variants/aws-gp2/volume-claims.yaml b/variants/aws-gp2/volume-claims.yaml new file mode 100644 index 00000000..fb20db89 --- /dev/null +++ b/variants/aws-gp2/volume-claims.yaml @@ -0,0 +1,32 @@ +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: kafka + namespace: kafka +spec: + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: [ "ReadWriteOnce" ] + storageClassName: kafka-broker + resources: + requests: + storage: 10Gi +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: pzoo + namespace: kafka +spec: + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: [ "ReadWriteOnce" ] + storageClassName: kafka-zookeeper + resources: + requests: + storage: 1Gi diff --git a/variants/dev-small/auto-create-topics.json b/variants/dev-small/auto-create-topics.json new file mode 100644 index 00000000..e607f0d8 --- /dev/null +++ b/variants/dev-small/auto-create-topics.json @@ -0,0 +1,8 @@ +[ + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"}, + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "advertised.listeners=PLAINTEXT://:9092,OUTSIDE://localhost:9094"}, + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"}, + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "auto.create.topics.enable=true"}, + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"}, + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "num.partitions=1"} +] diff --git a/variants/dev-small/jmx-disable.yaml b/variants/dev-small/jmx-disable.yaml new file mode 100644 index 00000000..fb7dd092 --- /dev/null +++ b/variants/dev-small/jmx-disable.yaml @@ -0,0 +1,13 @@ +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: kafka + namespace: kafka +spec: + template: + spec: + containers: + - name: broker + env: + - name: JMX_PORT + value: "" diff --git a/variants/dev-small/kustomization.yaml b/variants/dev-small/kustomization.yaml new file mode 100644 index 00000000..074bd1cf --- /dev/null +++ b/variants/dev-small/kustomization.yaml @@ -0,0 +1,19 @@ +# With kubectl -n kafka port-forward kafka-0 9094 +bases: +- ../scale-1-ephemeral +patchesStrategicMerge: +- jmx-disable.yaml +patchesJson6902: +# Currently these duplicate each others' directives to avoid an apply -k error: found conflict between different patches +- target: + group: apps + version: v1 + kind: StatefulSet + name: kafka + path: listener-localhost.json +- target: + group: apps + version: v1 + kind: StatefulSet + name: kafka + path: auto-create-topics.json diff --git a/variants/dev-small/listener-localhost.json b/variants/dev-small/listener-localhost.json new file mode 100644 index 00000000..e607f0d8 --- /dev/null +++ b/variants/dev-small/listener-localhost.json @@ -0,0 +1,8 @@ +[ + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"}, + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "advertised.listeners=PLAINTEXT://:9092,OUTSIDE://localhost:9094"}, + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"}, + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "auto.create.topics.enable=true"}, + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--override"}, + {"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "num.partitions=1"} +] diff --git a/configure/gke-storageclass-broker-pd.yml b/variants/gke-regional/gke-storageclass-broker-pd.yaml similarity index 100% rename from configure/gke-storageclass-broker-pd.yml rename to variants/gke-regional/gke-storageclass-broker-pd.yaml diff --git a/configure/gke-regional-euwest1cd-zookeeper-ssd.yml b/variants/gke-regional/gke-storageclass-zookeeper-regional-euwest1cd-ssh.yaml similarity index 100% rename from configure/gke-regional-euwest1cd-zookeeper-ssd.yml rename to variants/gke-regional/gke-storageclass-zookeeper-regional-euwest1cd-ssh.yaml diff --git a/configure/gke-storageclass-zookeeper-ssd.yml b/variants/gke-regional/gke-storageclass-zookeeper-ssd.yaml similarity index 100% rename from configure/gke-storageclass-zookeeper-ssd.yml rename to variants/gke-regional/gke-storageclass-zookeeper-ssd.yaml diff --git a/variants/gke-regional/kustomization.yaml b/variants/gke-regional/kustomization.yaml new file mode 100644 index 00000000..44acb293 --- /dev/null +++ b/variants/gke-regional/kustomization.yaml @@ -0,0 +1,8 @@ +bases: +- ../scale-3-5 +resources: +- gke-storageclass-broker-pd.yaml +- gke-storageclass-zookeeper-ssd.yaml +- gke-storageclass-zookeeper-regional-euwest1cd-ssh.yaml +patchesStrategicMerge: +- volume-claims.yaml diff --git a/variants/gke-regional/volume-claims.yaml b/variants/gke-regional/volume-claims.yaml new file mode 100644 index 00000000..42357194 --- /dev/null +++ b/variants/gke-regional/volume-claims.yaml @@ -0,0 +1,48 @@ +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: kafka + namespace: kafka +spec: + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: [ "ReadWriteOnce" ] + storageClassName: kafka-broker + resources: + requests: + storage: 10Gi +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: pzoo + namespace: kafka +spec: + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: [ "ReadWriteOnce" ] + storageClassName: kafka-zookeeper + resources: + requests: + storage: 1Gi +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: zoo + namespace: kafka +spec: + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: [ "ReadWriteOnce" ] + storageClassName: kafka-zookeeper-regional + resources: + requests: + storage: 1Gi diff --git a/variants/scale-1-ephemeral/kafka.yaml b/variants/scale-1-ephemeral/kafka.yaml index 4a6ac3e5..436fa722 100644 --- a/variants/scale-1-ephemeral/kafka.yaml +++ b/variants/scale-1-ephemeral/kafka.yaml @@ -4,7 +4,6 @@ metadata: name: kafka namespace: kafka spec: - replicas: 1 template: spec: volumes: diff --git a/variants/scale-1-ephemeral/kustomization.yaml b/variants/scale-1-ephemeral/kustomization.yaml index 9ecd5b7c..0cef8d2c 100644 --- a/variants/scale-1-ephemeral/kustomization.yaml +++ b/variants/scale-1-ephemeral/kustomization.yaml @@ -1,13 +1,5 @@ bases: -- ../../kafka -- ../../zookeeper +- ../scale-1 patchesStrategicMerge: - kafka.yaml - zookeeper.yaml -patchesJson6902: -- target: - group: apps - version: v1 - kind: StatefulSet - name: kafka - path: kafka-scale1-overrides.json diff --git a/variants/scale-1-ephemeral/zookeeper.yaml b/variants/scale-1-ephemeral/zookeeper.yaml index e78334f5..c2810703 100644 --- a/variants/scale-1-ephemeral/zookeeper.yaml +++ b/variants/scale-1-ephemeral/zookeeper.yaml @@ -11,16 +11,8 @@ metadata: name: pzoo namespace: kafka spec: - replicas: 1 template: spec: - initContainers: - - name: init-config - env: - - name: PZOO_REPLICAS - value: '1' - - name: ZOO_REPLICAS - value: '0' volumes: - name: data emptyDir: {} @@ -32,16 +24,9 @@ metadata: name: zoo namespace: kafka spec: - replicas: 0 template: spec: - initContainers: - - name: init-config - env: - # There's no validation on these numbers adding up to a coherent zk config, so watch out - - name: PZOO_REPLICAS - value: '1' - - name: ZOO_REPLICAS - value: '0' - - name: ID_OFFSET - value: '2' + volumes: + - name: data + emptyDir: {} + volumeClaimTemplates: [] diff --git a/variants/scale-1/kustomization.yaml b/variants/scale-1/kustomization.yaml index 9ecd5b7c..a9d8d3bb 100644 --- a/variants/scale-1/kustomization.yaml +++ b/variants/scale-1/kustomization.yaml @@ -1,4 +1,5 @@ bases: +- ../../rbac-namespace-default - ../../kafka - ../../zookeeper patchesStrategicMerge: diff --git a/variants/scale-2/kustomization.yaml b/variants/scale-2/kustomization.yaml index 2cc065d5..2251fe88 100644 --- a/variants/scale-2/kustomization.yaml +++ b/variants/scale-2/kustomization.yaml @@ -1,4 +1,5 @@ bases: +- ../../rbac-namespace-default - ../../kafka - ../../zookeeper patchesStrategicMerge: diff --git a/variants/as-is/kustomization.yaml b/variants/scale-3-5/kustomization.yaml similarity index 55% rename from variants/as-is/kustomization.yaml rename to variants/scale-3-5/kustomization.yaml index f54376f4..4936324d 100644 --- a/variants/as-is/kustomization.yaml +++ b/variants/scale-3-5/kustomization.yaml @@ -1,3 +1,4 @@ bases: +- ../../rbac-namespace-default - ../../kafka - ../../zookeeper diff --git a/zookeeper/50pzoo.yml b/zookeeper/50pzoo.yml index 18f68d7b..40645cc3 100644 --- a/zookeeper/50pzoo.yml +++ b/zookeeper/50pzoo.yml @@ -80,7 +80,7 @@ spec: name: data spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: kafka-zookeeper + storageClassName: standard resources: requests: storage: 1Gi diff --git a/zookeeper/51zoo.yml b/zookeeper/51zoo.yml index 80c80001..35cef3c5 100644 --- a/zookeeper/51zoo.yml +++ b/zookeeper/51zoo.yml @@ -83,7 +83,7 @@ spec: name: data spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: kafka-zookeeper-regional + storageClassName: standard resources: requests: storage: 1Gi