Support kafka without ZooKeeper

[hennr]

Hi @wurstmeister,

I just read this article and it seems that kafka 2.8.0 will be able to run without ZooKeeper:
https://www.confluent.io/blog/kafka-without-zookeeper-a-sneak-peek/?mkt_tok=NTgyLVFIWC0yNjIAAAF8ja2eKb-T_8HLC-RLQjnhN7whMmqTaB80OL6Z-QhVXk2aqZ-Pw2fITKR05NpEGy_yx87vKtO6jF2rcunGEGXCNXMfiAwzfUnj7WmK9HSs11FBoQ

I just wanted to open this issue to see if you plan to support this KRaft mode (they called it that way) and to be able to track progress on this.

[MollardMichael]

I actually set it up inside on docker to try it out (Nothing that's prod ready)

I add something really similar to this repo Dockerfile (I copied it).

And my start-kafka.sh looked like that

#!/bin/bash -e

rm -rf /tmp/kraft-combined-logs

CLUSTER_ID=$(kafka-storage.sh random-uuid)
kafka-storage.sh format -t ${CLUSTER_ID} -c ${KAFKA_HOME}/config/kraft/server.properties
kafka-server-start.sh ${KAFKA_HOME}/config/kraft/server.properties

As you can see this can't be used for any kind of long term use but you get a gist of what's need to set it up.

I would be interested to implement it in a better way If you can give me some guide on how you want it to live alongside the other versions

[sycured]

@MollardMichael Thanks for your snippet. It's something that needs to be merged inside PR #655 because this PR exclusively updates to 2.8.0 without including the possibility to start without a zookeeper.

[maguowei]

try https://github.com/maguowei/dockerfiles/tree/master/kafka for dev

[sycured]

Thiank you @maguowei

This is my actual patch to have the switch to start with or without zookeeper (-e KAFKA_WITHOUT_ZOOKEEPER=true):

diff --git a/start-kafka.sh b/start-kafka.sh
index 56927db..5c037c3 100755
--- a/start-kafka.sh
+++ b/start-kafka.sh
@@ -10,7 +10,7 @@ fi
 # Store original IFS config, so we can restore it at various stages
 ORIG_IFS=$IFS
 
-if [[ -z "$KAFKA_ZOOKEEPER_CONNECT" ]]; then
+if [[ -z "$KAFKA_WITHOUT_ZOOKEEPER" && -z "$KAFKA_ZOOKEEPER_CONNECT" ]]; then
     echo "ERROR: missing mandatory config: KAFKA_ZOOKEEPER_CONNECT"
     exit 1
 fi
@@ -118,7 +118,7 @@ echo "" >> "$KAFKA_HOME/config/server.properties"
 
     # Fixes #312
     # KAFKA_VERSION + KAFKA_HOME + grep -rohe KAFKA[A-Z0-0_]* /opt/kafka/bin | sort | uniq | tr '\n' '|'
-    EXCLUSIONS="|KAFKA_VERSION|KAFKA_HOME|KAFKA_DEBUG|KAFKA_GC_LOG_OPTS|KAFKA_HEAP_OPTS|KAFKA_JMX_OPTS|KAFKA_JVM_PERFORMANCE_OPTS|KAFKA_LOG|KAFKA_OPTS|"
+    EXCLUSIONS="|KAFKA_VERSION|KAFKA_HOME|KAFKA_DEBUG|KAFKA_GC_LOG_OPTS|KAFKA_HEAP_OPTS|KAFKA_JMX_OPTS|KAFKA_JVM_PERFORMANCE_OPTS|KAFKA_LOG|KAFKA_OPTS|KAFKA_WITHOUT_ZOOKEEPER|"
 
     # Read in env as a new-line separated array. This handles the case of env variables have spaces and/or carriage returns. See #313
     IFS=$'\n'
@@ -133,6 +133,7 @@ echo "" >> "$KAFKA_HOME/config/server.properties"
         if [[ $env_var =~ ^KAFKA_ ]]; then
             kafka_name=$(echo "$env_var" | cut -d_ -f2- | tr '[:upper:]' '[:lower:]' | tr _ .)
             updateConfig "$kafka_name" "${!env_var}" "$KAFKA_HOME/config/server.properties"
+            updateConfig "$kafka_name" "${!env_var}" "$KAFKA_HOME/config/kraft/server.properties"
         fi
 
         if [[ $env_var =~ ^LOG4J_ ]]; then
@@ -146,4 +147,9 @@ if [[ -n "$CUSTOM_INIT_SCRIPT" ]] ; then
   eval "$CUSTOM_INIT_SCRIPT"
 fi
 
-exec "$KAFKA_HOME/bin/kafka-server-start.sh" "$KAFKA_HOME/config/server.properties"
+if [[ -z "$KAFKA_WITHOUT_ZOOKEEPER" ]]; then
+    exec "$KAFKA_HOME/bin/kafka-server-start.sh" "$KAFKA_HOME/config/server.properties"
+else
+    bash -c "$KAFKA_HOME/bin/kafka-storage.sh format -t $($KAFKA_HOME/bin/kafka-storage.sh random-uuid) -c $KAFKA_HOME/config/kraft/server.properties"
+    exec "$KAFKA_HOME/bin/kafka-server-start.sh" "$KAFKA_HOME/config/kraft/server.properties"
+fi

[teddybee]

Improved Kraft emerged in Kafka 3.0.
https://www.youtube.com/watch?v=7SDwWFYnhGA