Merge remote-tracking branch 'container-up/main' into feat/support-databend

Author: hantmac

.github/workflows/ci-windows-trigger.yml

@@ -0,0 +1,27 @@
+name: windows-test command dispatch
+
+on:
+  issue_comment:
+    types: [created]
+
+permissions:
+  contents: read
+
+jobs:
+  windows-test-command-trigger:
+    permissions:
+      pull-requests: write  # for peter-evans/slash-command-dispatch to create PR reaction
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Trigger windows-test command
+        uses: peter-evans/slash-command-dispatch@13bc09769d122a64f75aa5037256f6f2d78be8c4 # v4.0.0
+        with:
+          token: ${{ secrets.WINDOWS_WORKERS_TOKEN }}
+          # The command to trigger the pipeline: e.g. /windows-test
+          # The command name must match the name of the repository_dispatch.type in 'ci-windows.yml' workflow, using '-command' as suffix. E.g. 'windows-test-command'
+          commands: windows-test
+          issue-type: pull-request
+          # The user that owns the above token must belong to the elevated role of 'Maintainers'
+          permission: maintain
+          reactions: false

.github/workflows/ci-windows.yml

@@ -32,6 +32,8 @@ on:
       - 'README.md'
       - 'RELEASING.md'
       - '.sdkmanrc'
+  repository_dispatch:
+    types: [windows-test-command]
 
 concurrency:
   group: "${{ github.workflow }}-${{ github.head_ref || github.sha }}"
@@ -42,7 +44,7 @@ permissions:
 
 jobs:
   core:
-    if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
+    if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event.client_payload.slash_command.command == 'windows-test' }}
     runs-on: self-hosted
     permissions:
       checks: write

docs/modules/kafka.md

@@ -4,18 +4,25 @@ Testcontainers can be used to automatically instantiate and manage [Apache Kafka
 
 Currently, two different Kafka images are supported:
 
-* `org.testcontainers.containers.KafkaContainer` supports 
+* `org.testcontainers.kafka.ConfluentKafkaContainer` supports 
 [confluentinc/cp-kafka](https://hub.docker.com/r/confluentinc/cp-kafka/)
 * `org.testcontainers.kafka.KafkaContainer` supports [apache/kafka](https://hub.docker.com/r/apache/kafka/) and [apache/kafka-native](https://hub.docker.com/r/apache/kafka-native/)
 
+!!! note
+    `org.testcontainers.containers.KafkaContainer` is deprecated.
+    Please use `org.testcontainers.kafka.ConfluentKafkaContainer` or `org.testcontainers.kafka.KafkaContainer` instead, depending on the used image.
+
 ## Benefits
 
 * Running a single node Kafka installation with just one line of code
 * No need to manage external Zookeeper installation, required by Kafka. But see [below](#zookeeper)
 
 ## Example
 
+### Using org.testcontainers.containers.KafkaContainer
+
 Create a `KafkaContainer` to use it in your tests:
+
 <!--codeinclude-->
 [Creating a KafkaContainer](../../modules/kafka/src/test/java/org/testcontainers/containers/KafkaContainerTest.java) inside_block:constructorWithVersion
 <!--/codeinclude-->
@@ -28,33 +35,54 @@ Now your tests or any other process running on your machine can get access to ru
 [Bootstrap Servers](../../modules/kafka/src/test/java/org/testcontainers/containers/KafkaContainerTest.java) inside_block:getBootstrapServers
 <!--/codeinclude-->
 
-## Options
+### Using org.testcontainers.kafka.ConfluentKafkaContainer
+
+!!! note
+    Compatible with `confluentinc/cp-kafka` images version `7.4.0` and later.
+
+Create a `ConfluentKafkaContainer` to use it in your tests:
+
+<!--codeinclude-->
+[Creating a ConlfuentKafkaContainer](../../modules/kafka/src/test/java/org/testcontainers/kafka/ConfluentKafkaContainerTest.java) inside_block:constructorWithVersion
+<!--/codeinclude-->
 
-!!! note 
-    The options below are only available for `org.testcontainers.containers.KafkaContainer`
+### Using org.testcontainers.kafka.KafkaContainer
+
+Create a `KafkaContainer` to use it in your tests:
+
+<!--codeinclude-->
+[Creating a KafkaContainer](../../modules/kafka/src/test/java/org/testcontainers/kafka/KafkaContainerTest.java) inside_block:constructorWithVersion
+<!--/codeinclude-->
+
+## Options
         
 ### <a name="zookeeper"></a> Using external Zookeeper
 
+!!! note
+    Only available for `org.testcontainers.containers.KafkaContainer`
+
 If for some reason you want to use an externally running Zookeeper, then just pass its location during construction:
 <!--codeinclude-->
 [External Zookeeper](../../modules/kafka/src/test/java/org/testcontainers/containers/KafkaContainerTest.java) inside_block:withExternalZookeeper
 <!--/codeinclude-->
 
 ### Using Kraft mode
 
-KRaft mode was declared production ready in 3.3.1 (confluentinc/cp-kafka:7.3.x)" 
+!!! note
+    Only available for `org.testcontainers.containers.KafkaContainer`
+
+KRaft mode was declared production ready in 3.3.1 (confluentinc/cp-kafka:7.3.x) 
 
 <!--codeinclude-->
 [Kraft mode](../../modules/kafka/src/test/java/org/testcontainers/containers/KafkaContainerTest.java) inside_block:withKraftMode
 <!--/codeinclude-->
 
-See the [versions interoperability matrix](https://docs.confluent.io/platform/current/installation/versions-interoperability.html) for more details. 
+See the [versions interoperability matrix](https://docs.confluent.io/platform/current/installation/versions-interoperability.html) for more details.
 
-## Register listeners
+### Register listeners
 
 There are scenarios where additional listeners are needed because the consumer/producer can be in another
-container in the same network or a different process where the port to connect differs from the default 
-exposed port `9093`. E.g [Toxiproxy](../../modules/toxiproxy/).
+container in the same network or a different process where the port to connect differs from the default exposed port. E.g [Toxiproxy](../../modules/toxiproxy/).
 
 <!--codeinclude-->
 [Register additional listener](../../modules/kafka/src/test/java/org/testcontainers/containers/KafkaContainerTest.java) inside_block:registerListener

docs/supported_docker_environment/continuous_integration/gitlab_ci.md

@@ -24,10 +24,10 @@ See below for an example runner configuration:
 Please also include the following in your GitlabCI pipeline definitions (`.gitlab-ci.yml`) that use Testcontainers:
 ```yml
 variables:
-  TESTCONTAINERS_HOST_OVERRIDE: "host.docker.internal"
+  TESTCONTAINERS_HOST_OVERRIDE: "<ip-docker-host>"
 ```
 
-The environment variable `TESTCONTAINERS_HOST_OVERRIDE` needs to be configured, otherwise, a wrong IP address would be used to resolve the Docker host, which will likely lead to failing tests.
+The environment variable `TESTCONTAINERS_HOST_OVERRIDE` needs to be configured, otherwise, a wrong IP address would be used to resolve the Docker host, which will likely lead to failing tests. For Windows and MacOS, use `host.docker.internal`.
 
 ## Example using DinD (Docker-in-Docker)
 

examples/build.gradle

@@ -14,6 +14,16 @@ subprojects {
         mavenCentral()
     }
 
+    test {
+        defaultCharacterEncoding = "UTF-8"
+        testLogging {
+            displayGranularity 1
+            showStackTraces = true
+            exceptionFormat = 'full'
+            events "STARTED", "PASSED", "FAILED", "SKIPPED"
+        }
+    }
+
     checkstyle {
         toolVersion = "9.3"
         configFile = rootProject.file('../config/checkstyle/checkstyle.xml')

modules/kafka/src/main/java/org/testcontainers/containers/KafkaContainer.java

@@ -26,7 +26,11 @@
  *     <li>Kafka: 9093</li>
  *     <li>Zookeeper: 2181</li>
  * </ul>
+ *
+ * @deprecated use {@link org.testcontainers.kafka.ConfluentKafkaContainer} or
+ * {@link org.testcontainers.kafka.KafkaContainer} instead
  */
+@Deprecated
 public class KafkaContainer extends GenericContainer<KafkaContainer> {
 
     private static final DockerImageName DEFAULT_IMAGE_NAME = DockerImageName.parse("confluentinc/cp-kafka");
@@ -327,13 +331,7 @@ void withRaft() {
             addEnvVar("KAFKA_LISTENERS", kafkaListeners());
             addEnvVar("KAFKA_PROCESS_ROLES", "broker,controller");
 
-            String firstNetworkAlias = getNetworkAliases().stream().findFirst().orElse(null);
-            String networkAlias = getNetwork() != null ? firstNetworkAlias : "localhost";
-            String controllerQuorumVoters = String.format(
-                "%s@%s:9094",
-                getEnvVars().get("KAFKA_NODE_ID"),
-                networkAlias
-            );
+            String controllerQuorumVoters = String.format("%s@localhost:9094", getEnvVars().get("KAFKA_NODE_ID"));
             this.envVars.computeIfAbsent("KAFKA_CONTROLLER_QUORUM_VOTERS", key -> controllerQuorumVoters);
             addEnvVar("KAFKA_CONTROLLER_LISTENER_NAMES", "CONTROLLER");
 

modules/kafka/src/main/java/org/testcontainers/kafka/ConfluentKafkaContainer.java

@@ -0,0 +1,140 @@
+package org.testcontainers.kafka;
+
+import com.github.dockerjava.api.command.InspectContainerResponse;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.images.builder.Transferable;
+import org.testcontainers.utility.DockerImageName;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Supplier;
+
+/**
+ * Testcontainers implementation for Confluent Kafka.
+ * <p>
+ * Supported image: {@code confluentinc/cp-kafka}
+ * <p>
+ * Exposed ports: 9092
+ */
+public class ConfluentKafkaContainer extends GenericContainer<ConfluentKafkaContainer> {
+
+    private static final DockerImageName DEFAULT_IMAGE_NAME = DockerImageName.parse("confluentinc/cp-kafka");
+
+    private final Set<String> listeners = new HashSet<>();
+
+    private final Set<Supplier<String>> advertisedListeners = new HashSet<>();
+
+    public ConfluentKafkaContainer(String imageName) {
+        this(DockerImageName.parse(imageName));
+    }
+
+    public ConfluentKafkaContainer(DockerImageName dockerImageName) {
+        super(dockerImageName);
+        dockerImageName.assertCompatibleWith(DEFAULT_IMAGE_NAME);
+
+        withExposedPorts(KafkaHelper.KAFKA_PORT);
+        withEnv(KafkaHelper.envVars());
+
+        withCommand(KafkaHelper.COMMAND);
+        waitingFor(KafkaHelper.WAIT_STRATEGY);
+    }
+
+    @Override
+    protected void configure() {
+        KafkaHelper.resolveListeners(this, this.listeners);
+
+        String controllerQuorumVoters = String.format("%s@localhost:9094", getEnvMap().get("KAFKA_NODE_ID"));
+        withEnv("KAFKA_CONTROLLER_QUORUM_VOTERS", controllerQuorumVoters);
+    }
+
+    @Override
+    protected void containerIsStarting(InspectContainerResponse containerInfo) {
+        String brokerAdvertisedListener = String.format(
+            "BROKER://%s:%s",
+            containerInfo.getConfig().getHostName(),
+            "9093"
+        );
+        List<String> advertisedListeners = new ArrayList<>();
+        advertisedListeners.add("PLAINTEXT://" + getBootstrapServers());
+        advertisedListeners.add(brokerAdvertisedListener);
+
+        advertisedListeners.addAll(KafkaHelper.resolveAdvertisedListeners(this.advertisedListeners));
+        String kafkaAdvertisedListeners = String.join(",", advertisedListeners);
+
+        String command = "#!/bin/bash\n";
+        // exporting KAFKA_ADVERTISED_LISTENERS with the container hostname
+        command += String.format("export KAFKA_ADVERTISED_LISTENERS=%s\n", kafkaAdvertisedListeners);
+
+        command += "/etc/confluent/docker/run \n";
+        copyFileToContainer(Transferable.of(command, 0777), KafkaHelper.STARTER_SCRIPT);
+    }
+
+    /**
+     * Add a listener in the format {@code host:port}.
+     * Host will be included as a network alias.
+     * <p>
+     * Use it to register additional connections to the Kafka broker within the same container network.
+     * <p>
+     * The listener will be added to the list of default listeners.
+     * <p>
+     * Default listeners:
+     * <ul>
+     *     <li>0.0.0.0:9092</li>
+     *     <li>0.0.0.0:9093</li>
+     *     <li>0.0.0.0:9094</li>
+     * </ul>
+     * <p>
+     * The listener will be added to the list of default advertised listeners.
+     * <p>
+     * Default advertised listeners:
+     * <ul>
+     *      <li>{@code container.getConfig().getHostName():9092}</li>
+     *      <li>{@code container.getHost():container.getMappedPort(9093)}</li>
+     * </ul>
+     * @param listener a listener with format {@code host:port}
+     * @return this {@link ConfluentKafkaContainer} instance
+     */
+    public ConfluentKafkaContainer withListener(String listener) {
+        this.listeners.add(listener);
+        this.advertisedListeners.add(() -> listener);
+        return this;
+    }
+
+    /**
+     * Add a listener in the format {@code host:port} and a {@link Supplier} for the advertised listener.
+     * Host from listener will be included as a network alias.
+     * <p>
+     * Use it to register additional connections to the Kafka broker from outside the container network
+     * <p>
+     * The listener will be added to the list of default listeners.
+     * <p>
+     * Default listeners:
+     * <ul>
+     *     <li>0.0.0.0:9092</li>
+     *     <li>0.0.0.0:9093</li>
+     *     <li>0.0.0.0:9094</li>
+     * </ul>
+     * <p>
+     * The {@link Supplier} will be added to the list of default advertised listeners.
+     * <p>
+     * Default advertised listeners:
+     * <ul>
+     *      <li>{@code container.getConfig().getHostName():9092}</li>
+     *      <li>{@code container.getHost():container.getMappedPort(9093)}</li>
+     * </ul>
+     * @param listener a supplier that will provide a listener
+     * @param advertisedListener a supplier that will provide a listener
+     * @return this {@link ConfluentKafkaContainer} instance
+     */
+    public ConfluentKafkaContainer withListener(String listener, Supplier<String> advertisedListener) {
+        this.listeners.add(listener);
+        this.advertisedListeners.add(advertisedListener);
+        return this;
+    }
+
+    public String getBootstrapServers() {
+        return String.format("%s:%s", getHost(), getMappedPort(KafkaHelper.KAFKA_PORT));
+    }
+}