Skip to content

[pull] main from googleapis:main #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
58e7555
feat: add kafka migration (#330)
johnjcasey Jun 18, 2024
a14a933
build(deps): update dependency com.google.cloud:google-cloud-shared-c…
renovate-bot Jun 18, 2024
33f2761
fix: report connector package version instead of kafka connect versio…
HurSungYun Jun 18, 2024
bc00e06
deps: update dependency org.slf4j:slf4j-api to v2.0.13 (#299)
renovate-bot Jun 18, 2024
290787b
test(deps): update dependency com.google.truth:truth to v1.4.2 (#309)
renovate-bot Jun 18, 2024
87c2dc1
chore: update requirements.txt to address dependabot security alerts.…
gcf-owl-bot[bot] Jun 18, 2024
18e6d8c
chore: remove owlbot postprocessor check (#350)
JoeWang1127 Sep 12, 2024
8c63561
chore: disable owlbot (#351)
JoeWang1127 Sep 12, 2024
773f67b
deps: Update kafka-clients and pubsublite-kafka and pubsublite client…
bcol-google Oct 7, 2024
25c152b
chore(main): release 1.3.0 (#325)
release-please[bot] Oct 8, 2024
4c107d8
fix: maxOutstandingMessages should be defined without maxOutstandingR…
mihai-ghita Oct 25, 2024
0907353
deps: update google-cloud-shared-config to 1.11.3 (#357)
bcol-google Nov 6, 2024
6b3d685
chore(main): release 1.3.1-SNAPSHOT (#355)
release-please[bot] Nov 6, 2024
41c6121
chore(main): release 1.3.1 (#362)
release-please[bot] Nov 6, 2024
5e6b566
build(deps): update dependency com.google.cloud:google-cloud-shared-c…
renovate-bot Apr 3, 2025
e0f7744
fix: CloudPubSubSourceTask fails to ACK messages in Kafka 4.0 due to …
Lance726 Apr 3, 2025
6f7e93d
build(deps): bump jinja2 from 3.1.3 to 3.1.6 in /.kokoro (#367)
dependabot[bot] Apr 3, 2025
abfb043
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
renovate-bot Apr 3, 2025
5d9071e
build(deps): bump zipp from 3.18.0 to 3.19.1 in /.kokoro (#340)
dependabot[bot] Apr 4, 2025
e2e2049
deps: update dependency org.slf4j:slf4j-api to v2.0.17 (#354)
renovate-bot Apr 4, 2025
d8efd0e
build(deps): bump certifi from 2024.2.2 to 2024.7.4 in /.kokoro (#339)
dependabot[bot] Apr 4, 2025
cb01ad1
build(deps): bump urllib3 from 2.2.1 to 2.2.2 in /.kokoro (#336)
dependabot[bot] Apr 4, 2025
a9cf65e
build(deps): bump idna from 3.4 to 3.7 in /.kokoro (#317)
dependabot[bot] Apr 4, 2025
346f1de
build(deps): bump urllib3 from 1.26.12 to 1.26.19 in /.kokoro (#331)
dependabot[bot] Apr 4, 2025
c5f2a5e
build(deps): bump jinja2 from 3.0.3 to 3.1.4 in /.kokoro (#320)
dependabot[bot] Apr 4, 2025
eb8c081
build(deps): update dependency com.google.cloud:google-cloud-shared-c…
renovate-bot Apr 16, 2025
1f2cfe5
fix: typo in comment (#377)
pradn Apr 22, 2025
63189e8
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
renovate-bot May 2, 2025
4749491
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
renovate-bot May 6, 2025
3847757
deps: update Kafka dependency version to v3.9.0 (#378)
matt-kwong May 30, 2025
08841f0
build(deps): bump certifi from 2024.2.2 to 2024.7.4 in /.kokoro (#382)
dependabot[bot] Jun 2, 2025
2fa9585
chore(main): release 1.3.2-SNAPSHOT (#363)
release-please[bot] Jun 2, 2025
c5e9ac3
chore(main): release 1.3.2 (#385)
release-please[bot] Jun 2, 2025
9607892
build(deps): bump requests from 2.31.0 to 2.32.4 in /.kokoro (#387)
dependabot[bot] Jun 11, 2025
c331e08
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
renovate-bot Jun 17, 2025
5d4004f
build(deps): update dependency com.google.cloud:google-cloud-shared-c…
renovate-bot Jun 17, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 0 additions & 16 deletions .github/.OwlBot.yaml

This file was deleted.

9 changes: 7 additions & 2 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,15 @@ updates:
schedule:
interval: "daily"
# Disable version updates for Maven dependencies
open-pull-requests-limit: 0
# we use renovate-bot as well as shared-dependencies BOM to update maven dependencies.
ignore:
- dependency-name: "*"
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "daily"
# Disable version updates for pip dependencies
open-pull-requests-limit: 0
# If a security vulnerability comes in, we will be notified about
# it via template in the synthtool repository.
ignore:
- dependency-name: "*"
12 changes: 0 additions & 12 deletions .github/generated-files-bot.yml

This file was deleted.

1 change: 1 addition & 0 deletions .github/release-trigger.yml
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
enabled: true
multiScmName: java-pubsub-group-kafka-connector
1 change: 0 additions & 1 deletion .github/sync-repo-settings.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ branchProtectionRules:
- units (8)
- units (11)
- cla/google
- OwlBot Post Processor
- 'Kokoro - Test: Integration'
permissionRules:
- team: yoshi-admins
Expand Down
25 changes: 25 additions & 0 deletions .github/workflows/renovate_config_check.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: Renovate Bot Config Validation

on:
pull_request:
paths:
- 'renovate.json'

jobs:
renovate_bot_config_validation:
runs-on: ubuntu-22.04

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v3
with:
node-version: '20'

- name: Install Renovate and Config Validator
run: |
npm install -g npm@latest
npm install --global renovate
renovate-config-validator
732 changes: 415 additions & 317 deletions .kokoro/requirements.txt

Large diffs are not rendered by default.

54 changes: 54 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,59 @@
# Changelog

## [1.3.2](https://github.com/googleapis/java-pubsub-group-kafka-connector/compare/v1.3.1...v1.3.2) (2025-06-02)


### Bug Fixes

* CloudPubSubSourceTask fails to ACK messages in Kafka 4.0 due to deprecated commitRecord method ([#371](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/371)) ([e0f7744](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/e0f7744869e2881f42ed28e2dc64cd1fe01de53b))
* Typo in comment ([#377](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/377)) ([1f2cfe5](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/1f2cfe5ecdab6b5405dd77755bdb9eb7761b5ff0))


### Dependencies

* Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.45.1 ([#365](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/365)) ([abfb043](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/abfb043086886f0466d65fdee243aa6a7744881a))
* Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.46.2 ([#376](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/376)) ([63189e8](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/63189e81188733cf421354b053b64212d5453e9d))
* Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.47.0 ([#379](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/379)) ([4749491](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/4749491ea90a57a8cad10c4a2f2f73e9445b57dc))
* Update dependency org.slf4j:slf4j-api to v2.0.17 ([#354](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/354)) ([e2e2049](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/e2e204961c429f3f359dbcfc38f097829a5948e6))
* Update Kafka dependency version to v3.9.0 ([#378](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/378)) ([3847757](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/3847757fa25409fc185106c90f02920e00ff371f))

## [1.3.1](https://github.com/googleapis/java-pubsub-group-kafka-connector/compare/v1.3.0...v1.3.1) (2024-11-06)


### Bug Fixes

* An optimization in guava 33.3.0 broke our tests by not calling `addListener()` on futures which are already `isDone()`. Add mock call to make our futures not `isDone()` in some tests. ([0907353](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/0907353b27144e7bd2a57439efbb9ce2da304ab3))
* MaxOutstandingMessages should be defined without maxOutstandingRequestBytes ([#349](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/349)) ([4c107d8](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/4c107d896930492a0794d0c133104d5574f92efc))


### Dependencies

* Depend on pubsublite libraries from google-cloud-bom ([0907353](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/0907353b27144e7bd2a57439efbb9ce2da304ab3))
* Update flogger to 0.8 ([0907353](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/0907353b27144e7bd2a57439efbb9ce2da304ab3))
* Update google-cloud-shared-config to 1.11.3 ([#357](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/357)) ([0907353](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/0907353b27144e7bd2a57439efbb9ce2da304ab3))
* Update google-cloud-shared-dependencies to 3.39.0 ([0907353](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/0907353b27144e7bd2a57439efbb9ce2da304ab3))
* Update pubsublite-kafka to 1.2.2 ([0907353](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/0907353b27144e7bd2a57439efbb9ce2da304ab3))
* Use google-cloud-bom 0.231.0 instead of libraries-bom ([0907353](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/0907353b27144e7bd2a57439efbb9ce2da304ab3))

## [1.3.0](https://github.com/googleapis/java-pubsub-group-kafka-connector/compare/v1.2.0...v1.3.0) (2024-10-07)


### Features

* Add kafka migration ([#330](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/330)) ([58e7555](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/58e7555831e63264d0b7fda19291629105a085ff))


### Bug Fixes

* Report connector package version instead of kafka connect version ([#266](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/266)) ([33f2761](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/33f2761b303f267b4fa6af9e2acf88a71b210f4b))


### Dependencies

* Update dependency org.apache.commons:commons-lang3 to v3.14.0 ([#302](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/302)) ([d5cad34](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/d5cad346dfd68c5c86186bbc44a5acf2174760cb))
* Update dependency org.slf4j:slf4j-api to v2.0.13 ([#299](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/299)) ([bc00e06](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/bc00e06b260de129dc6c0e5e2d36c6fa0e393ffa))
* Update kafka-clients and pubsublite-kafka and pubsublite clients ([#353](https://github.com/googleapis/java-pubsub-group-kafka-connector/issues/353)) ([773f67b](https://github.com/googleapis/java-pubsub-group-kafka-connector/commit/773f67b2bd8c20c9620b05c0c09a334ceac22567))

## [1.2.0](https://github.com/googleapis/java-pubsub-group-kafka-connector/compare/v1.1.0...v1.2.0) (2023-05-10)


Expand Down
1 change: 1 addition & 0 deletions migration/.gcp/gmk_bootstrap_servers
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
bootstrap.<google-managed-kafka-cluster-name>.<google-managed-kafka-cluster-region name>.managedkafka.<google-managed-cluster-host-project-name>.cloud.goog:9092
1 change: 1 addition & 0 deletions migration/.gcp/gmk_sasl_service_account
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<service-account-name>@<gcp-project>.iam.gserviceaccount.com
1 change: 1 addition & 0 deletions migration/.gcp/gmk_sasl_service_account_key
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<base64 encoded sasl service account key>
1 change: 1 addition & 0 deletions migration/.gcp/kafka_config_storage_topic
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<kafka topic name used by kafka connect for tracking the config>
1 change: 1 addition & 0 deletions migration/.gcp/kafka_connect_group_id
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<kafka connect group id(unique per worker group) for the kafka connect workers in distributed mode>
1 change: 1 addition & 0 deletions migration/.gcp/kafka_offset_storage_topic
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<kafka topic name used by kafka connect for tracking the offsets>
1 change: 1 addition & 0 deletions migration/.gcp/kafka_sink_topic
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<target sink kafka topic name used by kafka connect for migrating the data from pubsub-lite topic>
1 change: 1 addition & 0 deletions migration/.gcp/kafka_ssl_truststore_location
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<full path of the ssl truststore jks file location>
1 change: 1 addition & 0 deletions migration/.gcp/kafka_ssl_truststore_password
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<password for the ssl truststore jks>
1 change: 1 addition & 0 deletions migration/.gcp/kafka_status_storage_topic
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<kafka topic name used by kafka connect for tracking the status>
1 change: 1 addition & 0 deletions migration/.gcp/pubsub_lite_gcp_location
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<GCP location for the pubsub lite source subscription to be used for migrating the pubsub lite topic to sink kafka topic>
1 change: 1 addition & 0 deletions migration/.gcp/pubsub_lite_gcp_project
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<GCP project that hosts the pubsub lite source subscription to be used for migrating the pubsub lite topic to sink kafka topic>
1 change: 1 addition & 0 deletions migration/.gcp/pubsub_lite_gcp_subscription
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<pubsub lite source subscription name to be used for migrating the pubsub lite topic to kafka topic>
1 change: 1 addition & 0 deletions migration/.gcp/pubsub_lite_job_name
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
PubSubLiteSourceConnector
84 changes: 84 additions & 0 deletions migration/docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
FROM --platform=linux/amd64 eclipse-temurin:21
ARG KAFKA_VERSION="3.4.0"
ARG KAFKA_CONNECT_VERSION="${KAFKA_VERSION}"
ARG KAFKA_SCALA_VERSION="2.13"
ARG PUBSUB_GROUP_KAFKA_CONNECTOR_VERSION="1.2.0"
ARG KAFKA_HOME_ROOT="/opt"
ARG KAFKA_CONFIG_DIR="${KAFKA_HOME}/config"
ARG KAFKA_RELEASE="kafka_${KAFKA_SCALA_VERSION}-${KAFKA_VERSION}"
ARG KAFKA_TARBALL="${KAFKA_RELEASE}.tgz"
ARG KAFKA_DOWNLOAD_URL="https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/${KAFKA_TARBALL}"
ENV KAFKA_HEAP_OPTS="-Xms2G -Xmx2G"
ENV KAFKA_HOME="${KAFKA_HOME_ROOT}/kafka"
ARG KAFKA_PLUGINS_DIR="${KAFKA_HOME}/plugins"
# The pubsub-group-kafka-connector file needs to be pre-built/downloaded using maven or other similar tool.
# References:
# 1) https://github.com/googleapis/java-pubsub-group-kafka-connector/releases/
# 2) https://central.sonatype.com/artifact/com.google.cloud/pubsub-group-kafka-connector
ARG PUBSUB_GROUP_KAFKA_CONNECTOR_JAR="pubsub-group-kafka-connector-${PUBSUB_GROUP_KAFKA_CONNECTOR_VERSION}.jar"
ARG KAFKA_CONNECT_CONFIGURE_SCRIPT="configure-kafka-connect.sh"
ARG BUILD_KAFKA_CONNECT_STARTUP_SCRIPT="start-kafka-connect.sh"
ARG BUILD_PUBSUB_LITE_JOB_STARTUP_SCRIPT="start-pubsub-lite-connector.sh"
ARG BUILD_KAFKA_CONNECT_CONFIG_FILE="kafka-connect.properties"
ARG BUILD_PUBSUB_LITE_JOB_FILE="pubsub_lite_job.json"
ENV JAVA_HOME="/opt/java/openjdk"
ENV PATH="${KAFKA_HOME}/bin:${JAVA_HOME}/bin:${PATH}"
ENV KAFKA_CONNECT_STARTUP_SCRIPT="${KAFKA_HOME}/bin/${BUILD_KAFKA_CONNECT_STARTUP_SCRIPT}"
ENV PUBSUB_LITE_JOB_STARTUP_SCRIPT="${KAFKA_HOME}/bin/${BUILD_PUBSUB_LITE_JOB_STARTUP_SCRIPT}"
ENV KAFKA_CONNECT_CONFIG_FILE="${KAFKA_CONFIG_DIR}/${BUILD_KAFKA_CONNECT_CONFIG_FILE}"
ENV PUBSUB_LITE_JOB_FILE="${KAFKA_CONFIG_DIR}/${BUILD_PUBSUB_LITE_JOB_FILE}"

RUN apt-get -y -qq update \
&& apt-get -y -qq install iproute2 bind9-dnsutils

RUN wget -q ${KAFKA_DOWNLOAD_URL} \
&& tar -xzf ${KAFKA_TARBALL} -C ${KAFKA_HOME_ROOT} \
&& ln -s ${KAFKA_HOME_ROOT}/${KAFKA_RELEASE} ${KAFKA_HOME} \
&& rm -f ${KAFKA_TARBALL}

RUN mkdir -p ${KAFKA_PLUGINS_DIR}
COPY ${PUBSUB_GROUP_KAFKA_CONNECTOR_JAR} \
${KAFKA_PLUGINS_DIR}/${PUBSUB_GROUP_KAFKA_CONNECTOR_JAR}
COPY ${BUILD_KAFKA_CONNECT_CONFIG_FILE} ${KAFKA_CONNECT_CONFIG_FILE}
COPY ${BUILD_PUBSUB_LITE_JOB_FILE} ${PUBSUB_LITE_JOB_FILE}
COPY ${KAFKA_CONNECT_CONFIGURE_SCRIPT} .
COPY ${BUILD_KAFKA_CONNECT_STARTUP_SCRIPT} ${KAFKA_CONNECT_STARTUP_SCRIPT}
COPY ${BUILD_PUBSUB_LITE_JOB_STARTUP_SCRIPT} ${PUBSUB_LITE_JOB_STARTUP_SCRIPT}
RUN chmod +x ${KAFKA_CONNECT_CONFIGURE_SCRIPT}
RUN chmod +x ${KAFKA_CONNECT_STARTUP_SCRIPT}
RUN chmod +x ${PUBSUB_LITE_JOB_STARTUP_SCRIPT}
RUN --mount=type=secret,id=gmk_bootstrap_servers \
--mount=type=secret,id=gmk_sasl_service_account \
--mount=type=secret,id=gmk_sasl_service_account_key \
--mount=type=secret,id=kafka_sink_topic \
--mount=type=secret,id=kafka_connect_group_id \
--mount=type=secret,id=pubsub_lite_gcp_project \
--mount=type=secret,id=pubsub_lite_gcp_location \
--mount=type=secret,id=pubsub_lite_gcp_subscription \
--mount=type=secret,id=pubsub_lite_job_name \
--mount=type=secret,id=kafka_config_storage_topic \
--mount=type=secret,id=kafka_offset_storage_topic \
--mount=type=secret,id=kafka_status_storage_topic \
--mount=type=secret,id=kafka_ssl_truststore_location \
--mount=type=secret,id=kafka_ssl_truststore_password \
KAFKA_CONNECT_CONFIG_FILE="${KAFKA_CONNECT_CONFIG_FILE}" \
KAFKA_BOOTSTRAP_SERVERS="$(cat /run/secrets/gmk_bootstrap_servers)" \
KAFKA_SASL_SERVICE_ACCOUNT="$(cat /run/secrets/gmk_sasl_service_account)"\
KAFKA_SASL_SERVICE_ACCOUNT_KEY="$(cat /run/secrets/gmk_sasl_service_account_key)" \
KAFKA_SINK_TOPIC="$(cat /run/secrets/kafka_sink_topic)" \
KAFKA_CONNECT_GROUP_ID="$(cat /run/secrets/kafka_connect_group_id)" \
KAFKA_PLUGINS_DIR=${KAFKA_PLUGINS_DIR} \
PUBSUB_LITE_GCP_PROJECT="$(cat /run/secrets/pubsub_lite_gcp_project)" \
PUBSUB_LITE_GCP_LOCATION="$(cat /run/secrets/pubsub_lite_gcp_location)" \
PUBSUB_LITE_GCP_SUBSCRIPTION="$(cat /run/secrets/pubsub_lite_gcp_subscription)" \
PUBSUB_LITE_JOB_NAME="$(cat /run/secrets/pubsub_lite_job_name)" \
KAFKA_CONFIG_STORAGE_TOPIC="$(cat /run/secrets/kafka_config_storage_topic)" \
KAFKA_OFFSET_STORAGE_TOPIC="$(cat /run/secrets/kafka_offset_storage_topic)" \
KAFKA_STATUS_STORAGE_TOPIC="$(cat /run/secrets/kafka_status_storage_topic)" \
KAFKA_SSL_TRUSTSTORE_LOCATION="$(cat /run/secrets/kafka_ssl_truststore_location)" \
KAFKA_SSL_TRUSTSTORE_PASSWORD="$(cat /run/secrets/kafka_ssl_truststore_password)" \
./${KAFKA_CONNECT_CONFIGURE_SCRIPT} \
&& rm -f ./${KAFKA_CONNECT_CONFIGURE_SCRIPT}

EXPOSE 8083
CMD ${KAFKA_CONNECT_STARTUP_SCRIPT}
18 changes: 18 additions & 0 deletions migration/docker/build-image.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
SELF_DIR="$(dirname $(readlink -f $0))"
SECRETS_DIR="$(dirname ${SELF_DIR})/.gcp"
docker build --platform=linux/amd64 --file Dockerfile --tag psl-to-gmk:latest \
--secret id=gmk_sasl_service_account,src="${SECRETS_DIR}/gmk_sasl_service_account" \
--secret id=gmk_sasl_service_account_key,src="${SECRETS_DIR}/gmk_sasl_service_account_key" \
--secret id=gmk_bootstrap_servers,src="${SECRETS_DIR}/gmk_bootstrap_servers" \
--secret id=kafka_sink_topic,src="${SECRETS_DIR}/kafka_sink_topic" \
--secret id=kafka_connect_group_id,src="${SECRETS_DIR}/kafka_connect_group_id" \
--secret id=pubsub_lite_gcp_project,src="${SECRETS_DIR}/pubsub_lite_gcp_project" \
--secret id=pubsub_lite_gcp_location,src="${SECRETS_DIR}/pubsub_lite_gcp_location" \
--secret id=pubsub_lite_gcp_subscription,src="${SECRETS_DIR}/pubsub_lite_gcp_subscription" \
--secret id=pubsub_lite_job_name,src="${SECRETS_DIR}/pubsub_lite_job_name" \
--secret id=kafka_config_storage_topic,src="${SECRETS_DIR}/kafka_config_storage_topic" \
--secret id=kafka_offset_storage_topic,src="${SECRETS_DIR}/kafka_offset_storage_topic" \
--secret id=kafka_status_storage_topic,src="${SECRETS_DIR}/kafka_status_storage_topic" \
--secret id=kafka_ssl_truststore_location,src="${SECRETS_DIR}/kafka_ssl_truststore_location" \
--secret id=kafka_ssl_truststore_password,src="${SECRETS_DIR}/kafka_ssl_truststore_password" \
--no-cache .
25 changes: 25 additions & 0 deletions migration/docker/configure-kafka-connect.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
# All the variables must be supplied as environment variables for this script
# Update Kafka Connect Sink config
sed -i -e "s#__KAFKA_BOOTSTRAP_SERVERS__#${KAFKA_BOOTSTRAP_SERVERS}#g;" \
"${KAFKA_CONNECT_CONFIG_FILE}"
# Update Kafka Connect internal topics config
sed -i -e "s#__KAFKA_CONFIG_STORAGE_TOPIC__#${KAFKA_CONFIG_STORAGE_TOPIC}#g; s#__KAFKA_OFFSET_STORAGE_TOPIC__#${KAFKA_OFFSET_STORAGE_TOPIC}#g; s#__KAFKA_STATUS_STORAGE_TOPIC__#${KAFKA_STATUS_STORAGE_TOPIC}#g" \
"${KAFKA_CONNECT_CONFIG_FILE}"
# Update Kafka Connect group id and Kafka Connect plugins directory. Kafka Connect group id needs to be unique and must not conflict with the consumer group ids
sed -i -e "s#__KAFKA_CONNECT_GROUP_ID__#${KAFKA_CONNECT_GROUP_ID}#g; s#__KAFKA_PLUGINS_DIR__#${KAFKA_PLUGINS_DIR}#g" \
"${KAFKA_CONNECT_CONFIG_FILE}"
# Update Kafka Connect SASL config
sed -i -e "s#__KAFKA_SASL_SERVICE_ACCOUNT__#${KAFKA_SASL_SERVICE_ACCOUNT}#g; s#__KAFKA_SASL_SERVICE_ACCOUNT_KEY__#${KAFKA_SASL_SERVICE_ACCOUNT_KEY}#g" \
"${KAFKA_CONNECT_CONFIG_FILE}"
# Update Kafka Connect SSL truststore config
sed -i -e "s#__KAFKA_SSL_TRUSTSTORE_LOCATION__#${KAFKA_SSL_TRUSTSTORE_LOCATION}#g; s#__KAFKA_SSL_TRUSTSTORE_PASSWORD__#${KAFKA_SSL_TRUSTSTORE_PASSWORD}#g" \
"${KAFKA_CONNECT_CONFIG_FILE}"

#Update PubSub Lite Job File
sed -i -e "s#__PUBSUB_LITE_JOB_NAME__#${PUBSUB_LITE_JOB_NAME}#g; s#__KAFKA_SINK_TOPIC__#${KAFKA_SINK_TOPIC}#g; s#__PUBSUB_LITE_GCP_PROJECT__#${PUBSUB_LITE_GCP_PROJECT}#g; s#__PUBSUB_LITE_GCP_LOCATION__#${PUBSUB_LITE_GCP_LOCATION}#g; s#__PUBSUB_LITE_GCP_SUBSCRIPTION__#${PUBSUB_LITE_GCP_SUBSCRIPTION}#g;" \
"${PUBSUB_LITE_JOB_FILE}"

#Update PSL Job Start Script
sed -i -e "s#__PUBSUB_LITE_JOB_NAME__#${PUBSUB_LITE_JOB_NAME}#g;" \
"${PSL_JOB_STARTUP_SCRIPT}"
Loading