Kafka Connect: Support VARIANT when record convert by seokyun-ha-toss · Pull Request #15283 · apache/iceberg

seokyun-ha-toss · 2026-02-10T09:11:28Z

Summary

Add support for converting arbitrary Java objects (e.g. Map<String, Object>, lists, primitives) into Iceberg Variant type in the Kafka Connect RecordConverter. Nested maps and lists are converted recursively so that structures like {"user": {"name": "alice", "address": {"city": "Seoul", "zip": "12345"}}} are correctly represented as a single Variant.

Motivation

Kafka Connect payloads often come as schema-less or JSON-like maps. To write them into Iceberg tables with a Variant column, the connector must convert these Java objects into the Variant format (metadata + value) and support nested maps/arrays without losing structure or key names.

Behaviour

Input	Result
Primitives (String, int, long, boolean, etc.)	Single metadata (empty) + corresponding Variant primitive.
Flat map e.g. `{"a": 1, "b": "x"}`	One metadata with keys `["a", "b"]`, one ShreddedObject with two fields.
Nested map e.g. `{"user": {"name": "alice", "address": {"city": "Seoul", "zip": "12345"}}}`	One shared metadata for all keys; root and nested objects as ShreddedObjects with consistent field IDs.
Lists	Converted to VariantArray with elements converted recursively.
Already Variant / ByteBuffer	Pass-through or `Variant.from(ByteBuffer)` where appropriate.

Relates

Variant Data Type Support #10392

Thanks, Good Day!

…ersion methods for nested structures

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java

...connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/TestRecordConverter.java

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java

alexkot1394 · 2026-03-11T22:11:02Z

Hi @seokyun-ha-toss,
I've opened a PR that's trying to solve the same issue: #15498
I'm happy to help with your PR to get this change into main. Let me know if you want to collaborate on your PR or where you're at with the review comments above.

seokyun-ha-toss · 2026-03-12T08:12:31Z

Hi @seokyun-ha-toss, I've opened a PR that's trying to solve the same issue: #15498 I'm happy to help with your PR to get this change into main. Let me know if you want to collaborate on your PR or where you're at with the review comments above.

Hi @alexkot1394, thanks for reaching out.

I stepped away for a bit to validate my changes in our production environment.

I took a look at your PR as well. From what I see, it treats Variant as a string primitive, whereas I believe it should be handled recursively to properly construct the Variant structure.

I’d prefer to continue working on this PR and incorporate the review comments as soon as possible.

Thanks again!

…Set for uniqueness

…ordConverter apache#15283 (comment)

… for null, primitive types, lists, maps, and mixed types

…xity

seokyun-ha-toss · 2026-03-13T03:19:47Z

Hello, @danielcweeks, @emkornfield, and @alexkot1394, I'm ready to get a review on this. I've tested this for a month in production, and it works well! I can query the output Iceberg tables with Variant columns using Spark and Snowflake.

Please take a look at the latest updates. Thanks!

brandonstanleyappfolio · 2026-03-17T01:24:17Z

👋 Hi @seokyun-ha-toss, thanks for introducing this change! Is there a reason Struct types aren't included in this PR?

seokyun-ha-toss · 2026-03-17T02:31:04Z

👋 Hi @seokyun-ha-toss, thanks for introducing this change! Is there a reason Struct types aren't included in this PR?

Good catch! I missed the case of org.apache.kafka.connect.data.Struct. I will handle it as well. Thanks!

seokyun-ha-toss · 2026-03-17T10:21:43Z

Hello, @brandonstanleyappfolio. I've added Struct type handling and unit tests in the following two commits:

Thanks for pointing this out!

brandonstanleyappfolio · 2026-03-17T20:06:38Z

@seokyun-ha-toss Thank you - I’ve tested the changes locally, and everything is working as expected!

seokyun-ha-toss · 2026-03-25T05:14:35Z

@danielcweeks @emkornfield Gentle ping — would appreciate your review when you have a moment. Thank you!

emkornfield · 2026-03-25T20:38:06Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java

+          (key, val) -> {
+            if (key != null && key instanceof String) {
+              object.put((String) key, objectToVariantValue(val, metadata));
+            }


seems like we should throw here if the key is null or the value is an instance of string?

(one way would be checking the ShreddedObject size is the same?

emkornfield · 2026-03-25T20:39:24Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java

+    }
+    if (value instanceof Map) {
+      Map<?, ?> map = (Map<?, ?>) value;
+      ShreddedObject object = Variants.object(metadata);


Is there a non-shredded version of Object, any performance differences?

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java

emkornfield · 2026-03-25T20:43:17Z

...connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/TestRecordConverter.java

+    assertThat(variant.value().asPrimitive().get()).isEqualTo(true);
+  }
+
+  @Test


does this test add any value of the test below?

emkornfield · 2026-03-25T20:44:14Z

...connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/TestRecordConverter.java

+
+  @Test
+  public void testConvertVariantValueFromMap() {
+    Variant variant = variantConverter().convertVariantValue(ImmutableMap.of("hello", 1));


should we have multiple value types? Also we should make sure we test the case when the key is null or not a string.

emkornfield · 2026-03-25T20:46:26Z

...connect/kafka-connect/src/test/java/org/apache/iceberg/connect/data/TestRecordConverter.java

+  }
+
+  @Test
+  public void testConvertVariantValueFromMixedNested() {


should we tested nested values within a struct?

emkornfield

My primary concern is potentially dropping keys that a null or not a string and being defensive against data loss.

brandonstanleyappfolio · 2026-03-26T01:45:14Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java

+    }
+    if (value instanceof Short) {
+      return Variants.of((Short) value);
+    }


Should we also consider Dates?

I interpreted from the pr description that these are derived from JSON which doesn't have dates? But this is a good clarifying question.

That makes sense! I can provide some context around my use case if that helps.

I am loading Confluent Avro-serialized data that contains timestamp fields. When those fields are deserialized by the connector, they are converted to java.util.Date, which this PR does not currently handle. I added the following code to RecordConverter.java to resolve the issue:

if (value instanceof Date) { return Variants.of(((Date) value).getTime()); }

I also see that @seokyun-ha-toss added support for dates here!

…nverter https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/math/BigDecimal.html#%3Cinit%3E(java.math.BigInteger)

emkornfield · 2026-03-30T20:35:21Z

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java

+      return Variants.ofTime(DateTimeUtil.microsFromTime((LocalTime) value));
+    }
+    if (value instanceof Date) {
+      int days = (int) (((Date) value).getTime() / 1000 / 60 / 60 / 24);


The java.util.Date class is poorly named, i think we should be mapping this to Timestamp TZ so we don't lose precision. https://docs.oracle.com/javase/8/docs/api/java/util/Date.html for details.

That’s a good point. Do we need to add a check to determine the exact logical type?
I’ve opened this PR against @seokyun-ha-toss's branch. Let me know your thoughts @seokyun-ha-toss @emkornfield!

Wow, really appreciate it @brandonstanleyappfolio , Thanks! I'll take a look!!

Sadly, I don't have much time to work on in these days.... I'll resume this work as soon as possible. Thanks for guys! 🙏

seokyun-ha-toss added 2 commits February 10, 2026 17:17

feat: Implement support for VARIANT type in RecordConverter with conv…

033f47f

…ersion methods for nested structures

test: implement unit test for RecordConverter for Variant

a53af92

github-actions bot added the KAFKACONNECT label Feb 10, 2026

chore: lint spotlessApply

73ea656

emkornfield reviewed Feb 10, 2026

View reviewed changes

danielcweeks reviewed Feb 10, 2026

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java Outdated Show resolved Hide resolved

danielcweeks reviewed Feb 10, 2026

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java Outdated Show resolved Hide resolved

danielcweeks reviewed Feb 10, 2026

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java Outdated Show resolved Hide resolved

fix: there's no already Variant value, and use Lists.of()

2b3fcb1

danielcweeks mentioned this pull request Mar 11, 2026

Kafka Connect: Add variant type support #15498

Open

seokyun-ha-toss added 8 commits March 13, 2026 07:24

fix: handle BigDecimal type in numberToVariantValue()

47f6fe5

fix: explicit error when unknown numeric type

e8f57f1

refactor: optimize field name collection in RecordConverter to use a …

b57e0d8

…Set for uniqueness

fix: ensure keys in map are non-null strings before processing in Rec…

0117174

…ordConverter apache#15283 (comment)

refactor: loop entrySet once

be09349

test: enhance unit tests for Variant conversion with additional cases…

75ec7ce

… for null, primitive types, lists, maps, and mixed types

lint: apply gh actions auto-review and lints, split Cyclomatic Comple…

c6e558a

…xity

lint: spotlessApply

787663d

seokyun-ha-toss added 3 commits March 17, 2026 18:53

feat: support kafka Struct type, and align if checking ordering

75dc8a2

test: implt unittest for Variant for Struct

d9c8721

lint: gradlew spotlessApply

e517884

emkornfield reviewed Mar 25, 2026

View reviewed changes

kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java Show resolved Hide resolved

emkornfield reviewed Mar 25, 2026

View reviewed changes

emkornfield suggested changes Mar 25, 2026

View reviewed changes

brandonstanleyappfolio reviewed Mar 26, 2026

View reviewed changes

seokyun-ha-toss added 4 commits March 27, 2026 13:07

feat: add support for BigInteger conversion to BigDecimal in RecordCo…

8bb8740

…nverter https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/math/BigDecimal.html#%3Cinit%3E(java.math.BigInteger)

feat: support for Date conversion to Variants

aa2bcce

feat: support additional Date, Datetime, time types

5be9fa4

lint: spotlessApply

179b07a

emkornfield reviewed Mar 30, 2026

View reviewed changes

Conversation

seokyun-ha-toss commented Feb 10, 2026

Summary

Motivation

Behaviour

Relates

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexkot1394 commented Mar 11, 2026

Uh oh!

seokyun-ha-toss commented Mar 12, 2026

Uh oh!

seokyun-ha-toss commented Mar 13, 2026

Uh oh!

brandonstanleyappfolio commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seokyun-ha-toss commented Mar 17, 2026

Uh oh!

seokyun-ha-toss commented Mar 17, 2026

Uh oh!

brandonstanleyappfolio commented Mar 17, 2026

Uh oh!

seokyun-ha-toss commented Mar 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brandonstanleyappfolio Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

brandonstanleyappfolio commented Mar 17, 2026 •

edited

Loading

brandonstanleyappfolio Mar 30, 2026 •

edited

Loading