Skip to content

Commit ad623e0

Browse files
DOC-6088 updated summary list
1 parent e96a46a commit ad623e0

File tree

1 file changed

+74
-69
lines changed

1 file changed

+74
-69
lines changed

content/integrate/redis-data-integration/data-pipelines/supported-types.md

Lines changed: 74 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -18,17 +18,57 @@ weight: 80
1818
The sections below describe the data types supported by RDI for each source database.
1919
There are also some cross‑cutting considerations that apply to all source databases.
2020

21+
## Quick configuration summary
22+
23+
The lists below summarize the extra configuration steps you may need to take
24+
for each source database to support all the types you need. Each
25+
database has its own section in the page with more detail about the supported types.
26+
27+
[**Oracle**](#oracle)<br/>
28+
29+
- Enable supplemental logging on tables/schemas.
30+
- Set `lob.enabled=true` if you need `CLOB`/`BLOB`/`XMLTYPE`.
31+
- Add Oracle XML JARs (`xdb.jar`, `xmlparserv2.jar`) if using `XMLTYPE`.
32+
- Decide on `binary.handling.mode` for `RAW`/`BLOB`.
33+
- Avoid unsupported types (LONG, BFILE, complex UDTs) or cast them to supported forms.
34+
35+
[**MySQL/MariaDB**](#mysql-and-mariadb)<br/>
36+
37+
- Enable binlog in **ROW** mode.
38+
- Configure `decimal.handling.mode` to balance precision vs convenience.
39+
- Be aware that spatial types are not supported.
40+
- Treat `BIT` columns as binary (or cast to integer/boolean in SQL).
41+
42+
[**PostgreSQL/Supabase/AlloyDB**](#postgresql-supabase-and-alloydb)<br/>
43+
44+
- Ensure WAL/replication settings match connector needs.
45+
- Avoid array/composite/spatial types in replicated columns (or cast them to JSON/TEXT).
46+
- Use RedisJSON target to get most value from `JSONB`.
47+
48+
[**SQL Server**](#sql-server)<br/>
49+
50+
- Enable CDC or transaction log access as required by the connector.
51+
- Mind `MONEY`/`DECIMAL` precision—store as strings when exactness is critical.
52+
- Exclude or transform unsupported types (`sql_variant`, spatial types).
53+
54+
[**MongoDB**](#mongodb)<br/>
55+
56+
- Ensure replica set/change streams are configured.
57+
- Prefer RedisJSON outputs to preserve document structure.
58+
- For large binary data, confirm Base64 encoding is acceptable to your consumers.
59+
60+
2161
## Oracle
2262

23-
### Core Supported Types
63+
### Core supported types
2464

2565
RDI (via the Debezium Oracle connector) supports Oracle’s core scalar types:
2666

2767
- **Numeric**: `NUMBER`, `DECIMAL`
2868
- **Character**: `CHAR`, `VARCHAR2`, `NCHAR`, `NVARCHAR2`
2969
- **Temporal**: `DATE`, `TIMESTAMP`, `TIMESTAMP WITH TIME ZONE`, `TIMESTAMP WITH LOCAL TIME ZONE`
3070

31-
**Behavior & mapping:**
71+
**Behavior and mapping:**
3272

3373
- `NUMBER(p,s)` / `DECIMAL(p,s)`
3474
- Ingested with full precision and scale.
@@ -46,7 +86,7 @@ RDI (via the Debezium Oracle connector) supports Oracle’s core scalar types:
4686
- `TIMESTAMP WITH TIME ZONE`
4787
- Normalized to UTC, then converted to epoch milliseconds.
4888

49-
### Large Objects (LOBs)
89+
### Large objects (LOBs)
5090

5191
Oracle LOBs are supported **with additional configuration**:
5292

@@ -70,7 +110,7 @@ lob.enabled=true
70110
- `BLOB` → Base64‑encoded string (by default).
71111
- In Redis Hashes, these are string fields; in RedisJSON they are JSON strings.
72112

73-
### Binary & RAW
113+
### Binary and RAW
74114

75115
- **Types**: `BLOB`, `RAW`, `LONG RAW` (see unsupported below).
76116

@@ -105,7 +145,7 @@ Oracle `XMLTYPE` is backed by LOBs and **needs extra setup**:
105145
- `XMLTYPE` values are captured as XML **text**.
106146
- Redis stores them as strings (Hash or JSON). If desired, you can post‑process XML into JSON with an RDI job.
107147

108-
### Unsupported / Special Oracle Types
148+
### Unsupported / special Oracle types
109149

110150
Not supported (or skipped) by the Debezium Oracle connector and therefore by RDI:
111151

@@ -122,7 +162,7 @@ Not supported (or skipped) by the Debezium Oracle connector and therefore by RDI
122162
- In **LogMiner** mode, `ROWID` is supported and captured as a string.
123163
- In **XStream** mode, `ROWID` is not exposed.
124164

125-
### Oracle → Redis Mapping Summary
165+
### Oracle → Redis mapping summary
126166

127167
| Oracle type | Support level | Notes / requirements |
128168
|---------------------------|--------------------|------------------------------------------------------------|
@@ -139,9 +179,9 @@ Not supported (or skipped) by the Debezium Oracle connector and therefore by RDI
139179

140180
---
141181

142-
## MySQL & MariaDB
182+
## MySQL and MariaDB
143183

144-
### Core Types
184+
### Core types
145185

146186
RDI (via Debezium MySQL) supports all common MySQL and MariaDB types:
147187

@@ -153,9 +193,10 @@ RDI (via Debezium MySQL) supports all common MySQL and MariaDB types:
153193

154194
**Mapping:**
155195

156-
- Integers & floats: string in Hash, numeric in JSON.
196+
- Integers and floats: string in Hash, numeric in JSON.
157197
- `DECIMAL` / `NUMERIC`:
158-
- Default: preserved precision via `decimal.handling.mode`.
198+
- Default: preserved precision via `decimal.handling.mode` (default setting
199+
is `debezium.source.decimal.handling.mode=string`).
159200
- Recommended: treat high‑precision values as strings to avoid rounding.
160201
- Text types: stored as strings.
161202
- Temporal:
@@ -187,21 +228,21 @@ No special configuration is required.
187228
- Treated as integer year (e.g. `2025`).
188229
- String in Hash, numeric in JSON.
189230

190-
### Binary & BLOBs
231+
### Binary and BLOBs
191232

192233
- `BINARY`, `VARBINARY`, `BLOB` variants:
193234
- Debezium emits bytes using configured `binary.handling.mode` (Base64 by default).
194235
- RDI stores the resulting encoded string.
195236

196-
### Unsupported / Partial Types
237+
### Unsupported / partial types
197238

198239
- **Spatial / GIS types**: `GEOMETRY`, `POINT`, `LINESTRING`, `POLYGON`, etc.
199240
- Not supported out of the box. Columns with these types are skipped.
200241
- **BIT**:
201242
- Debezium may represent it as raw bytes.
202243
- RDI stores as an encoded binary string; no automatic boolean conversion.
203244

204-
### MySQL/MariaDB → Redis Mapping Summary
245+
### MySQL/MariaDB → Redis mapping summary
205246

206247
| MySQL/MariaDB type | Support level | Notes |
207248
|-------------------------|----------------|-----------------------------------------------------------|
@@ -218,9 +259,9 @@ No special configuration is required.
218259

219260
---
220261

221-
## PostgreSQL, Supabase & AlloyDB
262+
## PostgreSQL, Supabase, and AlloyDB
222263

223-
### Core Types
264+
### Core types
224265

225266
Supported PostgreSQL scalar types:
226267

@@ -229,7 +270,7 @@ Supported PostgreSQL scalar types:
229270
- **Binary**: `BYTEA`.
230271
- **Temporal**: `DATE`, `TIME`, `TIMESTAMP`, `TIMESTAMPTZ`.
231272
- **Boolean**: `BOOLEAN`.
232-
- **Identifiers & network**: `UUID`, `INET`, `CIDR`, `MACADDR`.
273+
- **Identifiers and network**: `UUID`, `INET`, `CIDR`, `MACADDR`.
233274
- **Enums**: PostgreSQL `ENUM`.
234275

235276
**Mapping:**
@@ -251,7 +292,7 @@ Supported PostgreSQL scalar types:
251292
- Stored as a key/value representation (e.g. `'"k1"=>"v1","k2"=>"v2"'`) as a string by default.
252293
- You can post‑process to a JSON object with an RDI job if desired.
253294

254-
### Unsupported / Complex Types
295+
### Unsupported / complex types
255296

256297
Not supported in Debezium’s PostgreSQL connector (and thus RDI):
257298

@@ -264,13 +305,13 @@ Not supported in Debezium’s PostgreSQL connector (and thus RDI):
264305

265306
If these are present, the connector generally skips those fields and logs warnings.
266307

267-
### Supabase & AlloyDB Notes
308+
### Supabase and AlloyDB notes
268309

269310
- **Supabase**: is PostgreSQL; all behavior above applies.
270311
- **AlloyDB**: also PostgreSQL‑compatible; same behavior for standard types.
271312
- Newer types like vector/embedding columns are generally not handled unless explicitly cast (e.g. to `TEXT`).
272313

273-
### PostgreSQL → Redis Mapping Summary
314+
### PostgreSQL → Redis mapping summary
274315

275316
| PostgreSQL type | Support level | Notes |
276317
|-------------------------|---------------|------------------------------------------------------------|
@@ -293,7 +334,7 @@ If these are present, the connector generally skips those fields and logs warnin
293334

294335
## SQL Server
295336

296-
### Core Types
337+
### Core types
297338

298339
Supported SQL Server scalar types:
299340

@@ -316,7 +357,7 @@ Supported SQL Server scalar types:
316357
- 8‑byte binary value.
317358
- Debezium usually exposes this as hex; RDI stores a hex string.
318359

319-
### Temporal Types
360+
### Temporal types
320361

321362
- `DATETIME`, `SMALLDATETIME`:
322363
- Converted to epoch ms (with `SMALLDATETIME` rounded to minute).
@@ -328,7 +369,7 @@ Supported SQL Server scalar types:
328369
- `DATE`: ms at midnight.
329370
- `TIME`: ms from midnight.
330371

331-
### LOBs & Binary
372+
### LOBs and binary
332373

333374
- `VARBINARY(MAX)`, `IMAGE`:
334375
- Captured as bytes and encoded (Base64, etc.).
@@ -339,15 +380,15 @@ Supported SQL Server scalar types:
339380

340381
No special connector tuning is usually required beyond standard CDC configuration.
341382

342-
### Unsupported / Special Types
383+
### Unsupported / special types
343384

344385
- `sql_variant`: not supported.
345386
- `hierarchyid`: not supported.
346387
- Spatial types:
347388
- `geometry`, `geography` – not supported; columns are skipped.
348389
- Table‑valued types: not applicable as column types in normal tables.
349390

350-
### SQL Server → Redis Mapping Summary
391+
### SQL Server → Redis mapping summary
351392

352393
| SQL Server type | Support level | Notes |
353394
|---------------------------|---------------|-------------------------------------------------------------|
@@ -367,7 +408,7 @@ No special connector tuning is usually required beyond standard CDC configuratio
367408

368409
## MongoDB
369410

370-
### Core BSON Types
411+
### Core BSON types
371412

372413
MongoDB’s JSON‑like model maps very naturally into RedisJSON (or Hashes). Debezium’s Mongo connector (used by RDI) supports:
373414

@@ -394,7 +435,7 @@ MongoDB’s JSON‑like model maps very naturally into RedisJSON (or Hashes). De
394435
- `DBRef` (as a small document)
395436
- MinKey/MaxKey – rarely used, mostly internal.
396437

397-
### Mapping Behavior
438+
### Mapping behavior
398439

399440
By default, RDI maps **one MongoDB document → one Redis key** (often RedisJSON):
400441

@@ -419,18 +460,18 @@ By default, RDI maps **one MongoDB document → one Redis key** (often RedisJSON
419460
- Binary:
420461
- Binary data Base64‑encoded into a string.
421462

422-
- Documents & Arrays:
463+
- Documents and Arrays:
423464
- RedisJSON target: nested objects / arrays preserved exactly.
424465
- Hash target: nested documents and arrays are **stringified** JSON (e.g. `address` field contains the JSON string of the subdocument).
425466

426-
### Less Common BSON Types
467+
### Less common BSON types
427468

428469
- **Regex**: stored as a string representation (e.g. `"/^abc/i"`).
429470
- **JavaScript / JavaScriptWithScope**: captured as code text.
430471
- **DBRef**: captured as a small document with `$ref` and `$id`, then mapped like a normal subdocument.
431472
- **MinKey / MaxKey**: may be treated as extreme sentinels or omitted; rarely used in user data.
432473

433-
### MongoDB → Redis Mapping Summary
474+
### MongoDB → Redis mapping summary
434475

435476
| MongoDB/BSON type | Support level | Redis representation |
436477
|---------------------|---------------|-------------------------------------------------------------|
@@ -450,7 +491,7 @@ By default, RDI maps **one MongoDB document → one Redis key** (often RedisJSON
450491

451492
---
452493

453-
## Cross‑Cutting Considerations
494+
## Cross-cutting considerations
454495

455496
### Nullability
456497

@@ -459,7 +500,7 @@ By default, RDI maps **one MongoDB document → one Redis key** (often RedisJSON
459500
- **RedisJSON**:
460501
- Null values become JSON `null`.
461502

462-
### Precision & Scale
503+
### Precision and scale
463504

464505
For high‑precision numeric values (money, scientific values):
465506

@@ -468,14 +509,14 @@ For high‑precision numeric values (money, scientific values):
468509
- In Redis:
469510
- Use JSON strings if you need exact decimal fidelity, or JSON numbers if range is within double precision.
470511

471-
### Time Zones
512+
### Time zones
472513

473514
- Source time zone handling is connector‑specific.
474515
- RDI generally:
475516
- Converts to UTC where the source type contains a time zone.
476517
- Stores epoch milliseconds for timestamps in Redis.
477518

478-
### Binary Handling
519+
### Binary handling
479520

480521
- Controlled by Debezium’s `binary.handling.mode` (exact property name may vary by connector).
481522
- Common values:
@@ -487,39 +528,3 @@ Ensure your consumer understands the chosen encoding.
487528

488529
---
489530

490-
## Quick Configuration Checklist
491-
492-
For advanced RDI users, here are the **common extra steps** by database:
493-
494-
### Oracle
495-
496-
- ✅ Enable supplemental logging on tables/schemas.
497-
- ✅ Set `lob.enabled=true` if you need `CLOB`/`BLOB`/`XMLTYPE`.
498-
- ✅ Add Oracle XML JARs (`xdb.jar`, `xmlparserv2.jar`) if using `XMLTYPE`.
499-
- ✅ Decide on `binary.handling.mode` for `RAW`/`BLOB`.
500-
- ✅ Avoid unsupported types (LONG, BFILE, complex UDTs) or cast them to supported forms.
501-
502-
### MySQL / MariaDB
503-
504-
- ✅ Binary log enabled in **ROW** mode.
505-
- ✅ Configure `decimal.handling.mode` to balance precision vs convenience.
506-
- ✅ Be aware that spatial types are not supported.
507-
- ✅ Treat `BIT` columns as binary (or cast to integer/boolean in SQL).
508-
509-
### PostgreSQL / Supabase / AlloyDB
510-
511-
- ✅ Ensure WAL/replication settings match connector needs.
512-
- ✅ Avoid array/composite/spatial types in replicated columns (or cast them to JSON/TEXT).
513-
- ✅ Use RedisJSON target to get most value from `JSONB`.
514-
515-
### SQL Server
516-
517-
- ✅ Enable CDC or transaction log access as required by the connector.
518-
- ✅ Mind `MONEY`/`DECIMAL` precision—store as strings when exactness is critical.
519-
- ✅ Exclude or transform unsupported types (`sql_variant`, spatial types).
520-
521-
### MongoDB
522-
523-
- ✅ Ensure replica set / change streams are configured.
524-
- ✅ Prefer RedisJSON outputs to preserve document structure.
525-
- ✅ For large binary data, confirm Base64 encoding is acceptable to your consumers.

0 commit comments

Comments
 (0)