Is it possible to serialize a 100GB data (vector of tables) into a single FlatBuffer?

[lee-junjie]

I appreciate the excellent work you’ve done on FlatBuffers.

I have a dataset that I’d like to serialize. It’s a vector of tables, and the total size is over 100GB (approximately 1000 items, each around 0.1GB).

I have a few questions that I hope you could help me with:

• Is it possible to serialize the entire dataset into a single FlatBuffer?
• Alternatively, do I need to divide the dataset into 1000 segments and serialize/deserialize each one individually?
• Do you have any other recommendations for achieving the fastest serialization/deserialization speed for my dataset?

Note: currently, offset64 doesn’t support vectors of tables. When I attempt to set offset64, I receive the following message: “only vectors of scalars are allowed to be 64-bit.”

Can you provide any guidance on this? Thanks in advance!

[Advait251206]

The main limitation you'll run into is that traditional FlatBuffers use 32-bit offsets (uoffset_t), which effectively limits what can be addressed within a single buffer.

A 100 GB FlatBuffer containing a vector of tables is far beyond the range that a 32-bit offset can represent, so I would not expect a single conventional FlatBuffer to work for this use case.

The error you're seeing:

"only vectors of scalars are allowed to be 64-bit"

suggests you're already running into the current limitations of offset64 support. As far as I know, 64-bit offsets are not universally supported across all FlatBuffers constructs, and vectors of tables are one of the notable gaps.

Given your dataset size, I would seriously consider partitioning it rather than trying to force everything into a single FlatBuffer. Some possible approaches:

• Split the data into multiple FlatBuffers (e.g. one per item, shard, or chunk).
• Maintain an external index that maps IDs to individual FlatBuffer files.
• Store large binary payloads separately and reference them logically.
• Use memory-mapped files so only the required portions are loaded on demand.

For a dataset consisting of ~1000 objects at ~0.1 GB each, chunking often has additional benefits:

• Lower peak memory requirements during serialization.
• Faster partial reads.
• Easier updates and replacements.
• Better fault isolation if a file becomes corrupted.
• Improved parallel processing opportunities.

For deserialization performance specifically, FlatBuffers already performs very well when data can be accessed directly from memory or via memory mapping. In many large-scale systems, the bottleneck becomes storage I/O rather than the FlatBuffers access layer itself.

So unless there has been recent work extending offset64 support for vectors of tables, I would expect the recommended solution to be a collection of FlatBuffers (potentially with an index structure) rather than a single 100 GB buffer.