Block Database

[DracoLi]

Why this should be merged

This PR introduces BlockDB, a specialized database optimized for block storage.

Avalanche VMs currently store blocks in a key-value database (LevelDB or PebbleDB). This method is not ideal for block storage because large blocks lead to frequent compactions, resulting in write amplification that degrades performance as the database expands. Additionally, key-value databases are designed for random key-value access, not the sequential patterns typical of blockchain operations.

For how BlockDB works see README.md.

Changes

• Added blockdb to x/.
• Updated our lru cache to support onEvict. This is used by the blockdb for storing opened file descriptors for the data files.

How this was tested

Units tests for now

Todos

• Split data across multiple files when MaxDataFileSize is reached

[Copilot]

Pull Request Overview

This PR introduces BlockDB, a specialized on-disk database optimized for blockchain block storage with improved write performance and automatic recovery.

• Implements dedicated tests for writing, reading, concurrency, and error cases.
• Introduces recovery logic and index management for efficient block lookups, along with detailed documentation in the README.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
x/blockdb/writeblock_test.go	Adds comprehensive tests verifying block writes, error conditions, and concurrency scenarios.
x/blockdb/recovery.go	Introduces recovery logic to reconcile the data and index file contents after crashes.
x/blockdb/readblock_test.go	Provides test coverage for reading full blocks, headers, and bodies in various conditions.
x/blockdb/index.go	Implements fixed-size index entries and header serialization/deserialization.
x/blockdb/database.go	Sets up file handling, header initialization, recovery trigger, and connection closure.
x/blockdb/block.go	Implements block header serialization, writing/reading blocks, and ensuring data integrity.
x/blockdb/config.go	Defines default and custom configuration options for the BlockDB.
x/blockdb/README.md	Documents design, file formats, recovery, and usage of BlockDB.

[rkuris]

Still unreviewed: recovery code and some of the block allocation logic, but there is enough here to get started with some changes.

[rkuris]

Why do we need a reserved area here?

[DracoLi]

I wanted to account for we might add features that will require us to store more data in the header in future versions. If this happens, we can add it here without needing to reindex.

[yacovm]

Made a first pass, great work.

[yacovm]

I think we can merge it but we still should address and implement the issues opened before we actually let users activate this.