Skip to content

triedb/pathdb: introduce file-based state journal #32060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 19 commits into from
Jul 15, 2025
Merged

triedb/pathdb: introduce file-based state journal #32060

merged 19 commits into from
Jul 15, 2025

Conversation

jsvisa
Copy link
Contributor

@jsvisa jsvisa commented Jun 18, 2025

allow write trie journal into a local file instead of disk, used when the trie jounral is >= 4GB, ref gballet#546 (comment)

@jsvisa jsvisa requested a review from rjl493456442 as a code owner June 18, 2025 08:01
@rjl493456442 rjl493456442 self-assigned this Jun 18, 2025
rjl493456442
rjl493456442 reviewed Jun 19, 2025
View reviewed changes
jsvisa
jsvisa commented Jun 19, 2025
View reviewed changes
jsvisa
jsvisa commented Jun 19, 2025
View reviewed changes
@jsvisa jsvisa force-pushed the journal-into-file branch from 2ca455d to 2d222b2 Compare June 19, 2025 13:35
@rjl493456442 rjl493456442 mentioned this pull request Jun 20, 2025
Open
2 tasks
@jsvisa jsvisa force-pushed the journal-into-file branch from 2d222b2 to 9abf10a Compare June 23, 2025 09:14
@fjl
Copy link
Contributor

fjl commented Jun 24, 2025

The idea is good, but the path should not be configurable. We should just resolve the path in the datadir and store the file there unconditionally.

@fjl fjl removed the status:triage label Jun 24, 2025
@jsvisa jsvisa force-pushed the journal-into-file branch from 9abf10a to 516c1b3 Compare June 25, 2025 07:04
@rjl493456442
Copy link
Member

rjl493456442 commented Jun 27, 2025 •
edited
Loading

This pull request is backward-compatible:

  • Upgrade from old version, the journal will be resolved from database and persist into file
  • Downgrade to old version, Geth will attempt to resolve journal from database only and drop in-memory layers if not matched

The journal directory will be $DATADIR/triedb, with $DATADIR/triedb/merkle.journal for legacy merkle trie database and $DATADIR/triedb/verkle.journal for verkle respectively.

@rjl493456442
Copy link
Member

logs:

INFO [06-27|14:17:56.460] Load database journal from file          path=/home/gary/eth-sepolia/el/geth/triedb/merkle.journal

...

INFO [06-27|14:21:04.978] Persisting dirty state                   head=5,276,356 root=b0d1cb..19ba68 layers=1551
INFO [06-27|14:21:06.556] Persisted dirty state to file            path=/home/gary/eth-sepolia/el/geth/triedb/merkle.journal size=342.21MiB elapsed=1.578s

rjl493456442
rjl493456442 previously approved these changes Jun 27, 2025
View reviewed changes
Copy link
Member

@rjl493456442 rjl493456442 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

jsvisa and others added 15 commits July 1, 2025 10:09
@jsvisa @rjl493456442
all: dump trie journal into file (ethereum#546)
9db6f07 
* all: load/dump trie disklayer into file

Signed-off-by: jsvisa <[email protected]>

* cmd: set cache.trie.journal=trie-disklayer.rlp as default

Signed-off-by: jsvisa <[email protected]>

* journal: directly write into file

Signed-off-by: jsvisa <[email protected]>

---------

Signed-off-by: jsvisa <[email protected]>
@jsvisa @rjl493456442
cmd: no-legacy cache.trie.journal
bc358d5 
Signed-off-by: jsvisa <[email protected]>
@jsvisa @rjl493456442
disable by default
82d57f9 
Signed-off-by: jsvisa <[email protected]>
@jsvisa @rjl493456442
atomic file write
c7d2d30 
Signed-off-by: jsvisa <[email protected]>
@rjl493456442
cmd, core, eth, triedb/pathdb: improve journal file flushing
22e1c62 
Signed-off-by: jsvisa <[email protected]>
@rjl493456442
eth/ethconfig: regenerate config
ec518e3 
Signed-off-by: jsvisa <[email protected]>
@rjl493456442
cmd/geth: prevent double definition
4b5cf34 
Signed-off-by: jsvisa <[email protected]>
@rjl493456442
cmd, eth: fix journal path
f074797 
Signed-off-by: jsvisa <[email protected]>
@rjl493456442
triedb/pathdb: fix tests
d5da02a 
Signed-off-by: jsvisa <[email protected]>
@jsvisa @rjl493456442
Apply suggestions from code review
525aacb 
Signed-off-by: jsvisa <[email protected]>
@jsvisa @rjl493456442
dup assign
fef157d 
Signed-off-by: jsvisa <[email protected]>
@jsvisa @rjl493456442
set triedb.journal as a switch
3cc8e23 
Signed-off-by: jsvisa <[email protected]>
@rjl493456442
triedb/pathdb: change to the default datadir
6bb3dfc
@rjl493456442
eth, triedb: fix mkdir
7afd7d0
@rjl493456442
triedb/path: update logs
e9689c3
rjl493456442
rjl493456442 previously approved these changes Jul 1, 2025
View reviewed changes
@rjl493456442 rjl493456442 changed the title trie: write trie journal into disk triedb/pathdb: introduce file-based state journal Jul 1, 2025
@rjl493456442 rjl493456442 added this to the 1.16.1 milestone Jul 1, 2025
@fjl fjl modified the milestones: 1.16.1, 1.16.2 Jul 2, 2025
@rjl493456442 rjl493456442 merged commit 17903fe into ethereum:master Jul 15, 2025
2 of 3 checks passed
@fjl fjl mentioned this pull request Jul 31, 2025
Closed
@BrewTestBot BrewTestBot mentioned this pull request Aug 4, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rjl493456442 rjl493456442 rjl493456442 approved these changes

Assignees

@rjl493456442 rjl493456442

Labels
None yet
Projects
None yet
Milestone
1.16.2
Development

Successfully merging this pull request may close these issues.
3 participants
@jsvisa @fjl @rjl493456442