feat: use files hosted on cloudflare r2 instead of git lfs #94
Conversation
|
info: file combination (chunks) are no longer there as the 4 GiB file limit is an LFS thing, so we can directly host the combined compressed state file on R2 instead of having to combine the chunks ourselves. |
dapplion
left a comment
dapplion
left a comment
There was a problem hiding this comment.
Nice, better hosting solution. But you need to check integrity
| )]; | ||
|
|
||
| fn get_chunks(chain: &str) -> Vec<(&'static str, u64)> { | ||
| const R2_BASE: &str = "https://initstate.gnosischain.com"; |
There was a problem hiding this comment.
Can you make this a CLI arg that defaults to this value? Ok to do in another PR
There was a problem hiding this comment.
created an issue for later
| fn get_compressed_state_size(chain: &str) -> u64 { | ||
| match chain { | ||
| "gnosis" => 5_672_943_444, | ||
| "chiado" => 22_468_068, |
There was a problem hiding this comment.
You should hardcode the file hash and check integrity after downloading. Much better to check local file is fine that just the size.
There was a problem hiding this comment.
The trust assumption otherwise it's very sketchy: hey download a random file from our URL that you can't change and hope it's fine. Even if no-one uses it you should add a hackmd somewhere with steps to recreate those files and serve them in an independent route
There was a problem hiding this comment.
made hash verification for the downloaded files, and added a markdown for recreating the state files
| println!("✅ compressed state verified"); | ||
| } else { | ||
| println!("✅ compressed state already present"); | ||
| println!("✅ compressed state verified"); |
There was a problem hiding this comment.
Use tracing to be consistent with the rest of reth logging
There was a problem hiding this comment.
tracing cannot be used here as this part of the code is before tracing is initialized (at node startup)
we are reaching git lfs limits with a small number of downloads, this fixes the problem by enabling unlimited downloads from r2 and ditching git lfs (along with the complexity it came with).
the files are still canonical with the git repo ones so everyone can check the served files.