core/rawdb, triedb/pathdb: implement history indexer

[rjl493456442]

This pull request is part-1 for shipping the core part of archive node over path mode.
These following things have been implemented:

• state history index definition
• state history indexer
• state history reader

[rjl493456442]

Can we get rid of the fields of id and min?

[MariusVanDerWijden]

I think you can get easily rid of min and its also possible to not store id, I think. Since id is part of the key, we could just add that to the entry when its loaded and not store it in the db

[rjl493456442]

I would keep the ID for a while, reasons:

It's technically possible to resolve the IDs from the database key by iterating the database. But

• it's IO expensive
• it's not robust, e.g., the stale index blocks will also be scanned
• it's fairly easy to purge the existing indexes and regenerate them later

[MariusVanDerWijden]

Just thinking out loud, the whole following code depends on the historyIDs to be sorted. So if we ever called process with an out of order historyID, this would not work anymore. Should we make sure before writing out the lists, that they are sorted?

[rjl493456442]

The histories are resolved from the freezer in batch. I think the assumption is held that histories are processed in order.

[MariusVanDerWijden]

Are we loosing something if we use the account hash over the account address?

[MariusVanDerWijden]

I don't understand this logic, if the range delete errors with tooManyKeys, we will continuously try to delete it. Wouldn't it always error with TooManyKeys then?

[rjl493456442]

Nope, each call will make some progress by deleting items. Ultimately it will remove all the items in the range.

[rjl493456442]

Totat storage size of a fully-sych'd archive node is around 1.9TB

gary@dev:~/hdd2$ du -sh geth-ancient-mainnet-archive/
1.5T    geth-ancient-mainnet-archive/
gary@dev:~/hdd2$ du -sh geth-ancient-mainnet-archive/ancient/chain/
921G    geth-ancient-mainnet-archive/ancient/chain/
gary@dev:~/hdd2$ du -sh geth-ancient-mainnet-archive/ancient/state
539G    geth-ancient-mainnet-archive/ancient/state

gary@dev:~$ du -sh mount/geth/geth
413G    mount/geth/geth

+-----------------------+-----------------------------+------------+------------+
|       DATABASE        |          CATEGORY           |    SIZE    |   ITEMS    |
+-----------------------+-----------------------------+------------+------------+
| Key-Value store       | Headers                     | 2.37 MiB   |       3640 |
| Key-Value store       | Bodies                      | 333.02 MiB |       3640 |
| Key-Value store       | Receipt lists               | 316.00 MiB |       3639 |
| Key-Value store       | Difficulties (deprecated)   | 0.00 B     |          0 |
| Key-Value store       | Block number->hash          | 149.26 KiB |       3639 |
| Key-Value store       | Block hash->number          | 888.96 MiB |   22735156 |
| Key-Value store       | Transaction index           | 13.84 GiB  |  401645299 |
| Key-Value store       | Log index filter-map rows   | 0.00 B     |          0 |
| Key-Value store       | Log index last-block-of-map | 0.00 B     |          0 |
| Key-Value store       | Log index block-lv          | 0.00 B     |          0 |
| Key-Value store       | Log bloombits (deprecated)  | 0.00 B     |          0 |
| Key-Value store       | Contract codes              | 10.19 GiB  |    1699192 |
| Key-Value store       | Hash trie nodes             | 0.00 B     |          0 |
| Key-Value store       | Path trie state lookups     | 888.86 MiB |   22732693 |
| Key-Value store       | Path trie account nodes     | 47.26 GiB  |  409852077 |
| Key-Value store       | Path trie storage nodes     | 179.98 GiB | 1791011965 |
| Key-Value store       | Path state history indexes  | 290.62 GiB | 4112383456 |
| Key-Value store       | Verkle trie nodes           | 0.00 B     |          0 |
| Key-Value store       | Verkle trie state lookups   | 0.00 B     |          0 |
| Key-Value store       | Trie preimages              | 2.07 MiB   |      31025 |
| Key-Value store       | Account snapshot            | 13.75 GiB  |  299285136 |
| Key-Value store       | Storage snapshot            | 95.42 GiB  | 1322797351 |
| Key-Value store       | Beacon sync headers         | 18.25 MiB  |      29397 |
| Key-Value store       | Clique snapshots            | 0.00 B     |          0 |
| Key-Value store       | Singleton metadata          | 202.94 MiB |         15 |
| Ancient store (Chain) | Headers                     | 10.92 GiB  |   22731517 |
| Ancient store (Chain) | Hashes                      | 823.78 MiB |   22731517 |
| Ancient store (Chain) | Bodies                      | 655.40 GiB |   22731517 |
| Ancient store (Chain) | Receipts                    | 253.37 GiB |   22731517 |
| Ancient store (State) | Storage.Index               | 203.80 GiB |   22732692 |
| Ancient store (State) | Account.Data                | 141.52 GiB |   22732692 |
| Ancient store (State) | Storage.Data                | 50.38 GiB  |   22732692 |
| Ancient store (State) | History.Meta                | 1.67 GiB   |   22732692 |
| Ancient store (State) | Account.Index               | 141.45 GiB |   22732692 |
+-----------------------+-----------------------------+------------+------------+
|                                    TOTAL            |  2.06 TIB  |            |
+-----------------------+-----------------------------+------------+------------+