Recovery from snapshot

[jumaffre]

Resolves #1303

This PR adds support for resuming from a snapshot on recovery, making the whole recovery procedure a lot faster (from 1 minute to 11 seconds in this example):

# Start a liblogging service ...

# Issue 5000 transactions at the same key (i.e. long ledger but tiny state)
$ for ITER in {1..5000}; do curl https://127.212.141.250:38787/app/log/private -H "Content-Type: application/json" --data '{"id":1,"msg":"xyz"}' ...  ; done
...
x-ccf-global-commit: 5304
x-ccf-tx-seqno: 5316
x-ccf-tx-view: 2

# Kill the service. Copy ledger and snapshot directories and network encryption key.

# Recover service only using ledger (i.e. no snapshot): ~1 minute
$ ../start_test_network.sh -p liblogging --recover ...
Python environment successfully setup
[15:02:38.758] Starting 3 CCF nodes...
[15:02:38.758] Recovering network from:
[15:02:38.758]  - Defunct network public encryption key: ./network_enc_pubk.pem
[15:02:38.758]  - Common directory: ./workspace/test_network_common/
[15:02:38.758]  - Ledger: 0.ledger/
[15:02:38.758] No available snapshot to recover from. Entire transaction history will be replayed. <<<<< SLOW!
[15:03:37.505] Started CCF network with the following nodes:
[15:03:37.505]   Node [ 3] = 127.160.143.234:33661
[15:03:37.505]   Node [ 4] = 127.153.80.188:44753
[15:03:37.505]   Node [ 5] = 127.81.23.35:39129

# Same recovery with snapshots: ~11 seconds
$ ../start_test_network.sh -p liblogging --recover ... --snapshot-dir snapshots
[15:07:08.529] Starting 3 CCF nodes...
[15:07:08.529] Recovering network from:
[15:07:08.530]  - Defunct network public encryption key: ./network_enc_pubk.pem
[15:07:08.530]  - Common directory: ./workspace/test_network_common/
[15:07:08.530]  - Ledger: 0.ledger/
[15:07:08.530]  - Snapshots: snapshots/    <<<<< FAST!
[15:07:19.469] Started CCF network with the following nodes:
[15:07:19.469]   Node [ 3] = 127.185.213.5:45741
[15:07:19.469]   Node [ 4] = 127.116.172.203:35685
[15:07:19.469]   Node [ 5] = 127.187.176.189:44623

Implementation details:

1. The snapshot directory can now be passed to cchost on recovery (just like when joining a new network).
2. On recovery, the public part of the snapshot is first applied. Subsequent entries in the ledger are read until the end of the ledger is reached (speed-up part I.). The snapshot is kept in enclave memory for now.
3. Once the recovery shares have been submitted, the ledger secrets from the ones contained in the snapshot till the current ones are decrypted. The private part of the snapshot is applied and subsequent ledger entries are also read (speed-up part II.). The snapshot is then discarded.

Notes:

• New recovery_snapshot_test e2e test with corresponding changes to the Python infra.
• start_test_network.sh can recover from a snapshot.
• Snapshots are not generated when the service is in public mode. This is because such snapshots would have to be generated without direct integrity protection (because the ledger key is not recoverable until the public recovery is finished). We may want to revisit this in the future if this is an issue.
• Getting further snapshots generated by the recovered service at the right intervals requires some juggling of indices, etc. I believe this is working now but will be systematically tested when End-to-end testing of ledger between nodes #1608 is worked on.
• If a node has recovered from a snapshot, all other nodes should join from a snapshot that is at least as recent as that recovery snapshot. This is because the entire history of ledger secrets is contained in the ledger. This isn't yet automatically checked on join but will be addressed in Support for historical queries after recovery from snapshot #1648.

[ghost]

recovery_from_snapshot@13241 aka 20200929.10 vs master ewma over 50 builds from 12699 to 13236

[ashamis]

minor: can you simplify the code by passing view_history by reference (into the funtion) as you always want to return the view history and then just deserializing into view_history?

[jumaffre]

I don't think we want to pass a reference here as we don't necessarily want to deserialise the view history. If the source KV store doesn't have a consensus, then the view history won't be included in the snapshot. This is useful to test snapshots on the KV in isolation of other components.

[achamayou]

Maybe it's time for 857 then.