Blobs store refactor

[rklaehn]

Big blobs refactor with a redesign of the store to properly take into account availability bitfields.

The main reasons for the redesign are

• simplify the API and internals for a 1.0 version of blobs that can be supported long term
• support multiprovider downloads, which do make use of the bitfields to compute download plans.

Goals

• get rid of cruft/dependencies that accumulated due to the iroh cli/rpc/bindings use cases
• get rid of the publicly visible LocalPool, S type parameter and GC bits hanging out
• provide a better API for the in-process use case which has become more important since the strategy change
• decoupling the store internals so we can write an optimized linux iroh-blobs version using a true thread per core runtime and io-uring
• performance improvement and preparing for using another database like fjall or even something manual
• make it at least possible to support WASM for the fs store

What stays:

• fs store file format will remain exactly the same for now. You will be able to open an old blobs db with the new store and vice versa -¹
• iroh wire protocol for get
  • new blobs can make request against any program using iroh-blobs or even old all in one iroh since 2023.
  • old blobs can make request against new store

Link to sausage factory:
https://github.com/n0-computer/blobs2

Current state:

• interop with old blobs - requesting
• interop with old blobs - providing
• opening old fs store
• persistent bitfield support
• ability to observe bitfields
• external file support
• import/export using COW copy
• deletion of non-needed files like .sizes4 or .bitfield
• redb ops batching
• temp tags
• GC (gc will be invisible in the API!)
• Cross process RPC
• tests
• proptests
• performance better or equal to old
  • large files (both data and outboard in fs)
  • tiny files (no outboard, data in mem -> inline)
  • medium files (small inline outboard, data on fs)
• api sufficient to write sendme send/recv
• progress API
• api sufficient to write nice sendme without too much code
• wire protocol to observe remote blob (blobs proto observe request)

Footnotes

1. modulo the bitfield stuff of course ↩

[rklaehn]

Idea: for all metadata updates, sync db if the entry affects the file system.

• Small (<= 16 KiB) entries will still be fast
• No complex deletion after the fact for medium to large entries