db: batch write block a single k/v write

[leowkong]

I’m using blockchain system similar like Ethereum fullnode to perform block syncing. The underlying storage engine is Pebble. While investigating performance issues, I encountered the following situation:

A main thread periodically writes a key-value pair where the key is 32 bytes and the value is 8 bytes. Under normal circumstances, this write operation complete within microseconds. At the same time, there's a background thread that periodically performs a batch write of around 256MB and 826535 keys. However, I noticed that if the batch write is executing by the backgroud thread when the main thread attempts to write a single key, the latency for this single key write can increase dramatically to around 800ms.

Some relevant details about the environment: 1）Disk bandwidth: 8 GB/s OPS: 100,000+ 2）pebble versin : github.com/cockroachdb/pebble v1.1.2, Pebble memTable size: 1.8 GB and write options are configured with sync: true

does a small key write being delayed by hundreds of milliseconds during a large batch write match Pebble’s expected behavior under these configurations? How to optimize this problem

[jbowens]

Hi @leowkong, batches are written sequentially to the write-ahead log. A small write that gets queued behind the large write will not be completed until both writes have been synced durably to disk. For large writes like a 256MB batch, I recommend using the ingestion API: https://pkg.go.dev/github.com/cockroachdb/pebble#DB.IngestWithStats

You can write the keys that you wish to persist to a sstable on the same filesystem as the Pebble database. Then you pass the path to the sstable to Ingest. This allows the I/O of the large write to occur outside the Pebble commit pipeline without blocking small writes. It also reduces write amplification and should generally translate to less write bandwidth from Pebble.

[leowkong]

@jbowens thanks . And I wrote a test case to simulate this scenario and tuned the parameters of pebble's memtable. I found that when the memtable configuration was adjusted from memTableSize = 1.7G size and MemTableStopWritesThreshold = 2 to memTableSize = 500M size and MemTableStopWritesThreshold = 4 or 6, this phenomenon would not occur. It seems that reducing the memTable and increasing the memtableStopWriteThreshold can allow a certain amount of memtable to accumulate and allows writes to be flushed more smoothly which help reduce write stall.

[leowkong]

similar issue:ethereum/go-ethereum#31970. @jbowens If it is caused by write-ahead log, I think reducing the memTable and increasing the memtableStopWriteThreshold will not improve the the write stall . So the key reson is too many memtables accumulate and waited to be flushed ?

[jbowens]

Yeah, MemTableStopWritesThreshold = 2 does not give a lot of buffer. If the flush 1 memtable doesn't finish before the second memtable completes, you'll have a write stall. It's recommended to use a value ≥4 for MemTableStopWritesThreshold, and update your MemTableSize accordingly.