bound the queue

helpfully, jetstream got stuck for a bunch of hours, so there was a lot of firehose to backfill and find out the actual performance limit and how much the queue can grow.

the queue grew to millions before the raspi finally ran out of memory and we died with a failed allocation. i bounded it at 100k to let it finish catching up, and that worked.

2048 is still seems fairly arbitrary, but it's not obviously too wildly high or low.

Author: uniphil

link_aggregator/src/consumer/mod.rs

@@ -42,7 +42,7 @@ pub fn consume(mut store: impl LinkStorage, qsize: Arc<AtomicU32>, fixture: Opti
             thread::spawn(move || consume_jsonl_file(f, sender)),
         )
     } else {
-        let (sender, receiver) = flume::unbounded(); // eek
+        let (sender, receiver) = flume::bounded(2048); // eek
         let cursor = store.get_cursor().unwrap();
         (
             receiver,