Dropping alerts from ruler

[ilantnt]

[ilantnt]

Hi all,
I'm using Thanos as HA for Prometheus for 4 months. Recently I'm facing an issue that my Thanos Ruler doesn't send the alerts to my Alertmanager.
I have these following erros: "alert notifications full" and "dropping alerts".
I have 150 Rules on one Ruler. Has anyone faced this incident before?
Thanks in advance

[yeya24]

Hello, this problem is because the alert queue in ruler is full, so some of your alerts are dropped.
Currently, the capacity of the alert queue is a hardcoded value 10000, it is not configurable. Maybe we can consider adding a flag for it.

A possible solution now is to run 2 (or more) Thanos Ruler. And each has 75 alerting rules. Then I guess the alerts won't be dropped. Could you please give a try? @ilantnt

[robincw-gr]

I have had this problem and today found the bug in the alert queue that causes this. You might notice the gauge metric for alert queue length, thanos_alert_queue_length, is always near the 10,000 limit.

[robincw-gr]

Hitting the 10,000 queue limit is obviously suspicious. Running more instances won't help work around it. This bug can be seen when your alert rules fire lots of alerts that are queued more often than they can be sent in batches of 100. The alert queue has push and pop functions. The bug is the morec channel used by push to signal that alerts have been queued. It has a buffer size of 1. Push can run many times and simply ensures that the channel is full, i.e. has one signal, each time it is done; if it already does then it continues via select default. Meanwhile pop blocks until there is a signal on morec, then pops maxBatchSize alerts (100) off the queue. If it has already run and cleared the channel then it waits even if push has run multiple times to fill the queue beyond 100 beforehand. So the queue length increases when push is called more frequently than pop, empties only when push is called less frequently than pop, and remains steady if no alerts are being pushed.
Work in progress now :)