Quorum queue segment consume all disc space

[PaniniHode]

Describe the bug

Facing the exact same issue as in this discussions/bugs:
#6447
rabbitmq/discussions#164
#5258

In some random moment(Can't find the pattern) queue start to grown until disk is full(Or reach the disk watermark), one queue can consume 35-70GB from nothing, rps is low(About 100-1000 rps), total stored messages less then 10k and consumers are in reasonable amounts ~10-20. It happened about 5 times in two month.

server version is 3.13.6 official alpine based docker image.

What strange is, reduction(In runtime queue metrics) shows huge rps, always 5+ millions and it always goes down, when node is reach storage quota

Reproduction steps

As i said earlier, we can't find pattern, it's just happened and ruined one of the cluster nodes

Expected behavior

Normal disk usage :)

Additional context

No response

[mkuratczyk]

Would it be possible to share the queue folder with all those segment files? That should allow us to understand what special sequence of events leads to this. You can contact us directly on slack/discord to avoid making them public.

[PaniniHode]

Would it be possible to share the queue folder with all those segment files? That should allow us to understand what special sequence of events leads to this. You can contact us directly on slack/discord to avoid making them public.

I have already deleted this queues, because node was down, but in the next time i think i can do that, but it's about 40GB, you need all of that or something specific?

[kjnilsson]

Did the data grow on only one node or on all nodes that the quorum queue spans?

[PaniniHode]

On all queue members

[kjnilsson]

If that happens again please capture the output of rabbitmq-queues quorum_status for the given queue.

then try to restart all consumers for the queue such that their connections are terminated and re-established.

Please also share the full configuration that is in place for the quorum queue, e.g. what delivery-limit, overflow and such settings are in place.

And if possible share segment data from one of the nodes.

[mkuratczyk]

Probably the oldest segment file will be the most interesting but please copy everything just in case we need more than that.

[PaniniHode]

Probably the oldest segment file will be the most interesting but please copy everything just in case we need more than that.

Copy that

[michaelklishin]

We don't have any specifics to work with => this belongs to a discussion @PaniniHode

[PaniniHode]

In some reasons github don't send me any notification, sorry for the duplicate

[Eyald6]

We've experienced something similar, quorum_status shows that the Snapshot Index is much lower than the Commit Index:

│ Node Name          │ Raft State │ Log Index │ Commit Index │ Snapshot Index │ Term │ Machine Version │
│ rabbitmq@... │ follower   │ 90609276  │ 90609276     │ 11872694       │ 2    │ 3               │
├────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbitmq@... │ follower   │ 90609276  │ 90609276     │ 11872694       │ 2    │ 3               │
├────────────────────┼────────────┼───────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbitmq@... │ leader     │ 90609276  │ 90609276     │ 11872694       │ 2    │ 3

[michaelklishin]

@Eyald6 if message 1 is not acknowledged, then the quorum queue that contains/-ed those messages will not be able to compact its Raft log and reclaim the disk space for subsequent messages, even if they have been delivered and then acknowledged.

The oldest unacknowledged message in a quorum queue will be the point up to which the log can be compacted. This is mentioned in the Quorum Queues guide.

This generally does not apply for streams and Khepri because they have very different operational characteristics: streams have non-destructive consumers, and Khepri does not have external clients that can potentially hog a delivery and not acknowledge it.

[Eyald6]

I see, an oldest_unacknowledged_message would be suffice in that case, to monitor these cases (this could also happen when a message is nacked intentionally, in which case the acknowledgement timeout will not be triggered)

Thank you for clarifying 👑

[michaelklishin]

I have shared this suggestion with the team, we will consider it because its usefulness is fairly clear. In fact, even if QQs supported parallel (to consumer activity) compaction today.

[kjnilsson]

An this is why you should use a delivery limit and why we have set a default for delivery limit in 4.0.

https://www.rabbitmq.com/docs/quorum-queues#poison-message-handling

https://www.rabbitmq.com/blog/2024/08/28/quorum-queues-in-4.0#delivery-limit-is-now-20-by-default