growing segment files in quorum queues

[ptitvert]

Hello,
I have no idea where to ask this question, so I thought that maybe you can help us, or at least give us an hint.

We have implemented in a rabbitmq cluster (of one node for the moment) with a lot of quorum queues. Our problem is that the segment files are simply growing, growing, growing and never been cleaned up.
We get to the point that the filesystem is full, and it seems that even by adding gigs and gigs of disk space, rabbitmq is never satisfied. So periodically we just shut it down, delete the segement files, when we know that there are no waiting messages, and restart rabbitmq-server. It works, but it is a pain, and we are not sure that this is good and without risks.

We have the same behavior in a plain rabbitmq docker image, without any customization.

Maybe something is wrong on our side with the development, but for the moment, we have found nothing... and the documentation is quite extensive, and quite good, and at the same time on the subject of segment files... well we find nothing really helping in our case.

Again, would it be possible to give us some hints on where to look, and what to do in our case?

Regards,
Alessandro

[lukebakken]

Please start by providing basic information we need to help:

• RabbitMQ and Erlang versions
• Log files from all nodes

[kjnilsson]

Which version are you on? The latest version of RabbitMQ has some changes to clean up segments files more regularly in cases where queues always keep some degree of backlog. The slight downside here is that you may need to delete and re-create the queues to ensure the changes take effect.
Do you do repeated re-queues in any of your applications? What is the average size of the messages you put through?

[kjnilsson]

Are the segments spread over multiple queues (directories) or do they tend to gather in particular ones?

[kjnilsson]

Right ok so it seems the change I am referring to hasn't been released yet - it will be in 3.8.10 - here is the latest beta https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.10-beta.1

[kjnilsson]

That said if a queue completely empties occasionally it should take snapshot and delete the segments anyway. If this doesn't happen could you try issuing a purge command (just to keep you going) when the queue does not have any messages in it? That may trigger a snapshot.

[michaelklishin]

What's the point of using quorum queues if you only have one node and you are willing to delete the backlog?

[ptitvert]

Sorry, yes I forgot to put the minimum of informations:
RabbitMQ 3.8.9
Erlang 23.1.2

We don't delete the queues at all, we keep them alive as long as possible (month/years if possible :-P)
The segments are in some queues, not all. The one that have the most activities.

[ptitvert]

I wasn't the one having chosen the technology, I am just the one trying to fix the problem...
Why only one node? like always, costs, costs, costs...
the idea, from what I've understood, is that quorum queue are persistent, compared to others. And that's why it was so implemented.

[michaelklishin]

Quorum queues primarily make sense in multi-node clusters. They don't offer anything particularly appealing for single-node environments and they have limitations and differences in operational behavior from classic queues. Classic durable queues are also persistent.

One very likely reason for this behavior is that one of your applications does not acknowledge some of its deliveries. This means that the queue has to "hold on to" data up to a certain point in the past. For applications (or instances) that do not do this, no backlog is accumulated. Only application-level metrics or a long enough traffic capture can prove this hypothesis right or wrong.

Or you can simply switch to a durable classic queue and sidestep the problem entirely. You are deleting the backlog anyway, which means you can afford to delete the queue and re-declare it as classic.

[michaelklishin]

One reason why an application can fail to acknowledge only a certain delivery is poor exception handling in consumers: an exception occurs (e.g. a dependency of some kind of not available at the moment), the delivery is never acknowledged or requeued, it just "sits there" and the quorum queue it is in has to hold on to it and newer log entries until it is acknowledged and can be marked for deletion. Application-level logging of exceptions (both handled and unhandled) and metrics can help detect such conditions. What would be the correct thing to do, requeueing or a negative acknowledgement, is an application-specific choice. But holding on to a delivery without acknowledging is never the right thing to do.

[kjnilsson]

I am pretty certain that 3.8.10, once available will address the segment file log growth. Prior versions have some internal defaults that were too targeted towards small message, high-throughput scenarios.

[kjnilsson]

@ptitvert is there any chance you could try the 3.8.10 beta linked above with a test workload to see if it addresses your issue?

[ptitvert]

Hi, I will try to do it until the end of the week.
I'll keep you inform