Scale down

[ChunyiLyu]

Is your feature request related to a problem? Please describe.

Scaling down a rabbitmq cluster does not work at the moment. We should look into why it fails, and what we need to do to support scaling down.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context

• rabbitmqctl forget_cluster_node might be needed before we deleting pods
• we might need to skip preStop hooks
• what do we do with non-HA queues that only exists in nodes that will be deleted?

[coro]

From today's sync-up:

rabbitmq-server might not support scaling down - in which case, should we support it?

I didn't know what 'scaling down doesn't work' meant in this issue, so I dug into it; there are three/four failure modes to scaling down not working:

• If we lose the majority of nodes in a cluster during scaling down (e.g. 5->1), we go into pause minority and the whole cluster goes down
• If any of the nodes are quorum critical or mirror sync critical (judged with rabbitmq-queues command), and this does not self-resolve (e.g. under heavy load / insufficient consumers), the pod will fail to terminate as the preStop hook never completes
• Quorum queues require the terminating node to be removed from the RAFT algorithm through rabbitmq-queues shrink
• Durable, classic queues (i.e. not mirrored/quorum) whose primary node is the node going down will be lost irrecoverably
  • This one is tenuous in my opinion - if you want replication, classic queues aren't going to provide that

[coro]

For reference, the bitnami helm chart does allow for scaling down, but from my testing it does this by recreating the cluster, rather than removing single nodes.

[coro]

@harshac and I played about with the helm chart today. It seems it has a persistent disk so that even when you scale the cluster & it recreates all of the nodes, it still persists any durable queues or persistent messages between iterations of the cluster.

[michaelklishin]

Removing individual nodes would be much less disruptive with the introduction of maintenance mode in 3.8.x. RabbitMQ nodes are never implicitly removed from the cluster: rabbitmqctl forget_cluster_node would indeed be required to permanently remove a node.

[michaelklishin]

RabbitMQ server does support permanent node removal from an existing cluster. Raft-based features such as quorum queues and MQTT client ID tracker require such node to also be explicitly removed from the Raft member list using, see

rabbitmq-queues help delete_member

rabbitmqctl help decommission_mqtt_node

The problematic part with downsizing is that the reduced number of nodes may or may not handle the same load on the system gracefully. A five-node cluster where all nodes run with the default open file handle limit of 1024 can sustain about 5000 connections but a three-node cluster would not be able to do that.

[coro]

It doesn't look like there is any native StatefulSet hook for specifically scaling down. The closest I've found is this issue, which covers our case exactly. It points out that there is no way to gracefully decommission a node in Kubernetes StatefulSets.

We will likely have to implement custom logic in the controller for this. Perhaps we can detect in the controller where the spec.replicas field in the CR is decreased, and use this to set flags on all of the pods in the stateful set, which lead to different preStop hook logic that does the maintenance mode work mentioned above? There would be no way for us to tell which pods would be brought down in this case, but any that do would have this behaviour. I have a suspicion that that would lead to all of the pods running this anyway when they are eventually brought down.

[mkuratczyk]

Indeed, it'd be great to see that as a native feature as described in that issue. For the time being, we could implement what you said which is more or less what we already have when deleting the cluster: https://github.com/rabbitmq/cluster-operator/blob/main/controllers/rabbitmqcluster_controller.go#L471. We can keep the replicas count untouched until the pod/node is successfully deleted/forgotten.

I'm not sure about this part: There would be no way for us to tell which pods would be brought down in this case. I'm pretty sure StatefulSet guarantees that if you scale down, the nodes with the higher index number will be deleted and even they will be deleted one by one. So the process could look something like this:

1. Enable RabbitMQ maintenance mode for the node with the highest index
2. Wait for the node to be drained
3. Label (or in some other way mark) the node with the highest index for deletion
4. Reduce the number of replicas in the StatefulSet
5. The node with the highest index should be terminated quickly (it's empty anyway)
6. We can run the forget/delete node command (unless done as part of node's termination)

[github-actions]

This issue has been marked as stale due to 60 days of inactivity. Stale issues will be closed after a further 30 days of inactivity; please remove the stale label in order to prevent this occurring.

[adthonb]

Do you have a solution to scale down the pod at this moment? I edit a configuration on RabbitMQ instance for a replica from 3 to 1 but nothing happened.