stuck terminating: check_if_node_is_quorum_critical false positive

[awoimbee]

Describe the bug

I had a pod stuck in terminating state because of rabbitmq-diagnostics check_if_node_is_quorum_critical telling me 4 queues are critical when they seem to have 1 leader + 2 followers each.
It can also happen that check_if_node_is_quorum_critical gets stuck if a queue just doesn't have enough followers as it's not growing the queues following.
And there are no alerts sent if the process takes time or even times out.

To Reproduce

This only happens on my big cluster (>300 queues, a small amount of churn).

Expected behavior

• No false positives
• The preStop is not passive (grows the qeues following)
• Alerts are sent

Version and environment information

• RabbitMQ: 4.1.1
• RabbitMQ Cluster Operator: 2.15.0
• Kubernetes: 1.32.5
• Cloud provider or hardware configuration: AWS EKS

[awoimbee]

Example of false positive:

rabbitmq 2025-07-04 08:39:50.970816+00:00 [info] <0.777151.0> Waiting for 4 queues and streams to have quorum+1 replicas online. You can list them with `rabbitmq-diagnostics check_if_node_is_quorum_critical`  

I have no name!@rabbitmq-server-1:/$ rabbitmq-diagnostics check_if_node_is_quorum_critical
Checking if node rabbit@rabbitmq-server-1.rabbitmq-nodes.rabbitmq-cluster is critical for quorum of any queues/streams ...
queue 'session_01976879-5e09-74c1-b93d-56c56ef6f5c9<redacted>' in vhost '/' will become unavailable if node rabbit@rabbitmq-server-1.rabbitmq-nodes.rabbitmq-cluster stops
queue 'session_0196e9f2-04e7-7311-a21b-050ac6bd10e0<redacted>' in vhost '/' will become unavailable if node rabbit@rabbitmq-server-1.rabbitmq-nodes.rabbitmq-cluster stops
queue 'session_0197687a-402a-756d-b138-96349372a1b2<redacted>' in vhost '/' will become unavailable if node rabbit@rabbitmq-server-1.rabbitmq-nodes.rabbitmq-cluster stops
queue 'session_0197684e-238c-7a8b-bd66-294909c25098<redacted>' in vhost '/' will become unavailable if node rabbit@rabbitmq-server-1.rabbitmq-nodes.rabbitmq-cluster stops

[Zerpet]

I can't see how this command could return a false positive. All it does is listing all nodes in a queue, and filters out the nodes that are not followers nor leader:

https://github.com/rabbitmq/rabbitmq-server/blob/0b3ec492b6cf9ccadb9fffd98b5feaed3e355050/deps/rabbit/src/rabbit_quorum_queue.erl#L524-L541

Assuming the issue is still happening, what do you get for those queues with the command rabbitmq-queues quroum_status <queue-name> ?
You can run this command from any node.

[awoimbee]

Indeed, these queues all have a node in noproc raft state:

I have no name!@rabbitmq-server-0:/$ rabbitmq-queues quorum_status session_01976879-5e09-74c1-b93d-56c56ef6f5c9<redacted>
Status of quorum queue session_01976879-5e09-74c1-b93d-56c56ef6f5c9<redacted> on node rabbit@rabbitmq-server-0.rabbitmq-nodes.rabbitmq-cluster ...
Error:
{{:case_clause, %{state: :noproc, membership: :unknown, machine_version: 7}}, [{:rabbit_quorum_queue, :"-status/2-lc$^0/1-0-", 2, [file: ~c"src/rabbit_quorum_queue.erl", line: 1251]}, {:rabbit_quorum_queue, :status, 2, []}]}
I have no name!@rabbitmq-server-0:/$ rabbitmq-queues quorum_status session_0196e9f2-04e7-7311-a21b-050ac6bd10e0<redacted>
Status of quorum queue session_0196e9f2-04e7-7311-a21b-050ac6bd10e0<redacted> on node rabbit@rabbitmq-server-0.rabbitmq-nodes.rabbitmq-cluster ...
┌──────────────────────────────────────────────────────────┬────────────┬────────────┬────────────────┬──────────────┬──────────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                                                │ Raft State │ Membership │ Last Log Index │ Last Written │ Last Applied │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├──────────────────────────────────────────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-server-0.rabbitmq-nodes.rabbitmq-cluster │ follower   │ voter      │ 0              │ 0            │ 0            │ 0            │ -1             │ 0    │ 7               │
├──────────────────────────────────────────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-server-1.rabbitmq-nodes.rabbitmq-cluster │ follower   │ voter      │ 0              │ 0            │ 0            │ 0            │ -1             │ 0    │ 7               │
├──────────────────────────────────────────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq-cluster │ noproc     │ unknown    │ 9              │ 9            │ 6            │ 8            │ -1             │ 1    │ 7               │
└──────────────────────────────────────────────────────────┴────────────┴────────────┴────────────────┴──────────────┴──────────────┴──────────────┴────────────────┴──────┴─────────────────┘
I have no name!@rabbitmq-server-0:/$ rabbitmq-queues quorum_status session_0197687a-402a-756d-b138-96349372a1b2<redacted>
Status of quorum queue session_0197687a-402a-756d-b138-96349372a1b2<redacted> on node rabbit@rabbitmq-server-0.rabbitmq-nodes.rabbitmq-cluster ...
┌──────────────────────────────────────────────────────────┬────────────┬────────────┬────────────────┬──────────────┬──────────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                                                │ Raft State │ Membership │ Last Log Index │ Last Written │ Last Applied │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├──────────────────────────────────────────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-server-0.rabbitmq-nodes.rabbitmq-cluster │ follower   │ voter      │ 0              │ 0            │ 0            │ 0            │ -1             │ 0    │ 7               │
├──────────────────────────────────────────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-server-1.rabbitmq-nodes.rabbitmq-cluster │ follower   │ voter      │ 0              │ 0            │ 0            │ 0            │ -1             │ 0    │ 7               │
├──────────────────────────────────────────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq-cluster │ noproc     │ unknown    │ 65             │ 65           │ 22           │ 61           │ -1             │ 2    │ 7               │
└──────────────────────────────────────────────────────────┴────────────┴────────────┴────────────────┴──────────────┴──────────────┴──────────────┴────────────────┴──────┴─────────────────┘
I have no name!@rabbitmq-server-0:/$ rabbitmq-queues quorum_status session_0197684e-238c-7a8b-bd66-294909c25098<redacted>
Status of quorum queue session_0197684e-238c-7a8b-bd66-294909c25098<redacted> on node rabbit@rabbitmq-server-0.rabbitmq-nodes.rabbitmq-cluster ...
┌──────────────────────────────────────────────────────────┬────────────┬────────────┬────────────────┬──────────────┬──────────────┬──────────────┬────────────────┬──────┬─────────────────┐
│ Node Name                                                │ Raft State │ Membership │ Last Log Index │ Last Written │ Last Applied │ Commit Index │ Snapshot Index │ Term │ Machine Version │
├──────────────────────────────────────────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-server-0.rabbitmq-nodes.rabbitmq-cluster │ noproc     │ unknown    │ 7              │ 7            │ 6            │ 7            │ -1             │ 1    │ 7               │
├──────────────────────────────────────────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-server-1.rabbitmq-nodes.rabbitmq-cluster │ follower   │ voter      │ 0              │ 0            │ 0            │ 0            │ -1             │ 0    │ 7               │
├──────────────────────────────────────────────────────────┼────────────┼────────────┼────────────────┼──────────────┼──────────────┼──────────────┼────────────────┼──────┼─────────────────┤
│ rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq-cluster │ follower   │ voter      │ 0              │ 0            │ 0            │ 0            │ -1             │ 0    │ 7               │
└──────────────────────────────────────────────────────────┴────────────┴────────────┴────────────────┴──────────────┴──────────────┴──────────────┴────────────────┴──────┴─────────────────┘

No idea how I can fix this state, I unsuccessfully tried this:

I have no name!@rabbitmq-server-0:/$ rabbitmq-queues delete_member session_0197687a-402a-756d-b138-96349372a1b2<redacted> rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq-cluster
Removing a replica of queue session_0197687a-402a-756d-b138-96349372a1b2<redacted> on node rabbit@rabbitmq-server-2.rabbitmq-nodes.rabbitmq-cluster...
Error:
{:no_more_servers_to_try, [timeout: {:"%2F_session_0197687a-402a-756d-b138-96349372a1b2<redacted>", :"rabbit@rabbitmq-server-1.rabbitmq-nodes.rabbitmq-cluster"}, timeout: {:"%2F_session_0197687a-402a-756d-b138-96349372a1b2<redacted>", :"rabbit@rabbitmq-server-0.rabbitmq-nodes.rabbitmq-cluster"}, error: :noproc]}

Relates to rabbitmq/rabbitmq-server#10423

[mkuratczyk]

if there are only two running/healthy members, then check_if_node_is_quorum_critical is doing exactly what it was designed to do I think - it prevents you from stopping one of them, which would render the queue unavailable. I think the real question here is how you ended up in this situation - why is there no process on one of the nodes, why they report inconsistent metrics (different term, log index), etc. It seems like these queues never started properly in the first place.

1. Can you share full logs from all nodes?
2. Can you enable debug logging and see if anything new is logged about these queues?
3. Can you tell us anything more about how the queues were declared, whether you grew them from 1 node to 3 later on or anything like that? Any chance these queues were declared when there was only 1 node in the cluster? (eg. when the first node started, but before the other two did)?

[awoimbee]

1. I would prefer to avoid exporting all logs with all labels, the TL;DR is catastrophic failure:
   • rabbitmq-0: Mnesia('rabbit@rabbitmq-server-0.rabbitmq-nodes.rabbitmq-cluster'): ** WARNING ** Mnesia is overloaded: {dump_log,time_threshold}
   • rabbitmq-0: Non-AMQP exit reason '{{case_clause, [...] timeout
   • rabbitmq-1: Could not delete quorum 'queue 'session_0197684e-238c-7a8b-bd66-294909c25098<redacted>' in vhost '/'', not enough nodes online to reach a quorum
   • rabbitmq-1: Non-AMQP exit reason '{{exception,partition_parallel_timeout},
   • rabbitmq-2: Non-AMQP exit reason '{{exception,partition_parallel_timeout},
   • ** Reason for termination = exit:{'EXIT', [...]
2. Sorry, I deleted them
3. I didn't grow them, queues are defined with:

   BroadwayRabbitMQ.Producer ->
       {mod,
        opts
        |> Keyword.put(:name, "mycontroller-broadway")
        |> Keyword.put(:queue, queue)
        |> Keyword.put_new(:connection, rmq_connection_conf)
        |> Keyword.put_new(:on_failure, :reject)
        |> Keyword.put_new(:qos, prefetch_count: 50)
        |> Keyword.put_new(:declare,
          durable: true,
          arguments: [{"x-queue-type", :longstr, "quorum"}]
        )}

BTW I'm using cluster_partition_handling = autoheal.