Support for liveness probe in rabbitmq pods

[hunnygarg-ag]

When deploying rabbitmq cluster backed by EFS volume, sometimes we see it stop working with below CRASH logs

CRASH REPORT Process rabbit_web_dispatch_registry with 0 neighbours exited with reason: {aborted,{node_not_running,'[email protected]'}} in mnesia:abort/1 line 361

Output of rabbitmqctl status inside the pod during this CRASH

rabbitmqctl status
Status of node [email protected] ...
Error:
{:aborted, {:no_exists, :rabbit_vhost}}

Rabbitmq pod remain in running state during this crash and k8s does not restart it due to missing liveness probe.
Can liveness probe support be added to handle such situations.

[mkuratczyk]

Can you please provide full logs? Generally speaking, this kind of issues should be resolved by Erlang - it has the concept of process supervision and restart, so if a RabbitMQ component crashed, it should be restarted. If Erlang crashes, the pod will be restarted without a livenessProble (because it's the main process in the container).

You can add a livenessProbe even now (using StatefulSet overrides), the question is what the probe should monitor. For example a vhost crash can be a good trigger in your case, but not for someone who's running multiple vhosts.