[Questions] Cannot upgrade from v3.13 to v4.0 "bad_config_entry_decoder,missing_passphrase" #14233
Replies: 1 comment 4 replies
-
That error means that a configuration file is using an encrypted secret, and when RabbitMQ starts, the passphrase is not being provided. Since you have one environment that upgraded successfully, and one that didn't, you must very carefully determine what is different between the two environments. I would start by examining / diffing the configuration in each cluster's running containers. I'm not 100% certain on k8s, but RabbitMQ configs usually live in |
Beta Was this translation helpful? Give feedback.
-
|
I compared the configuration between the Dev and Test cluster but I couldn't find a meaningfull difference. RabbitMQ on k8s uses a config directory I also compared the kubernetes pod configurations and mounted secrets, environment variables and storage volumes. I then used the diagnostics tool to inspect the environment configuration in rabbitMQ You can also see in the So I don't understand why it's suddenly a problem on the Test cluster? I also did a full comparison of I also tried upgrading from v3.13 to v4.1 directly since this is supported but that gave the same Do you have any further guidance for me as to what I can try to solve this? Can I just start providing my own passphrase for a cluster whose config was previously not encrypted? |
Beta Was this translation helpful? Give feedback.
-
confirms that the passphrase for encrypted value decoding is not configured. We will not suggest anything more for |
Beta Was this translation helpful? Give feedback.
-
|
I am not asking for 3.13.x. The error on startup happens on a supported 4.x version that I am trying to upgrade too. And my currently supported 4.1 dev cluster that I did upgrade also has the passphrase for encrypted value decoding not configured. So why does it work on A but not on B? Is there a way to tell rabbitmq to ignore it? And can I add a passphrase later to a previously unencrypted version or will that cause problems? |
Beta Was this translation helpful? Give feedback.
-
|
Not using decrypt({encrypted, _} = EncValue,
{Cipher, Hash, Iterations, PassPhrase} = Algo) ->
{rabbit_pbe:decrypt_term(Cipher, Hash, Iterations, PassPhrase, EncValue),
Algo};
decrypt({encrypted, _} = EncValue,
ConfigEntryDecoder)
when is_list(ConfigEntryDecoder) ->
Algo = config_entry_decoder_to_algo(ConfigEntryDecoder),
decrypt(EncValue, Algo);
decrypt(List, Algo) when is_list(List) ->
decrypt_list(List, Algo, []);
decrypt(Value, Algo) ->
{Value, Algo}.In |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Community Support Policy
RabbitMQ version used
4.0.9
Erlang version used
27.3.x
Operating system (distribution) used
Linux
How is RabbitMQ deployed?
Kubernetes Operator(s) from Team RabbitMQ
rabbitmq-diagnostics status output
See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics
I couldn't run this on node 3 since it keeps crashin so I ran these commands on node 1.
rabbitmq-diagnostics status
rabbitmq-diagnostics cluster_status
Logs from node 1 (with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
Logs from node 2 (if applicable, with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
Logs from node 3 (if applicable, with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
=INFO REPORT==== 15-Jul-2025::07:18:23.665360 === alarm_handler: {set,{system_memory_high_watermark,[]}} BOOT FAILED =========== Exception during startup: throw:{bad_config_entry_decoder,missing_passphrase} rabbit_prelaunch_conf:decrypt_app/3, line 514 rabbit_prelaunch_conf:decrypt_config/2, line 494 rabbit_prelaunch_conf:setup/1, line 59 rabbit_prelaunch:do_run/0, line 113 rabbit_prelaunch:run_prelaunch_first_phase/0, line 32 supervisor:do_start_child_i/3, line 959 supervisor:do_start_child/3, line 945 supervisor:-start_children/2-fun-0-/3, line 929 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> BOOT FAILED 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> =========== 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> Exception during startup: 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> throw:{bad_config_entry_decoder,missing_passphrase} 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> rabbit_prelaunch_conf:decrypt_app/3, line 514 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> rabbit_prelaunch_conf:decrypt_config/2, line 494 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> rabbit_prelaunch_conf:setup/1, line 59 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> rabbit_prelaunch:do_run/0, line 113 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> rabbit_prelaunch:run_prelaunch_first_phase/0, line 32 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> supervisor:do_start_child_i/3, line 959 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> supervisor:do_start_child/3, line 945 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> supervisor:-start_children/2-fun-0-/3, line 929 2025-07-15 07:18:24.599845+00:00 [error] <0.153.0> 2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> supervisor: {local,rabbit_prelaunch_sup} 2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> errorContext: start_error 2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> reason: {bad_config_entry_decoder,missing_passphrase} 2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> offender: [{pid,undefined}, 2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> {id,prelaunch}, 2025-07-15 07:18:25.614599+00:00 [error[] <0.153.0> {mfargs,{rabbit_prelaunch,run_prelaunch_first_phase,]}}, 2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> {restart_type,transient}, 2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> {significant,false}, 2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> {shutdown,5000}, 2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> {child_type,worker}] 2025-07-15 07:18:25.614599+00:00 [error] <0.153.0> 2025-07-15 07:18:25.615180+00:00 [notice[] <0.45.0> Application rabbitmq_prelaunch exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{bad_config_entry_decoder,missing_passphrase}}},{rabbit_prelaunch_app,start,[normal,[]]}} {exit,terminating,[{application_controller,call,2,[{file,"application_controller.erl"},{line,511}]},{application,'-ensure_all_started/3-lc$^0/1-0-',1,[{file,"application.erl"},{line,367}]},{application,ensure_all_started,3,[{file,"application.erl"},{line,367}]},{rabbit,'-start_it/1-fun-0-',1,[{file,"rabbit.erl"},{line,423}]},{timer,tc,2,[{file,"timer.erl"},{line,595}]},{rabbit,start_it,1,[{file,"rabbit.erl"},{line,421}]},{init,start_it,1,[]},{init,start_em,1,]}]} 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> crasher: 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> initial call: application_master:init/3 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> pid: <0.151.0> 2025-07-15 07:18:25.615961+00:00 [error[] <0.151.0> registered_name: ] 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> exception exit: {{shutdown, 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> {failed_to_start_child,prelaunch, 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> {bad_config_entry_decoder,missing_passphrase}}}, 2025-07-15 07:18:25.615961+00:00 [error[] <0.151.0> {rabbit_prelaunch_app,start,[normal,[]]}} 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> in function application_master:init/3 (application_master.erl, line 143) 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> ancestors: [application_controller,<0.10.0>] 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> message_queue_len: 1 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> messages: [{'EXIT',<0.152.0>,normal}] 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> links: [<0.45.0>] 2025-07-15 07:18:25.615961+00:00 [error[] <0.151.0> dictionary: ] 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> trap_exit: true 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> status: running 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> heap_size: 233 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> stack_size: 29 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> reductions: 68 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> neighbours: 2025-07-15 07:18:25.615961+00:00 [error] <0.151.0> 2025-07-15 07:18:25.629024+00:00 [notice] <0.84.0> alarm_handler: {clear,system_memory_high_watermark} Kernel pid terminated (application_controller) ("{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{bad_config_entry_decoder,missing_passphrase}}},{rabbit_prelaunch_app,start,[normal,[]]}}}") stream closed EOF for kaspar123-test/rabbit-mq-server-2 (rabbitmq)rabbitmq.conf
See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location
10-defaults.conf
10-operatorDefaults.conf
11-default_user.conf
90-userDefinedConfiguration.conf
Steps to deploy RabbitMQ cluster
Our RabbitMQ cluster is controlled by A rabbitMQ operator on kubernetes. We specify the version of RabbitMQ we want and the required plugins. Then the operator will perform a Rolling upgrade of each Rabbitmq node until each node in the cluster has the same version. The operator config is a yaml file that is applied to the AKS cluster by a Azure Release pipeline. The pipeline uses the kubectl apply command.
Steps to reproduce the behavior in question
Currrently the cluster is running v3.13. When I upgrade the cluster operator config to use Rabbitmq:4.0-management-alpine, the first node is upgraded but fails to start due to a
throw:{bad_config_entry_decoder,missing_passphrase}Error.advanced.config
N/A
Application code
N/A
Kubernetes deployment file
What problem are you trying to solve?
I am currently running upgrades to update our RabbitMQ clusters on k8s from 3.11 to the latest 4.1.
We have multiple environments dev, test and prod.
I upgraded RabbitMQ on our dev environment from version v3.11 to v4.1 without any issues. Upgrading one version at a time and enabling all stable feature flags in the management UI before proceeding.
Now I am repeating the same steps on our Test environement using the same configuration.
But when I try to upgrade from v3.13 to v4.0 RabbitMQ fails to start with the following error:
throw:{bad_config_entry_decoder,missing_passphrase}(See logs from node 3)I looked around but couldn't find any other user who ran into that error.
When I downgrade the cluster back to v3.13 the failing nodes starts back up as a v3.13 node without any issues. All stable feature flags are enabled in the Management UI:
Can anyone please help?
Beta Was this translation helpful? Give feedback.
All reactions