Stream (Osiris) writer continuously restarts and logs errors #9998

plewam · 2023-11-28T07:51:01Z

plewam
Nov 28, 2023

Hi All,

I have been a long time user of classic queues within RMQ. Recently I started using RMQ streams.
RMQ Version: 3.13-RC2
Erlang Version:
Plattform: Windows Server 2019

My streams are configured to hold each dataset for 21 days. As of now this has resulted in a combined stream size of around a billion messages.
As the stream size grows I encounter multiple problems.

1. Problem - Crazy log writing of the below error


2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>   crasher:
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     initial call: osiris_writer:init/1
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     pid: <0.1700285.0>
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     registered_name: []
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     exception exit: {{case_clause,
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                          {ok,<<113,206,220,148,144,243,65,101,226,3,202,4,
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                                156,141,47,147,0,1,62,76,49,222,115,50,97,107,
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                                153,91,38,49,228,100,162,201,128,147,9,82,188,
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                                198,244,173,199,244,101,4,76,148>>}},
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                      [{osiris_log,recover_tracking,3,
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                           [{file,"src/osiris_log.erl"},{line,2629}]},
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                       {osiris_writer,handle_continue,2,
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                           [{file,"src/osiris_writer.erl"},{line,196}]},
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                       {gen_batch_server,handle_continue,2,
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                           [{file,"src/gen_batch_server.erl"},{line,413}]},
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                       {gen_batch_server,init_it,6,
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                           [{file,"src/gen_batch_server.erl"},{line,163}]},
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                       {proc_lib,init_p_do_apply,3,
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>                           [{file,"proc_lib.erl"},{line,241}]}]}
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>       in function  gen_batch_server:handle_continue/2 (src/gen_batch_server.erl, line 423)
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>       in call from gen_batch_server:init_it/6 (src/gen_batch_server.erl, line 163)
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     ancestors: [osiris_server_sup,osiris_sup,<0.216.0>]
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     message_queue_len: 0
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     messages: []
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     links: [<0.220.0>]
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     dictionary: []
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     trap_exit: true
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     status: running
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     heap_size: 610
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     stack_size: 28
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>     reductions: 51218
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0>   neighbours:
2023-11-28 07:13:53.341000+00:00 [error] <0.1700285.0> 
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>     supervisor: {local,osiris_server_sup}
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>     errorContext: child_terminated
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>     reason: {{case_clause,{ok,<<113,206,220,148,144,243,65,101,226,3,202,4,
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                                 156,141,47,147,0,1,62,76,49,222,115,50,97,
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                                 107,153,91,38,49,228,100,162,201,128,147,9,
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                                 82,188,198,244,173,199,244,101,4,76,148>>}},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>              [{osiris_log,recover_tracking,3,
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                           [{file,"src/osiris_log.erl"},{line,2629}]},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>               {osiris_writer,handle_continue,2,
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                              [{file,"src/osiris_writer.erl"},{line,196}]},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>               {gen_batch_server,handle_continue,2,
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                                 [{file,"src/gen_batch_server.erl"},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                                  {line,413}]},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>               {gen_batch_server,init_it,6,
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                                 [{file,"src/gen_batch_server.erl"},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                                  {line,163}]},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>               {proc_lib,init_p_do_apply,3,
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                         [{file,"proc_lib.erl"},{line,241}]}]}
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>     offender: [{pid,<0.1700285.0>},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                {id,<<"__Stream_DataAgentEntityStatisticsScaleEvent_16996012557283323">>},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                {mfargs,{osiris_writer,start_link,undefined}},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                {restart_type,temporary},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                {significant,false},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                {shutdown,5000},
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>                {child_type,worker}]
2023-11-28 07:13:53.341000+00:00 [error] <0.220.0>

2. Problem - Increase in memory (Other ETS Tables)

3. Problem - Long startup times and continuously high CPU usage
A start of RabbitMQ takes up to 5 minutes until the management interface becomes responsive. Until then I also have connectivity problems. Throughout the whole time the CPU usage is at 100%

All these three problems might be linked to each other. In order to temporarily fix this I have reduced the number of days a message is hold inside the stream to 3 days again. It seems I am stable again now.
Any help would be highly appreciated, especially about Problem 1.

michaelklishin · 2023-11-28T13:06:43Z

michaelklishin
Nov 28, 2023
Maintainer

@plewam can you please share the stream directory that we can use to reproduce this behavior, together with an (edited to not contain any sensitive info) definition file?

36 replies

lukebakken Nov 30, 2023
Maintainer

@plewam please take a second to look and see how I edited your comment to "hide" the log. Otherwise, it takes up a huge amount of screen space.

Another alternative is to upload a file, rather than paste a wall of text.

plewam Nov 30, 2023
Author

@lukebakken Got it thanks, didn't know that

plewam Nov 30, 2023
Author

@kjnilsson So I just did a VM restart and attached the log for you. I immediately have corrupted stream files, See log

VM restart log

2:39.187000+00:00 [info] <0.100.0> RabbitMQ is asked to stop...
2023-11-30 13:02:39.236000+00:00 [info] <0.2139.0> closing AMQP connection <0.2139.0> (127.0.0.1:53032 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.236000+00:00 [info] <0.2458.0> closing AMQP connection <0.2458.0> (127.0.0.1:53075 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.236000+00:00 [info] <0.2452.0> closing AMQP connection <0.2452.0> (127.0.0.1:53077 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.236000+00:00 [info] <0.1831.0> closing AMQP connection <0.1831.0> (127.0.0.1:53037 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.236000+00:00 [info] <0.2032.0> closing AMQP connection <0.2032.0> (127.0.0.1:53025 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.252000+00:00 [info] <0.2464.0> closing AMQP connection <0.2464.0> (127.0.0.1:53079 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.252000+00:00 [info] <0.2136.0> closing AMQP connection <0.2136.0> (127.0.0.1:53029 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.267000+00:00 [info] <0.2339.0> closing AMQP connection <0.2339.0> (127.0.0.1:53034 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.298000+00:00 [info] <0.1925.0> closing AMQP connection <0.1925.0> (127.0.0.1:53023 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.314000+00:00 [info] <0.1875.0> closing AMQP connection <0.1875.0> (127.0.0.1:53022 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.392000+00:00 [info] <0.1834.0> closing AMQP connection <0.1834.0> (127.0.0.1:53038 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.439000+00:00 [info] <0.2455.0> closing AMQP connection <0.2455.0> (127.0.0.1:53074 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.455000+00:00 [info] <0.2342.0> closing AMQP connection <0.2342.0> (127.0.0.1:53035 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0> Stopping RabbitMQ applications and their dependencies in the following order:
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     rabbitmq_management
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     rabbitmq_management_agent
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     rabbitmq_web_dispatch
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     cowboy
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     cowlib
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     rabbitmq_stream
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     rabbit
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     rabbitmq_prelaunch
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     amqp_client
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     rabbit_common
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     rabbitmq_stream_common
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     khepri
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     ra
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     os_mon
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     sysmon_handler
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     osiris
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0>     mnesia
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0> 
2023-11-30 13:02:39.455000+00:00 [info] <0.100.0> Stopping application 'rabbitmq_management'
2023-11-30 13:02:39.470000+00:00 [warning] <0.1606.0> HTTP listener registry could not find context rabbitmq_management_tls
2023-11-30 13:02:39.564000+00:00 [notice] <0.44.0> Application rabbitmq_management exited with reason: stopped
2023-11-30 13:02:39.564000+00:00 [info] <0.100.0> Stopping application 'rabbitmq_management_agent'
2023-11-30 13:02:39.580000+00:00 [notice] <0.44.0> Application rabbitmq_management_agent exited with reason: stopped
2023-11-30 13:02:39.580000+00:00 [info] <0.100.0> Stopping application 'rabbitmq_web_dispatch'
2023-11-30 13:02:39.689000+00:00 [notice] <0.44.0> Application rabbitmq_web_dispatch exited with reason: stopped
2023-11-30 13:02:39.689000+00:00 [info] <0.100.0> Stopping application 'cowboy'
2023-11-30 13:02:39.736000+00:00 [notice] <0.44.0> Application cowboy exited with reason: stopped
2023-11-30 13:02:39.736000+00:00 [info] <0.100.0> Stopping application 'cowlib'
2023-11-30 13:02:39.736000+00:00 [notice] <0.44.0> Application cowlib exited with reason: stopped
2023-11-30 13:02:39.736000+00:00 [info] <0.100.0> Stopping application 'rabbitmq_stream'
2023-11-30 13:02:39.736000+00:00 [info] <0.1709.0> stopped Stream TCP listener on 0.0.0.0:5552
2023-11-30 13:02:39.767000+00:00 [info] <0.2216.0> closing AMQP connection <0.2216.0> (127.0.0.1:53030 -> 127.0.0.1:47490, vhost: '/', user: 'logreader')
2023-11-30 13:02:39.814000+00:00 [notice] <0.44.0> Application rabbitmq_stream exited with reason: stopped
2023-11-30 13:02:39.814000+00:00 [info] <0.100.0> Stopping application 'rabbit'
2023-11-30 13:02:39.892000+00:00 [info] <0.246.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping unregistration.
2023-11-30 13:02:39.892000+00:00 [info] <0.1727.0> stopped TCP listener on 0.0.0.0:47490
2023-11-30 13:02:39.892000+00:00 [error] <0.1730.0> Error on AMQP connection <0.1730.0> (127.0.0.1:53033 -> 127.0.0.1:47490, vhost: '/', user: 'logreader', state: running), channel 0:
2023-11-30 13:02:39.892000+00:00 [error] <0.1730.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-11-30 13:02:39.892000+00:00 [error] <0.2093.0> Error on AMQP connection <0.2093.0> (127.0.0.1:53027 -> 127.0.0.1:47490, vhost: '/', user: 'logreader', state: running), channel 0:
2023-11-30 13:02:39.892000+00:00 [error] <0.2093.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-11-30 13:02:39.892000+00:00 [error] <0.2449.0> Error on AMQP connection <0.2449.0> (127.0.0.1:53076 -> 127.0.0.1:47490, vhost: '/', user: 'logreader', state: running), channel 0:
2023-11-30 13:02:39.892000+00:00 [error] <0.2449.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-11-30 13:02:39.892000+00:00 [error] <0.2461.0> Error on AMQP connection <0.2461.0> (127.0.0.1:53078 -> 127.0.0.1:47490, vhost: '/', user: 'logreader', state: running), channel 0:
2023-11-30 13:02:39.892000+00:00 [error] <0.2461.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-11-30 13:02:39.892000+00:00 [error] <0.2584.0> Error on AMQP connection <0.2584.0> (172.17.12.240:45818 -> 172.26.224.23:47490, vhost: '/', user: 'logreader', state: running), channel 0:
2023-11-30 13:02:39.892000+00:00 [error] <0.2584.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-11-30 13:02:39.892000+00:00 [error] <0.2597.0> Error on AMQP connection <0.2597.0> (127.0.0.1:53081 -> 127.0.0.1:47490, vhost: '/', user: 'logreader', state: running), channel 0:
2023-11-30 13:02:39.892000+00:00 [error] <0.2597.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-11-30 13:02:39.892000+00:00 [error] <0.1979.0> Error on AMQP connection <0.1979.0> (127.0.0.1:53026 -> 127.0.0.1:47490, vhost: '/', user: 'logreader', state: running), channel 0:
2023-11-30 13:02:39.892000+00:00 [error] <0.1979.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-11-30 13:02:39.892000+00:00 [error] <0.2397.0> Error on AMQP connection <0.2397.0> (127.0.0.1:53036 -> 127.0.0.1:47490, vhost: '/', user: 'logreader', state: running), channel 0:
2023-11-30 13:02:39.892000+00:00 [error] <0.2397.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-11-30 13:02:39.892000+00:00 [error] <0.2142.0> Error on AMQP connection <0.2142.0> (127.0.0.1:53031 -> 127.0.0.1:47490, vhost: '/', user: 'logreader', state: running), channel 0:
2023-11-30 13:02:39.892000+00:00 [error] <0.2142.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-11-30 13:02:39.892000+00:00 [error] <0.2133.0> Error on AMQP connection <0.2133.0> (127.0.0.1:53028 -> 127.0.0.1:47490, vhost: '/', user: 'logreader', state: running), channel 0:
2023-11-30 13:02:39.892000+00:00 [error] <0.2133.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2023-11-30 13:02:40.048000+00:00 [warning] <0.2571.0> Channel is stopping with 2 pending publisher confirms
2023-11-30 13:02:40.173000+00:00 [warning] <0.2209.0> Channel is stopping with 8 pending publisher confirms
2023-11-30 13:02:40.173000+00:00 [warning] <0.2557.0> Channel is stopping with 10 pending publisher confirms
2023-11-30 13:02:40.189000+00:00 [warning] <0.2290.0> Channel is stopping with 36 pending publisher confirms
2023-11-30 13:02:40.189000+00:00 [warning] <0.2117.0> Channel is stopping with 106 pending publisher confirms
2023-11-30 13:02:40.189000+00:00 [info] <0.15657.0> Closing all connections in vhost '/' on node 'rabbit@PON-WPAS99' because the vhost is stopping
2023-11-30 13:02:40.642000+00:00 [info] <0.477.0> Stopping message store for directory 'd:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2023-11-30 13:02:40.674000+00:00 [info] <0.477.0> Message store for directory 'd:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2023-11-30 13:02:40.674000+00:00 [info] <0.473.0> Stopping message store for directory 'd:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2023-11-30 13:02:40.689000+00:00 [info] <0.473.0> Message store for directory 'd:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2023-11-30 13:02:40.705000+00:00 [info] <0.415.0> Management plugin: to stop collect_statistics.
2023-11-30 13:02:45.736000+00:00 [notice] <0.44.0> Application rabbit exited with reason: stopped
2023-11-30 13:02:45.736000+00:00 [info] <0.100.0> Stopping application 'rabbitmq_prelaunch'
2023-11-30 13:02:45.752000+00:00 [notice] <0.44.0> Application rabbitmq_prelaunch exited with reason: stopped
2023-11-30 13:02:45.752000+00:00 [info] <0.100.0> Stopping application 'amqp_client'
2023-11-30 13:02:45.767000+00:00 [notice] <0.44.0> Application amqp_client exited with reason: stopped
2023-11-30 13:02:45.767000+00:00 [info] <0.100.0> Stopping application 'rabbit_common'
2023-11-30 13:02:45.767000+00:00 [notice] <0.44.0> Application rabbit_common exited with reason: stopped
2023-11-30 13:02:45.767000+00:00 [info] <0.100.0> Stopping application 'rabbitmq_stream_common'
2023-11-30 13:02:45.767000+00:00 [notice] <0.44.0> Application rabbitmq_stream_common exited with reason: stopped
2023-11-30 13:02:45.767000+00:00 [info] <0.100.0> Stopping application 'khepri'
2023-11-30 13:02:45.892000+00:00 [notice] <0.44.0> Application khepri exited with reason: stopped
2023-11-30 13:02:45.892000+00:00 [info] <0.100.0> Stopping application 'ra'
2023-11-30 13:02:45.908000+00:00 [notice] <0.44.0> Application ra exited with reason: stopped
2023-11-30 13:02:45.908000+00:00 [info] <0.100.0> Stopping application 'os_mon'
2023-11-30 13:02:45.939000+00:00 [notice] <0.44.0> Application os_mon exited with reason: stopped
2023-11-30 13:02:45.939000+00:00 [info] <0.100.0> Stopping application 'sysmon_handler'
2023-11-30 13:02:45.955000+00:00 [notice] <0.44.0> Application sysmon_handler exited with reason: stopped
2023-11-30 13:02:45.955000+00:00 [info] <0.100.0> Stopping application 'osiris'
2023-11-30 13:02:45.955000+00:00 [info] <0.1205.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityTransportTransitionEvent_17011569394755107 reason: shutdown
2023-11-30 13:02:45.955000+00:00 [info] <0.1200.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsPlcCpuLoadEvent_17011569370516482 reason: shutdown
2023-11-30 13:02:45.955000+00:00 [info] <0.1195.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsADPInspectionEvent_17011569333079809 reason: shutdown
2023-11-30 13:02:45.955000+00:00 [info] <0.1193.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsADPInspectionTurningEvent_17011569336758966 reason: shutdown
2023-11-30 13:02:45.955000+00:00 [info] <0.1095.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityOrderStateSnapshotEvent_17011569324980135 reason: shutdown
2023-11-30 13:02:45.955000+00:00 [info] <0.1094.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityOrderTransitionEvent_17011569386105538 reason: shutdown
2023-11-30 13:02:45.955000+00:00 [info] <0.1085.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsInductEvent_17011569358032130 reason: shutdown
2023-11-30 13:02:45.955000+00:00 [info] <0.1084.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityOrderItemEvent_17011569316896845 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1083.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsAirMeasurementEvent_17011569340384875 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1082.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityPlcEvent_17011569317579839 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1081.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityOrderItemTransitionEvent_17011569375703959 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1080.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityOrderItemStateSnapshotEvent_17011569322266591 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1079.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsSorterReportEvent_17011569374122936 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1078.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsPickBufferGeneralEvent_17011569369792529 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1077.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsPackstationEvent_17011569368329263 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1076.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsPickBufferBlockageEvent_17011569369067551 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1075.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsPackloopGeneralEvent_17011569366072413 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1074.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsLSPActivityEvent_17011569361313982 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1073.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsFlexBufferALPointEvent_17011569352374646 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1072.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsTPReaderEvent_17011569374829481 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1071.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsMotorEvent_17011569363576976 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1070.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsSeparationEvent_17011569372555224 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1069.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsAPWorkstationGeneralEvent_17011569342499392 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1068.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsSwitchEvent_17011569374427058 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1067.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsADPScaleEvent_17011569338543761 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1066.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityOrderEvent_17011569303737691 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1065.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityTransportSnapshotEvent_17011569327214457 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1064.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsComponentFillingEvent_17011569348610499 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1063.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsADPInspectionDirtEvent_17011569330605876 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1062.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsFlexBufferGeneralEvent_17011569354440011 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1061.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsEntryPointEvent_17011569350471069 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1060.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsBarcodeScanEvent_17011569345817084 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1059.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsADPDropEvaluationEvent_17011569328126823 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1031.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsADPInspectionGeneralEvent_17011569334846173 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1030.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsAPWorkstationGroupGeneralEvent_17011569344183838 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1029.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsPlcCodeExecutionEvent_17011569370202120 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1028.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsFlowSensorEvent_17011569356561696 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1027.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsScaleEvent_17011569371103222 reason: shutdown
2023-11-30 13:02:45.970000+00:00 [info] <0.1026.0> osiris_writer:terminate/2: name __Stream_DataAgentEntityStatisticsPickBufferALPointEvent_17011569368685607 reason: shutdown
2023-11-30 13:02:45.986000+00:00 [notice] <0.44.0> Application osiris exited with reason: stopped
2023-11-30 13:02:45.986000+00:00 [info] <0.100.0> Stopping application 'mnesia'
2023-11-30 13:02:46.002000+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0> Successfully stopped RabbitMQ and its dependencies
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0> Halting Erlang VM with the following applications:
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     khepri_mnesia_migration
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     horus
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     erts
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     amqp10_common
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     observer_cli
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     redbug
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     runtime_tools
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     stdout_formatter
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     seshat
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     gen_batch_server
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     aten
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     ranch
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     inets
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     systemd
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     enough
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     cuttlefish
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     credentials_obfuscation
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     recon
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     thoas
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     xmerl
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     tools
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     syntax_tools
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     ssl
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     public_key
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     asn1
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     crypto
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     compiler
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     sasl
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     stdlib
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0>     kernel
2023-11-30 13:02:46.002000+00:00 [info] <0.100.0> 
2023-11-30 13:03:12.779000+00:00 [notice] <0.246.0> Logging: configured log handlers are now ACTIVE
2023-11-30 13:03:12.794000+00:00 [info] <0.246.0> ra: starting system quorum_queues
2023-11-30 13:03:12.794000+00:00 [info] <0.246.0> starting Ra system: quorum_queues in directory: d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/quorum/rabbit@PON-WPAS99
2023-11-30 13:03:12.982000+00:00 [info] <0.261.0> ra system 'quorum_queues' running pre init for 0 registered servers
2023-11-30 13:03:12.998000+00:00 [info] <0.262.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2023-11-30 13:03:13.029000+00:00 [notice] <0.267.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2023-11-30 13:03:13.107000+00:00 [info] <0.246.0> ra: starting system coordination
2023-11-30 13:03:13.107000+00:00 [info] <0.246.0> starting Ra system: coordination in directory: d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/coordination/rabbit@PON-WPAS99
2023-11-30 13:03:13.107000+00:00 [info] <0.274.0> ra system 'coordination' running pre init for 2 registered servers
2023-11-30 13:03:13.138000+00:00 [info] <0.275.0> ra: meta data store initialised for system coordination. 2 record(s) recovered
2023-11-30 13:03:13.138000+00:00 [notice] <0.280.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
2023-11-30 13:03:13.185000+00:00 [info] <0.246.0> ra: starting system coordination
2023-11-30 13:03:13.185000+00:00 [info] <0.246.0> starting Ra system: coordination in directory: d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/coordination/rabbit@PON-WPAS99
2023-11-30 13:03:13.529000+00:00 [info] <0.246.0> Waiting for Khepri leader for 30000 ms, 9 retries left
2023-11-30 13:03:13.794000+00:00 [notice] <0.285.0> RabbitMQ metadata store: candidate -> leader in term: 13 machine version: 0
2023-11-30 13:03:13.794000+00:00 [info] <0.246.0> Khepri leader elected
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0> 
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>  Starting RabbitMQ 3.13.0-rc.2 on Erlang 26.1.2 [jit]
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>  Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0> 
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>  node           : rabbit@PON-WPAS99
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>  home dir       : c:/WINDOWS/system32/config/systemprofile
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>  config file(s) : d:/RabbitMQ/advanced.config
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>                 : d:/RabbitMQ/rabbitmq.conf
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>  cookie hash    : FvowNko4yx1RG/pmBG55FA==
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>  log(s)         : <stdout>
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>                 : d:/RabbitMQ/log/[email protected]
2023-11-30 13:03:14.810000+00:00 [info] <0.246.0>  data dir       : d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia
2023-11-30 13:03:23.169000+00:00 [info] <0.246.0> Running boot step pre_boot defined by app rabbit
2023-11-30 13:03:23.169000+00:00 [info] <0.246.0> Running boot step rabbit_global_counters defined by app rabbit
2023-11-30 13:03:23.169000+00:00 [info] <0.246.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2023-11-30 13:03:23.169000+00:00 [info] <0.246.0> Running boot step rabbit_core_metrics defined by app rabbit
2023-11-30 13:03:23.169000+00:00 [info] <0.246.0> Running boot step rabbit_alarm defined by app rabbit
2023-11-30 13:03:23.201000+00:00 [info] <0.388.0> Memory high watermark set to 6553 MiB (6871538073 bytes) of 16383 MiB (17178845184 bytes) total
2023-11-30 13:03:23.201000+00:00 [info] <0.390.0> Enabling free disk space monitoring (disk free space: 208158863360, total memory: 17178845184)
2023-11-30 13:03:23.201000+00:00 [info] <0.390.0> Disk free limit set to 50MB
2023-11-30 13:03:23.201000+00:00 [info] <0.246.0> Running boot step code_server_cache defined by app rabbit
2023-11-30 13:03:23.201000+00:00 [info] <0.246.0> Running boot step file_handle_cache defined by app rabbit
2023-11-30 13:03:23.201000+00:00 [info] <0.393.0> Limiting to approx 65439 file handles (58893 sockets)
2023-11-30 13:03:23.201000+00:00 [info] <0.394.0> FHC read buffering: OFF
2023-11-30 13:03:23.201000+00:00 [info] <0.394.0> FHC write buffering: ON
2023-11-30 13:03:23.201000+00:00 [info] <0.246.0> Running boot step worker_pool defined by app rabbit
2023-11-30 13:03:23.201000+00:00 [info] <0.330.0> Will use 8 processes for default worker pool
2023-11-30 13:03:23.201000+00:00 [info] <0.330.0> Starting worker pool 'worker_pool' with 8 processes in it
2023-11-30 13:03:23.201000+00:00 [info] <0.246.0> Running boot step database defined by app rabbit
2023-11-30 13:03:23.201000+00:00 [info] <0.246.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2023-11-30 13:03:23.201000+00:00 [info] <0.246.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-11-30 13:03:23.216000+00:00 [info] <0.246.0> Successfully synced tables from a peer
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Successfully synced tables from a peer
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step tracking_metadata_store defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.412.0> Setting up a table for connection tracking on this node: tracked_connection
2023-11-30 13:03:23.263000+00:00 [info] <0.412.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
2023-11-30 13:03:23.263000+00:00 [info] <0.412.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
2023-11-30 13:03:23.263000+00:00 [info] <0.412.0> Setting up a table for channel tracking on this node: tracked_channel
2023-11-30 13:03:23.263000+00:00 [info] <0.412.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step networking_metadata_store defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step feature_flags defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step codec_correctness_check defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step external_infrastructure defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_event defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_registry defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_priority_queue defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Priority queues enabled, real BQ is rabbit_variable_queue
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_queue_location_random defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step kernel_ready defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2023-11-30 13:03:23.263000+00:00 [info] <0.420.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2023-11-30 13:03:23.263000+00:00 [info] <0.246.0> Running boot step guid_generator defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_node_monitor defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.424.0> Starting rabbit_node_monitor (in ignore mode)
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step delegate_sup defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_memory_monitor defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step core_initialized defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_definitions_hashing defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_policies defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_policy defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_quorum_queue defined by app rabbit
2023-11-30 13:03:23.279000+00:00 [info] <0.246.0> Running boot step rabbit_stream_coordinator defined by app rabbit
2023-11-30 13:03:23.294000+00:00 [info] <0.246.0> Running boot step rabbit_vhost_limit defined by app rabbit
2023-11-30 13:03:23.294000+00:00 [info] <0.246.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2023-11-30 13:03:23.294000+00:00 [info] <0.246.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2023-11-30 13:03:23.294000+00:00 [info] <0.246.0> Management plugin: using rates mode 'basic'
2023-11-30 13:03:23.294000+00:00 [info] <0.246.0> Running boot step recovery defined by app rabbit
2023-11-30 13:03:23.294000+00:00 [info] <0.463.0> Making sure data directory 'd:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615034,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615035,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615036,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615037,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615038,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615039,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615040,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615041,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615042,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator: member started unexpected #{index => 50614306,node => 'rabbit@PON-WPAS99',pid => <0.1562.0>,system_time => 1701285935655,epoch => 587,machine_version => 4} {member,{running,587,<0.1243717.0>},{writer,587},false,undefined,#{name => [95,95,83,116,114,101,97,109,95,68,97,116,97,65,103,101,110,116,69,110,116,105,116,121,80,108,99,69,118,101,110,116,95,49,55,48,49,49,53,54,57,51,49,55,53,55,57,56,51,57],nodes => ['rabbit@PON-WPAS99'],reference => {resource,<<47>>,queue,<<83,116,114,101,97,109,95,68,97,116,97,65,103,101,110,116,69,110,116,105,116,121,80,108,99,69,118,101,110,116>>},epoch => 587,replica_nodes => [],leader_node => 'rabbit@PON-WPAS99',retention => [{max_age,259200000}],event_formatter => {rabbit_stream_queue,format_osiris_event,[{resource,<<47>>,queue,<<83,116,114,101,97,109,95,68,97,116,97,65,103,101,110,116,69,110,116,105,116,121,80,108,99,69,118,101,110,116>>}]}},running}
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615044,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615045,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615046,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615047,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615048,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615049,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615050,system_time => 1701286111718,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615051,system_time => 1701286111719,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615355,system_time => 1701286112040,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615357,system_time => 1701286112040,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615358,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615360,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615361,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615362,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615363,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615367,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615368,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615370,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615372,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615373,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615375,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615377,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615378,system_time => 1701286112041,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615473,system_time => 1701286112062,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615481,system_time => 1701286112071,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615482,system_time => 1701286112072,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615483,system_time => 1701286112072,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615484,system_time => 1701286112072,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615485,system_time => 1701286112072,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615486,system_time => 1701286112072,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615488,system_time => 1701286112072,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615490,system_time => 1701286112072,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615491,system_time => 1701286112072,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.310000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615492,system_time => 1701286112073,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615493,system_time => 1701286112073,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615494,system_time => 1701286112073,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615495,system_time => 1701286112073,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615497,system_time => 1701286112073,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615500,system_time => 1701286112073,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615501,system_time => 1701286112073,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615504,system_time => 1701286112073,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615505,system_time => 1701286112073,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615506,system_time => 1701286112073,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615509,system_time => 1701286112074,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615514,system_time => 1701286112074,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615541,system_time => 1701286112075,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615542,system_time => 1701286112077,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615546,system_time => 1701286112083,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615556,system_time => 1701286112083,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615562,system_time => 1701286112083,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615564,system_time => 1701286112083,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> rabbit_stream_coordinator failed to update stream:
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> function_clause
2023-11-30 13:03:23.326000+00:00 [warning] <0.449.0> [{rabbit_stream_coordinator,update_stream0,[#{index => 50615588,system_time => 1701286112087,machine_version => 4},{member_started,[95|...],#{}},{stream,[...],...}],[{file,[114|...]},{line,1178}]},{rabbit_stream_coordinator,update_stream,3,[{file,[...]},{line,...}]},{rabbit_stream_coordinator,apply,3,[{file,...},{...}]},{ra_server,apply_with,2,[{...}|...]},{ra_server,'-recover/1-fun-0-',3,[...]},{ra_log_segment,fold0,8,...},{ra_log_reader,'-segment_fold/5-fun-1-',...},{ra_log_reader,...}]
2023-11-30 13:03:23.326000+00:00 [info] <0.463.0> Starting message stores for vhost '/'
2023-11-30 13:03:23.326000+00:00 [info] <0.473.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2023-11-30 13:03:23.341000+00:00 [info] <0.463.0> Started message store of type transient for vhost '/'
2023-11-30 13:03:23.341000+00:00 [info] <0.477.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2023-11-30 13:03:23.341000+00:00 [info] <0.463.0> Started message store of type persistent for vhost '/'
2023-11-30 13:03:23.748000+00:00 [notice] <0.449.0> rabbit_stream_coordinator: candidate -> leader in term: 23 machine version: 4
2023-11-30 13:03:23.904000+00:00 [info] <0.1021.0> Stream: __Stream_DataAgentEntityStatisticsPickBufferBlockageEvent_17011569369067551 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsPickBufferBlockageEvent_17011569369067551 for osiris log data directory
2023-11-30 13:03:23.904000+00:00 [info] <0.1020.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsPickBufferBlockageEvent_17011569369067551 on 'rabbit@PON-WPAS99' in 8
2023-11-30 13:03:23.919000+00:00 [info] <0.1052.0> Stream: __Stream_DataAgentEntityStatisticsSwitchEvent_17011569374427058 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsSwitchEvent_17011569374427058 for osiris log data directory
2023-11-30 13:03:23.919000+00:00 [info] <0.1022.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsSwitchEvent_17011569374427058 on 'rabbit@PON-WPAS99' in 235676
2023-11-30 13:03:23.919000+00:00 [info] <0.1023.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityPlcEvent_17011569317579839 on 'rabbit@PON-WPAS99' in 590
2023-11-30 13:03:23.919000+00:00 [info] <0.1053.0> Stream: __Stream_DataAgentEntityPlcEvent_17011569317579839 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityPlcEvent_17011569317579839 for osiris log data directory
2023-11-30 13:03:23.919000+00:00 [info] <0.1024.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsADPDropEvaluationEvent_17011569328126823 on 'rabbit@PON-WPAS99' in 234019
2023-11-30 13:03:23.919000+00:00 [info] <0.1054.0> Stream: __Stream_DataAgentEntityStatisticsADPDropEvaluationEvent_17011569328126823 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsADPDropEvaluationEvent_17011569328126823 for osiris log data directory
2023-11-30 13:03:23.919000+00:00 [info] <0.1025.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityOrderEvent_17011569303737691 on 'rabbit@PON-WPAS99' in 231971
2023-11-30 13:03:23.919000+00:00 [info] <0.1055.0> Stream: __Stream_DataAgentEntityOrderEvent_17011569303737691 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityOrderEvent_17011569303737691 for osiris log data directory
2023-11-30 13:03:23.919000+00:00 [info] <0.1026.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsBarcodeScanEvent_17011569345817084 on 'rabbit@PON-WPAS99' in 235275
2023-11-30 13:03:23.919000+00:00 [info] <0.1056.0> Stream: __Stream_DataAgentEntityStatisticsBarcodeScanEvent_17011569345817084 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsBarcodeScanEvent_17011569345817084 for osiris log data directory
2023-11-30 13:03:23.919000+00:00 [info] <0.1027.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsSeparationEvent_17011569372555224 on 'rabbit@PON-WPAS99' in 238565
2023-11-30 13:03:23.919000+00:00 [info] <0.1057.0> Stream: __Stream_DataAgentEntityStatisticsSeparationEvent_17011569372555224 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsSeparationEvent_17011569372555224 for osiris log data directory
2023-11-30 13:03:23.919000+00:00 [info] <0.1028.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsFlexBufferGeneralEvent_17011569354440011 on 'rabbit@PON-WPAS99' in 236636
2023-11-30 13:03:23.919000+00:00 [info] <0.1029.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsLSPActivityEvent_17011569361313982 on 'rabbit@PON-WPAS99' in 235217
2023-11-30 13:03:23.919000+00:00 [info] <0.1058.0> Stream: __Stream_DataAgentEntityStatisticsFlexBufferGeneralEvent_17011569354440011 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsFlexBufferGeneralEvent_17011569354440011 for osiris log data directory
2023-11-30 13:03:23.919000+00:00 [info] <0.1030.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsTPReaderEvent_17011569374829481 on 'rabbit@PON-WPAS99' in 239013
2023-11-30 13:03:23.919000+00:00 [info] <0.1060.0> Stream: __Stream_DataAgentEntityStatisticsLSPActivityEvent_17011569361313982 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsLSPActivityEvent_17011569361313982 for osiris log data directory
2023-11-30 13:03:23.919000+00:00 [info] <0.1031.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityOrderStateSnapshotEvent_17011569324980135 on 'rabbit@PON-WPAS99' in 239442
2023-11-30 13:03:23.919000+00:00 [info] <0.1061.0> Stream: __Stream_DataAgentEntityStatisticsTPReaderEvent_17011569374829481 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsTPReaderEvent_17011569374829481 for osiris log data directory
2023-11-30 13:03:23.935000+00:00 [info] <0.1032.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsEntryPointEvent_17011569350471069 on 'rabbit@PON-WPAS99' in 238704
2023-11-30 13:03:23.919000+00:00 [info] <0.1063.0> Stream: __Stream_DataAgentEntityOrderStateSnapshotEvent_17011569324980135 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityOrderStateSnapshotEvent_17011569324980135 for osiris log data directory
2023-11-30 13:03:23.935000+00:00 [info] <0.1033.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityOrderItemStateSnapshotEvent_17011569322266591 on 'rabbit@PON-WPAS99' in 239451
2023-11-30 13:03:23.935000+00:00 [info] <0.1064.0> Stream: __Stream_DataAgentEntityStatisticsEntryPointEvent_17011569350471069 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsEntryPointEvent_17011569350471069 for osiris log data directory
2023-11-30 13:03:23.935000+00:00 [info] <0.1034.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsPlcCodeExecutionEvent_17011569370202120 on 'rabbit@PON-WPAS99' in 2466
2023-11-30 13:03:23.935000+00:00 [info] <0.1065.0> Stream: __Stream_DataAgentEntityOrderItemStateSnapshotEvent_17011569322266591 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityOrderItemStateSnapshotEvent_17011569322266591 for osiris log data directory
2023-11-30 13:03:23.935000+00:00 [info] <0.1037.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsAirMeasurementEvent_17011569340384875 on 'rabbit@PON-WPAS99' in 239115
2023-11-30 13:03:23.935000+00:00 [info] <0.1066.0> Stream: __Stream_DataAgentEntityStatisticsPlcCodeExecutionEvent_17011569370202120 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsPlcCodeExecutionEvent_17011569370202120 for osiris log data directory
2023-11-30 13:03:23.935000+00:00 [info] <0.1067.0> Stream: __Stream_DataAgentEntityStatisticsAirMeasurementEvent_17011569340384875 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsAirMeasurementEvent_17011569340384875 for osiris log data directory
2023-11-30 13:03:23.935000+00:00 [info] <0.1039.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsAPWorkstationGroupGeneralEvent_17011569344183838 on 'rabbit@PON-WPAS99' in 236704
2023-11-30 13:03:23.935000+00:00 [info] <0.1068.0> Stream: __Stream_DataAgentEntityStatisticsAPWorkstationGroupGeneralEvent_17011569344183838 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsAPWorkstationGroupGeneralEvent_17011569344183838 for osiris log data directory
2023-11-30 13:03:23.935000+00:00 [info] <0.1069.0> Stream: __Stream_DataAgentEntityOrderItemTransitionEvent_17011569375703959 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityOrderItemTransitionEvent_17011569375703959 for osiris log data directory
2023-11-30 13:03:23.935000+00:00 [info] <0.1035.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityOrderItemTransitionEvent_17011569375703959 on 'rabbit@PON-WPAS99' in 232253
2023-11-30 13:03:23.935000+00:00 [info] <0.1070.0> Stream: __Stream_DataAgentEntityStatisticsADPInspectionGeneralEvent_17011569334846173 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsADPInspectionGeneralEvent_17011569334846173 for osiris log data directory
2023-11-30 13:03:23.935000+00:00 [info] <0.1036.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsADPInspectionGeneralEvent_17011569334846173 on 'rabbit@PON-WPAS99' in 236957
2023-11-30 13:03:23.935000+00:00 [info] <0.1071.0> Stream: __Stream_DataAgentEntityStatisticsAPWorkstationGeneralEvent_17011569342499392 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsAPWorkstationGeneralEvent_17011569342499392 for osiris log data directory
2023-11-30 13:03:23.935000+00:00 [info] <0.1038.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsAPWorkstationGeneralEvent_17011569342499392 on 'rabbit@PON-WPAS99' in 236945
2023-11-30 13:03:23.935000+00:00 [info] <0.1072.0> Stream: __Stream_DataAgentEntityStatisticsADPInspectionEvent_17011569333079809 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsADPInspectionEvent_17011569333079809 for osiris log data directory
2023-11-30 13:03:23.951000+00:00 [info] <0.1040.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsADPInspectionEvent_17011569333079809 on 'rabbit@PON-WPAS99' in 234652
2023-11-30 13:03:23.951000+00:00 [info] <0.1073.0> Stream: __Stream_DataAgentEntityStatisticsComponentFillingEvent_17011569348610499 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsComponentFillingEvent_17011569348610499 for osiris log data directory
2023-11-30 13:03:23.951000+00:00 [info] <0.1041.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsComponentFillingEvent_17011569348610499 on 'rabbit@PON-WPAS99' in 239119
2023-11-30 13:03:23.951000+00:00 [info] <0.1074.0> Stream: __Stream_DataAgentEntityStatisticsPlcCpuLoadEvent_17011569370516482 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsPlcCpuLoadEvent_17011569370516482 for osiris log data directory
2023-11-30 13:03:23.951000+00:00 [info] <0.1042.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsPlcCpuLoadEvent_17011569370516482 on 'rabbit@PON-WPAS99' in 238858
2023-11-30 13:03:23.951000+00:00 [info] <0.1075.0> Stream: __Stream_DataAgentEntityStatisticsFlowSensorEvent_17011569356561696 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsFlowSensorEvent_17011569356561696 for osiris log data directory
2023-11-30 13:03:23.951000+00:00 [info] <0.1043.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsFlowSensorEvent_17011569356561696 on 'rabbit@PON-WPAS99' in 238887
2023-11-30 13:03:23.951000+00:00 [info] <0.1076.0> Stream: __Stream_DataAgentEntityStatisticsADPInspectionTurningEvent_17011569336758966 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsADPInspectionTurningEvent_17011569336758966 for osiris log data directory
2023-11-30 13:03:23.951000+00:00 [info] <0.1044.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsADPInspectionTurningEvent_17011569336758966 on 'rabbit@PON-WPAS99' in 236860
2023-11-30 13:03:23.951000+00:00 [info] <0.1077.0> Stream: __Stream_DataAgentEntityStatisticsADPScaleEvent_17011569338543761 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsADPScaleEvent_17011569338543761 for osiris log data directory
2023-11-30 13:03:23.951000+00:00 [info] <0.1045.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsADPScaleEvent_17011569338543761 on 'rabbit@PON-WPAS99' in 230808
2023-11-30 13:03:23.951000+00:00 [info] <0.1078.0> Stream: __Stream_DataAgentEntityStatisticsFlexBufferALPointEvent_17011569352374646 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsFlexBufferALPointEvent_17011569352374646 for osiris log data directory
2023-11-30 13:03:23.951000+00:00 [info] <0.1079.0> Stream: __Stream_DataAgentEntityStatisticsInductEvent_17011569358032130 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsInductEvent_17011569358032130 for osiris log data directory
2023-11-30 13:03:23.951000+00:00 [info] <0.1046.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsFlexBufferALPointEvent_17011569352374646 on 'rabbit@PON-WPAS99' in 238998
2023-11-30 13:03:23.966000+00:00 [info] <0.1048.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsPickBufferGeneralEvent_17011569369792529 on 'rabbit@PON-WPAS99' in 8
2023-11-30 13:03:23.966000+00:00 [info] <0.1080.0> Stream: __Stream_DataAgentEntityStatisticsPickBufferGeneralEvent_17011569369792529 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsPickBufferGeneralEvent_17011569369792529 for osiris log data directory
2023-11-30 13:03:23.966000+00:00 [info] <0.1047.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsInductEvent_17011569358032130 on 'rabbit@PON-WPAS99' in 238918
2023-11-30 13:03:23.966000+00:00 [info] <0.1081.0> Stream: __Stream_DataAgentEntityStatisticsPickBufferALPointEvent_17011569368685607 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsPickBufferALPointEvent_17011569368685607 for osiris log data directory
2023-11-30 13:03:23.966000+00:00 [info] <0.1049.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsPickBufferALPointEvent_17011569368685607 on 'rabbit@PON-WPAS99' in 8
2023-11-30 13:03:23.966000+00:00 [info] <0.1082.0> Stream: __Stream_DataAgentEntityStatisticsPackstationEvent_17011569368329263 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsPackstationEvent_17011569368329263 for osiris log data directory
2023-11-30 13:03:23.966000+00:00 [info] <0.1050.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsPackstationEvent_17011569368329263 on 'rabbit@PON-WPAS99' in 8
2023-11-30 13:03:23.966000+00:00 [info] <0.1084.0> Stream: __Stream_DataAgentEntityStatisticsSorterReportEvent_17011569374122936 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsSorterReportEvent_17011569374122936 for osiris log data directory
2023-11-30 13:03:23.966000+00:00 [info] <0.1051.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsSorterReportEvent_17011569374122936 on 'rabbit@PON-WPAS99' in 8
2023-11-30 13:03:24.169000+00:00 [info] <0.1021.0> osiris_writer:init/1: name: __Stream_DataAgentEntityStatisticsPickBufferBlockageEvent_17011569369067551 last offset: -1 committed chunk id: -1 epoch: 8
2023-11-30 13:03:24.185000+00:00 [info] <0.1125.0> Stream: __Stream_DataAgentEntityStatisticsADPInspectionDirtEvent_17011569330605876 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsADPInspectionDirtEvent_17011569330605876 for osiris log data directory
2023-11-30 13:03:24.185000+00:00 [info] <0.1115.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsADPInspectionDirtEvent_17011569330605876 on 'rabbit@PON-WPAS99' in 236840
2023-11-30 13:03:24.185000+00:00 [info] <0.1120.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsPackloopGeneralEvent_17011569366072413 on 'rabbit@PON-WPAS99' in 239143
2023-11-30 13:03:24.185000+00:00 [info] <0.1122.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityTransportSnapshotEvent_17011569327214457 on 'rabbit@PON-WPAS99' in 238967
2023-11-30 13:03:24.185000+00:00 [info] <0.1133.0> Stream: __Stream_DataAgentEntityTransportSnapshotEvent_17011569327214457 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityTransportSnapshotEvent_17011569327214457 for osiris log data directory
2023-11-30 13:03:24.185000+00:00 [info] <0.1128.0> Stream: __Stream_DataAgentEntityStatisticsPackloopGeneralEvent_17011569366072413 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsPackloopGeneralEvent_17011569366072413 for osiris log data directory
2023-11-30 13:03:24.185000+00:00 [info] <0.1124.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsMotorEvent_17011569363576976 on 'rabbit@PON-WPAS99' in 238938
2023-11-30 13:03:24.185000+00:00 [info] <0.1136.0> Stream: __Stream_DataAgentEntityStatisticsMotorEvent_17011569363576976 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsMotorEvent_17011569363576976 for osiris log data directory
2023-11-30 13:03:24.185000+00:00 [info] <0.1157.0> Stream: __Stream_DataAgentEntityTransportTransitionEvent_17011569394755107 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityTransportTransitionEvent_17011569394755107 for osiris log data directory
2023-11-30 13:03:24.185000+00:00 [info] <0.1155.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityTransportTransitionEvent_17011569394755107 on 'rabbit@PON-WPAS99' in 8
2023-11-30 13:03:24.232000+00:00 [info] <0.1080.0> osiris_writer:init/1: name: __Stream_DataAgentEntityStatisticsPickBufferGeneralEvent_17011569369792529 last offset: -1 committed chunk id: -1 epoch: 8
2023-11-30 13:03:24.232000+00:00 [info] <0.1084.0> osiris_writer:init/1: name: __Stream_DataAgentEntityStatisticsSorterReportEvent_17011569374122936 last offset: -1 committed chunk id: -1 epoch: 8
2023-11-30 13:03:24.232000+00:00 [info] <0.1082.0> osiris_writer:init/1: name: __Stream_DataAgentEntityStatisticsPackstationEvent_17011569368329263 last offset: -1 committed chunk id: -1 epoch: 8
2023-11-30 13:03:24.326000+00:00 [info] <0.1081.0> osiris_writer:init/1: name: __Stream_DataAgentEntityStatisticsPickBufferALPointEvent_17011569368685607 last offset: -1 committed chunk id: -1 epoch: 8
2023-11-30 13:03:24.435000+00:00 [info] <0.1190.0> Stream: __Stream_DataAgentEntityOrderTransitionEvent_17011569386105538 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityOrderTransitionEvent_17011569386105538 for osiris log data directory
2023-11-30 13:03:24.435000+00:00 [info] <0.1185.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityOrderTransitionEvent_17011569386105538 on 'rabbit@PON-WPAS99' in 232320
2023-11-30 13:03:24.435000+00:00 [info] <0.1191.0> Stream: __Stream_DataAgentEntityOrderItemEvent_17011569316896845 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityOrderItemEvent_17011569316896845 for osiris log data directory
2023-11-30 13:03:24.435000+00:00 [info] <0.1186.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityOrderItemEvent_17011569316896845 on 'rabbit@PON-WPAS99' in 231626
2023-11-30 13:03:24.435000+00:00 [info] <0.1200.0> Stream: __Stream_DataAgentEntityStatisticsScaleEvent_17011569371103222 will use d:/RabbitMQ/db/rabbit@PON-WPAS99-mnesia/stream/__Stream_DataAgentEntityStatisticsScaleEvent_17011569371103222 for osiris log data directory
2023-11-30 13:03:24.435000+00:00 [info] <0.1199.0> rabbit_stream_coordinator: started writer __Stream_DataAgentEntityStatisticsScaleEvent_17011569371103222 on 'rabbit@PON-WPAS99' in 239100
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>   crasher:
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     initial call: osiris_writer:init/1
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     pid: <0.1063.0>
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     registered_name: []
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     exception exit: {{case_clause,
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                          {ok,<<17,203,128,230,67,143,221,11,56,31,187,46,239,
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                                166,136,105,16,116,237,73,197,68,204,70,208,
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                                13,238,97,66,155,220,120,144,59,240,213,227,
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                                44,152,144,51,206,103,146,173,185,72,163>>}},
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                      [{osiris_log,recover_tracking,3,
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                           [{file,"src/osiris_log.erl"},{line,2629}]},
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                       {osiris_writer,handle_continue,2,
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                           [{file,"src/osiris_writer.erl"},{line,196}]},
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                       {gen_batch_server,handle_continue,2,
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                           [{file,"src/gen_batch_server.erl"},{line,413}]},
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                       {gen_batch_server,init_it,6,
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                           [{file,"src/gen_batch_server.erl"},{line,163}]},
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                       {proc_lib,init_p_do_apply,3,
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>                           [{file,"proc_lib.erl"},{line,241}]}]}
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>       in function  gen_batch_server:handle_continue/2 (src/gen_batch_server.erl, line 423)
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>       in call from gen_batch_server:init_it/6 (src/gen_batch_server.erl, line 163)
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     ancestors: [osiris_server_sup,osiris_sup,<0.216.0>]
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     message_queue_len: 0
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     messages: []
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     links: [<0.220.0>]
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     dictionary: []
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     trap_exit: true
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     status: running
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     heap_size: 376
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     stack_size: 28
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>     reductions: 17853
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0>   neighbours:
2023-11-30 13:03:25.029000+00:00 [error] <0.1063.0> 
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>     supervisor: {local,osiris_server_sup}
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>     errorContext: child_terminated
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>     reason: {{case_clause,{ok,<<17,203,128,230,67,143,221,11,56,31,187,46,
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                                 239,166,136,105,16,116,237,73,197,68,204,70,
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                                 208,13,238,97,66,155,220,120,144,59,240,213,
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                                 227,44,152,144,51,206,103,146,173,185,72,163>>}},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>              [{osiris_log,recover_tracking,3,
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                           [{file,"src/osiris_log.erl"},{line,2629}]},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>               {osiris_writer,handle_continue,2,
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                              [{file,"src/osiris_writer.erl"},{line,196}]},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>               {gen_batch_server,handle_continue,2,
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                                 [{file,"src/gen_batch_server.erl"},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                                  {line,413}]},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>               {gen_batch_server,init_it,6,
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                                 [{file,"src/gen_batch_server.erl"},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                                  {line,163}]},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>               {proc_lib,init_p_do_apply,3,
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                         [{file,"proc_lib.erl"},{line,241}]}]}
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>     offender: [{pid,<0.1063.0>},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                {id,<<"__Stream_DataAgentEntityOrderStateSnapshotEvent_17011569324980135">>},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                {mfargs,{osiris_writer,start_link,undefined}},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                {restart_type,temporary},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                {significant,false},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                {shutdown,5000},
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>                {child_type,worker}]
2023-11-30 13:03:25.029000+00:00 [error] <0.220.0>

plewam Nov 30, 2023
Author

And crazy high CPU usage and ram increase

plewam Nov 30, 2023
Author

And from there on the error remains (CPU load returns to normal however), the log file is flooded, and the memory increases until high watermark is reached. When reached the broker stops accepting connections.

michaelklishin · 2023-11-28T14:11:09Z

michaelklishin
Nov 28, 2023
Maintainer

Alternatively, a set of Stream PerfTest CLI flags that we can use to get a stream into that state would suffice. I assume that the size of an archive with 1 billion messages can be prohibitive in practice.

0 replies

kjnilsson · 2023-12-01T11:48:55Z

kjnilsson
Dec 1, 2023
Maintainer

I think I have found the issue. It occurs when the very first chunk in a stream isn't written fully which causes subsequent chunks to be written out of position.

@plewam I don't know why the data was lost as you say it was an ordered shutdown but stuff like that in a virtualised environment isn't uncommon and RabbitMQ streams are meant to cope with it.

It does also highlight that running streams without replication means you have fairly weak data safety guarantees.

Here is the PR to the streaming subsystem. Once that is merged I will update RabbitMQ main branch.

rabbitmq/osiris#153

12 replies

plewam Dec 21, 2023
Author

Can you see whether you can download it here: https://drive.google.com/file/d/1Lv6_r_UDa5Wi5YPny2RJ5nXfkzL48r-c/view?usp=sharing

plewam Jan 5, 2024
Author

@kjnilsson Should I give rmq 3.13.0-rc.4 a try?

plewam Jan 12, 2024
Author

@kjnilsson Maybe one more observation from my side. I use this client library to publish to the stream. https://github.com/rabbitmq/rabbitmq-stream-dotnet-client
As in my scenario RMQ and my client is running on the same machine I only have limited control about who is starting/ stopping first. The problem seems to get more likely when my client is continuously trying to publish messages to the stream while RMQ is still starting/ stopping.

kjnilsson Jan 12, 2024
Maintainer

When you tested the version with the fix did you upgrade an existing environment or did you perform a clean install and re-declared all the streams?

plewam Jan 12, 2024
Author

I upgraded an existing environment, but deleted the stream directory. However the issue persists.

Stream (Osiris) writer continuously restarts and logs errors #9998

Uh oh!

plewam Nov 28, 2023

Replies: 3 comments · 48 replies

Uh oh!

michaelklishin Nov 28, 2023 Maintainer

Uh oh!

lukebakken Nov 30, 2023 Maintainer

Uh oh!

plewam Nov 30, 2023 Author

Uh oh!

plewam Nov 30, 2023 Author

Uh oh!

plewam Nov 30, 2023 Author

Uh oh!

plewam Nov 30, 2023 Author

Uh oh!

michaelklishin Nov 28, 2023 Maintainer

Uh oh!

kjnilsson Dec 1, 2023 Maintainer

Uh oh!

plewam Dec 21, 2023 Author

Uh oh!

plewam Jan 5, 2024 Author

Uh oh!

plewam Jan 12, 2024 Author

Uh oh!

kjnilsson Jan 12, 2024 Maintainer

Uh oh!

plewam Jan 12, 2024 Author

plewam
Nov 28, 2023

Replies: 3 comments 48 replies

michaelklishin
Nov 28, 2023
Maintainer

lukebakken Nov 30, 2023
Maintainer

plewam Nov 30, 2023
Author

plewam Nov 30, 2023
Author

plewam Nov 30, 2023
Author

plewam Nov 30, 2023
Author

michaelklishin
Nov 28, 2023
Maintainer

kjnilsson
Dec 1, 2023
Maintainer

plewam Dec 21, 2023
Author

plewam Jan 5, 2024
Author

plewam Jan 12, 2024
Author

kjnilsson Jan 12, 2024
Maintainer

plewam Jan 12, 2024
Author