Skip to content

Error accept_chunk_out_of_order using Stream Producer while publishing filter enabled on a cluster #13897

@hiimjako

Description

@hiimjako

Describe the bug

When sending messages to a RabbitMQ cluster using a stream producer with the filter feature, an accept_chunk_out_of_order error log often appears on the replicas.

Error log example:

2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0> ** Reason for termination ==
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0> ** {{accept_chunk_out_of_order,42829465,42829465},
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0>     [{osiris_log,accept_chunk,2,[{file,"src/osiris_log.erl"},{line,808}]},
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0>      {osiris_replica,'-handle_incoming_data/3-fun-0-',2,
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0>                      [{file,"src/osiris_replica.erl"},{line,570}]},
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0>      {lists,foldl,3,[{file,"lists.erl"},{line,2146}]},
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0>      {osiris_replica,handle_incoming_data,3,
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0>                      [{file,"src/osiris_replica.erl"},{line,569}]},
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0>      {gen_server,try_handle_info,3,[{file,"gen_server.erl"},{line,2345}]},
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0>      {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,2433}]},
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0>      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,329}]}]}
2025-05-15 08:57:59.691750+00:00 [error] <0.2780.0> 
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>   crasher:
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     initial call: osiris_replica:init/1
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     pid: <0.2780.0>
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     registered_name: []
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     exception exit: {accept_chunk_out_of_order,42829465,42829465}
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>       in function  osiris_log:accept_chunk/2 (src/osiris_log.erl, line 808)
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>       in call from osiris_replica:'-handle_incoming_data/3-fun-0-'/2 (src/osiris_replica.erl, line 570)
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>       in call from lists:foldl/3 (lists.erl, line 2146)
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>       in call from osiris_replica:handle_incoming_data/3 (src/osiris_replica.erl, line 569)
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>       in call from gen_server:try_handle_info/3 (gen_server.erl, line 2345)
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>       in call from gen_server:handle_msg/6 (gen_server.erl, line 2433)
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     ancestors: [osiris_server_sup,osiris_sup,<0.1646.0>]
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     message_queue_len: 2
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     messages: [{'$gen_cast',{committed_offset,42829427}},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                   {'EXIT',<25744.2447.0>,
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                       {{badmatch,{error,einval}},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                        [{osiris_replica_reader,setopts,3,
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                             [{file,"src/osiris_replica_reader.erl"},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                              {line,390}]},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                         {osiris_replica_reader,do_sendfile0,1,
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                             [{file,"src/osiris_replica_reader.erl"},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                              {line,342}]},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                         {osiris_replica_reader,handle_cast,2,
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                             [{file,"src/osiris_replica_reader.erl"},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                              {line,213}]},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                         {gen_server,try_handle_cast,3,
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                             [{file,"gen_server.erl"},{line,2371}]},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                         {gen_server,handle_msg,6,
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                             [{file,"gen_server.erl"},{line,2433}]},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                         {proc_lib,init_p_do_apply,3,
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                             [{file,"proc_lib.erl"},{line,329}]}]}}]
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     links: [<0.1650.0>]
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.40079776>,
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                                 bits => 58,uniform => #Fun<rand.1.40079776>,
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                                 uniform_n => #Fun<rand.2.40079776>,
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                                 jump => #Fun<rand.3.40079776>},
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>                               [129485132136450126|59260559737362325]}}]
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     trap_exit: true
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     status: running
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     heap_size: 196650
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     stack_size: 29
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>     reductions: 37189241
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0>   neighbours:
2025-05-15 08:57:59.697121+00:00 [error] <0.2780.0> 
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>     supervisor: {local,osiris_server_sup}
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>     errorContext: child_terminated
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>     reason: {accept_chunk_out_of_order,42829465,42829465}
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>     offender: [{pid,<0.2780.0>},
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>                {id,"__test-2_1747299414973352688"},
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>                {mfargs,{osiris_replica,start_link,undefined}},
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>                {restart_type,temporary},
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>                {significant,false},
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>                {shutdown,5000},
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>                {child_type,worker}]
2025-05-15 08:57:59.697962+00:00 [error] <0.1650.0>

Reproduction steps

  • Set up a RabbitMQ cluster with streams enabled
  • Create a stream producer with filtering
  • Watch logs on replicas

More details and a demo here: https://github.com/hiimjako/rabbitmq-issue-accept_chunk_out_of_order

Expected behavior

Expected to not have accept_chunk_out_of_order error in the replicas.

Additional context

Related PR: rabbitmq/osiris#187

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions