Skip to content

New murmur3 based super stream exchange type #8319

Open
@kjnilsson

Description

@kjnilsson

Is your feature request related to a problem? Please describe.

Currently Super Streams are bound to a direct exchange type with their partition id as the routing key (e.g. 1, 2, 3). Super stream publishing clients use murmur3 to hash a provided key which is used for selecting the target stream partition.

When messages come from other protocols which would not naturally have any awareness of the super stream topology there is no means to re-create this routing in the broker.

Describe the solution you'd like

Instead of using a direct exchange when creating a super stream we should provide an option to instead use a new exchange type, rabbit_super_stream_exchange. This exchange type will take the routing key and use murmur3 hashing to select the partition to route the message to. This will provide the same routing logic as is done client side for stream clients but in the broker.

The exchange type would be similar to

-module(rabbit_sharding_exchange_type_modulus_hash).
but using murmur3 based hashing instead.

Describe alternatives you've considered

Alternatives tend to include leaking knowledge of super stream topology onto client applications which isn't always practical.

Additional context

murmur3 based hashing isn't available in OTP until OTP 27 at the earliest so until then we will have to use a slower erlang based implementation. This implementation needs to be validated against the java and .NET implementations and ideally against the SMHasher test framework.

Here is small potential starting point:

https://github.com/bipthelin/murmerl3/tree/master

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions