chore!: make sharding configuration explicit by fryorcraken · Pull Request #3468 · logos-messaging/logos-delivery

fryorcraken · 2025-06-24T07:18:51Z

Description

Make it explicit that the node can either be configured with auto sharding or static sharding. Treat static sharding as an happy path (prev. was implicit error or do-nothing-happy).
Make it clear that without auto-sharding, pubsub topics can't be derived from content topics.

Do so by improving the internal node configuration and make it clear that there are two options, and that the number of shards in the network is only needed for auto sharding.

Also proceed with a number of naming improvement, as per logos-messaging/specs#65

Cluster Conf is actually a (Waku) network conf - as previously mentioned by @Ivansete-status in other pr
tcp/ip "network" conf is renamed to "endpoint conf" to avoid name clashing

Some refactoring done along the way.

Changes

Auto sharding is enabled by default, with 1 shard in network
new shards (or subscribeShards) defaults:
- auto sharding: all shards
- static sharding: no shards (because we cannot guess the shard numbers)
cluster conf renamed to network conf
sharding conf is made explicit static vs auto, number of shards in network is only useful for autosharding
conf shards is actually the shards we want relay to subscribe too, so rename accordingly
use uint16 for cluster id and shard id more (but not totally) consistently across the code base.
move WebSocketConf to the right place

Follow-up changes

probably remove activeRelayShards in config, in favour of having wakunode2 just automatically subscribe to the shards passed on the CLI
Number of TODOs left in code
define a proper type for cluster id and shard ids

Issue

fryorcraken · 2025-06-26T04:35:50Z

waku/waku_lightpush/protocol.nim

-      let parsedTopic = NsContentTopic.parse(pushRequest.message.contentTopic).valueOr:
-        let msg = "Invalid content-topic:" & $error
-        error "lightpush request handling error", error = msg
-        return LightpushResponse(


several return in the error path skipped the metrics report!

fryorcraken · 2025-06-26T04:37:52Z

waku/waku_lightpush/protocol.nim

 ): Future[LightPushResponse] {.async.} =
-  let reqDecodeRes = LightpushRequest.decode(buffer)
-  var isSuccess = false
  var pushResponse: LightpushResponse


Storing results in a var is usually a bad smell because bigger nesting is needed (instead of just exiting on an error).
Bigger nesting = less readable code.
Proof-in-pudding: some return were including in nesting leading to metrics being missed: https://github.com/waku-org/nwaku/pull/3468/files#r2168086052

waku/waku_lightpush/protocol.nim

fryorcraken · 2025-06-26T06:28:16Z

waku/factory/internal_config.nim


  return netConfigRes
-
-# TODO: numShardsInNetwork should be mandatory with autosharding, and unneeded otherwise


github-actions · 2025-06-26T06:28:39Z

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:3468

Built from bd022cb

fryorcraken · 2025-06-26T06:29:39Z

waku/factory/waku_conf.nim

-proc validateShards(wakuConf: WakuConf): Result[void, string] =
-  let numShardsInNetwork = wakuConf.numShardsInNetwork
-
-  # TODO: fix up this behaviour


fryorcraken · 2025-06-26T06:31:06Z

waku/waku_core/topics/sharding.nim

-    s: Sharding,
-    pubsubTopic: Option[PubsubTopic],
-    contentTopics: ContentTopic | seq[ContentTopic],
+proc getShardsFromContentTopics*(


this function was always called when pubsubTopic was none...

fryorcraken · 2025-06-26T06:31:48Z

waku/waku_lightpush/protocol.nim

-        statusDesc: some(msg),
-      )
+  # TODO: is that fine?
+  let res = await handleRequest(wl, peerId, pushRequest)


@NagyZoltanPeter

Is that ok to call an await here? I am not yet familiar with Nim async and whether there are any gotcha

SionoiS

LGTM

I just hope we didn't mess yet again some edge cases...

I swear this is the 5e time we refactor shard config stuff!

SionoiS · 2025-06-26T14:23:34Z

tests/factory/test_external_config.nim

    ## Then
    let conf = res.get()
-    check conf.shards.len == expectedConf.numShardsInNetwork.int
+    check conf.activeRelayShards.len == expectedConf.shardingConf.numShardsInCluster.int


activeRelayShards Does that mean we can have inactive ones???

This is about expressing the fact that those "shards" are really shards we want to subscribe to.

Next step will be to remove that from the config entirely as it's not a node setup matter, but a run time matter.

SionoiS · 2025-06-26T14:30:00Z

waku/factory/node_factory.nim

-    let shard = node.wakuSharding.getShard(contentTopic).valueOr:
-      return err("Could not parse content topic: " & error)
+    let shard = node.wakuAutoSharding.get().getShard(contentTopic).valueOr:
+        return err("Could not parse content topic: " & error)


fryorcraken · 2025-06-27T06:45:31Z

Started to work on fixing interops tests: logos-messaging/logos-delivery-interop-tests#126

fryorcraken · 2025-06-30T05:33:42Z

interop-tests PR logos-messaging/logos-delivery-interop-tests#126 is ready for review.

Once merged, I'll rerun tests on this PR and it should be green.

gabrielmer

Woow amazing work! Thanks so much!

Added first some questions to further understand some points

gabrielmer · 2025-07-01T08:42:33Z

waku/factory/waku_conf.nim


  clusterId*: uint16
-  shards*: seq[uint16]
+  activeRelayShards*: seq[uint16]


There's something I still don't get about activeRelayShards. Shouldn't the user configure which shards they want and that's it? And isn't that what it should be in the configuration, the desired shards by the user?

spoiler: I will remove this parameter, as I am not including it in the Waku API.

these are the shards we want to subscribe to via relay when the node starts.

Define "desired shards by the user"

Define "desired shards by the user"

Exactly this: "the shards we want to subscribe to via relay when the node starts"

When a node is configured and the operator selects the shard they want to operate in. Why call them activeRelayShards instead of shards if it's the config desire of the user/operator and not a run time issue?

or that is the combination of both the shards specified by the user explicitly + the resolved shards by autosharding?

ok now I'm understanding better, that if we don't allow mixing autosharding and static sharding then makes sense!

When a node is configured and the operator selects the shard they want to operate in. Why call them activeRelayShards instead of shards if it's the config desire of the user/operator and not a run time issue?

Because the action is unclear, why does one need to specify shards ? Well, because that the shards we want relay to subscribe to. But that only applies in one specific case: when running a service node with relay.

When running an application, the users or developer does not subscribe to shards, it subscribe to specific content topics they want to receive messages on.

And when not using relay, they don't need shards either.

And even in this case, there is still the problem of:

I am using TWN

I want to run a service node to support my node

What "shard" do I select?

Well, you shouldn't need to "select" a shard, but a "content topic" instead (or a partial one), because autosharding abstract the concept of shards.

not a run time issue?

So we are clear, it is a run time issue. Getting relay to subscribe to shards is not an initialization matter, it is a run time matter.

gabrielmer · 2025-07-01T09:10:02Z

waku/waku_core/topics/sharding.nim

+    let shard =
+      if shardsRes.isErr():
+        return err("Cannot deduce shard from content topic: " & $shardsRes.error)


Maybe this was it's simpler?

Suggested change

let shard =

if shardsRes.isErr():

return err("Cannot deduce shard from content topic: " & $shardsRes.error)

let shard = s.getShard(content).valueOr:

return err("Cannot deduce shard from content topic: " & $error)

(and then removing the let shardsRes = s.getShard(content) and the else: shardsRes.get(), which GH doesn't let me add in the suggestion)

gabrielmer · 2025-07-01T10:10:49Z

waku/node/waku_node.nim

+        let msg =
+          "Static sharding is used, relay subscriptions must specify a pubsub topic"


I think this msg variable is not used anywhere, same for the ContentUnsubcase

waku/waku_lightpush/protocol.nim

gabrielmer · 2025-07-01T10:29:47Z

waku/factory/node_factory.nim

+  # TODO: when using autosharding, the user should not be expected to pass any shards, but only content topics
+  # Hence, this joint logic should be removed in favour of an either logic:
+  # use passed shards (static) or deduce shards from content topics (auto)


Not sure I understand or agree with this, one might subscribe to static shards (for example, default TWN shards) but want to use autosharding to publish messages without having to think about shards in their application's logic. Isn't that legitimate usage?

It's the same as #3468 (comment)

I think we first need to acknowledge that these are 2 separate use cases:

(1) A developer using nwaku (as a lib or otherwise) for their app. In this case, if they use autosharding then they only want to deal with content topic to subscribe to shards or send messages

(2) A node operator running a node on TWN. Assuming said operator don't want to run a node for all shards. Why would they do that? is it to support their app? in this case, which shard should they subscribe too? 1, 2, 3? How do they know. Well, this is back to step (1). They should just deal with content topics.

I do agree with @gabrielmer in this point.

Autosharding seems as a technique for both distributing the traffic randomly across the available shards and to simplify the publish operation.

It seems that we will always have a combination of both static and automatic sharding. For example, TWN, and any other network, would not be able to operate with only automatic sharding, because the content-topic is only available when publishing a message and not when subscribing.

A given network should either have static sharding, or auto sharding, not both. Do we agree with that?

I am a developer that is building my new foobar app on TWN, hence auto sharding is enabled.

Please walk me through how I need to select the "right shard" to subscribe to.

gabrielmer

Thanks so much! I would wait until @Ivansete-status's review before merging just in case :)

Ivansete-status

Thanks so much for it! 🙌

Not approving yet because I didn't review it 100%

I don't want to be a blocker so feel free to merge it if you consider appropriate.
Of course, happy to revisit again in the future if you want to apply any change or split it into smaller PRs :)

Thanks again! ❤️

Ivansete-status · 2025-07-01T20:46:55Z

waku/node/waku_node.nim

    wakuPeerExchange*: WakuPeerExchange
    wakuMetadata*: WakuMetadata
-    wakuSharding*: Sharding
+    wakuAutoSharding*: Option[Sharding]


The wakuAutoSharding shouldn't belong to WakuNode.
IMHO, this is linked to wakuRelay and should be there instead.

I would disagree here, light push, store and filter also uses the autosharding logic, as pubsub topic are part of the request body and needs to be inferred from the content topics.

Ivansete-status · 2025-07-01T20:56:28Z

waku/factory/conf_builder/waku_conf_builder.nim

-  numShardsInNetwork: Option[uint32]
-  shards: Option[seq[uint16]]
+  shardingConf: Option[ShardingConfKind]
+  numShardsInCluster: Option[uint16]


This is a great opportunity to have a good default.
I would not give freedom to select the number of shards in a cluster. No one is going to give a value there with certain decision.
Instead, we need to use the maximum allowed shards within a cluster, which is 1024, according to the spec: https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md

when you start a new network, you want to start with a small number of shards. So that you can have several users on the same shard. If you have no user on a shard, you cannot form a gossipsub network.

1024 shards by default means that you would not form any gossipsub networks, and users would not connect to each other because their application is likely to only connect to one shard.

There is a reason why we selected 8 shards for TWN.

Most new networks would be fine with 1 or 8 shards.

Also, are you thinking a default for service nodes? or for applications?

A default value of 1 may be better, for applications that don't want to use TWN.

Ivansete-status · 2025-07-01T20:57:20Z

waku/factory/conf_builder/waku_conf_builder.nim

-  shards: Option[seq[uint16]]
+  shardingConf: Option[ShardingConfKind]
+  numShardsInCluster: Option[uint16]
+  activeRelayShards: Option[seq[uint16]]


What are the inactive relay shards :)? I think that's been commented elsewhere: what's wrong with shards name?

because shards means nothing. I guess I shall rename this variable to relayShardsToSubscribeTo
These are the shards we want relay to subscribe to.

Do note that this is a pure wakunode2 service node matter. In an application, you would want the application to run subscribe at the start, this option is purely for CLI and I will remove it from WakuConf.

Ivansete-status · 2025-07-01T20:58:54Z

waku/factory/conf_builder/waku_conf_builder.nim

+proc withNumShardsInCluster*(b: var WakuConfBuilder, numShardsInCluster: uint16) =
+  b.numShardsInCluster = some(numShardsInCluster)
+
+proc withActiveRelayShards*(b: var WakuConfBuilder, shards: seq[uint16]) =


I prefer this. More concise ;P

Suggested change

proc withActiveRelayShards*(b: var WakuConfBuilder, shards: seq[uint16]) =

proc withShards*(b: var WakuConfBuilder, shards: seq[uint16]) =

More concise because it does not convey any meaning. Please start to see nwaku as an application library. what shards should a developer select? The answer is none when using autosharding. And for static sharding, this can be done with subscribe API, meaning they are the shards to subscribe to.

Ivansete-status · 2025-07-01T21:06:12Z

waku/factory/conf_builder/waku_conf_builder.nim

  clusterId: Option[uint16]
-  numShardsInNetwork: Option[uint32]
-  shards: Option[seq[uint16]]
+  shardingConf: Option[ShardingConfKind]


According to the spec, Automatic Sharding should be set by default.
Therefore, this should not be an Option and be set to AutoSharding by default.

https://github.com/waku-org/specs/blob/master/standards/core/relay-sharding.md

I am sorry but an Option means that it can be set, or it can be unset.

a default means that if the developer does not set it, then we apply a default value.

Meaning there is default value, does not mean it's the only possible value.

Also well spotted that there is a gap in the spec IMO. It says that auto sharding is the default choice, but does not recommend autosharding parameters such as the number of shards.

Cc @jm-clius: what should be the default number of shards, in the same vein as

Autosharding selects shards automatically and is the default behavior for shard choice

This is because Autosharding is only defined as a concept in this spec and presented as the preferred default option when considering sharding for Waku networks, followed by static (and even named sharding, which we don't even implement anymore). This spec does not define any specific application of autosharding for specific Waku Network(s), as this more properly belongs to the "consensus" specification for a specific network (e.g. the Status Network(s), TWN, etc.). The only network for which we have defined autosharding is TWN, as it's the only specified network with a consensus on cluster ID and number of shards: https://rfc.vac.dev/waku/standards/core/64/network#autosharding.
This out-of-band consensus on a network definition is a must for autosharding to work as expected. The spec should indeed be clarified in this regard.

My initial preference was that autosharding, as an app-level API simplification, should only be available when TWN is configured. However, it became clear for testing purposes and more modular use of Waku, that people preferred having the option of ad-hoc creating their own "network definitions" with autosharding by configuring a numShardsInCluster option. Note that this is a higher-level network definition and has nothing to do at all with which relay shards are actually being subscribed to. You could perfectly well set this parameter and not subscribe to any traffic at all. Also, it implies that these experimenters "know" that others participating in the experiment have configured their nodes in the same way (or are using the default config - see below). The way to formalise this is in the form of published network specifications for each new network, where the number of shards, services, etc. are agreed upon.

Autosharding also only makes sense in the API when applications build on top of a Waku node. In other words, it should only be used when the node is run within specific content topic (i.e. "App") context - it has no use for general service nodes that do not care about specific content topics.

My view of defaults:
Default configuration should always be for TWN.
If someone sets a different cluster ID, the default should be that that cluster is assumed to have only 1 shard. There's no need to pre-emptively "scale" these clusters - we had a specific reason for making TWN experimentally work with 8 out of the box, but that reasoning doesn't apply to the defaults.

Thanks @jm-clius. I actually came to the same conclusion overnight. Which hopefully is in line with @Ivansete-status 's original comment.

With the introduction of the preset concept, we are moving towards TWN being the default. It is not straightforward and as RLN plays a part in it.

I suggest to have nwaku default to autosharding with 1 shard when TWN is not set. For a few reason:

Autosharding means abstracting shards concept, which is better for dev ex

Autosharding is yet to be properly dogfood on large applications, TWN with 8 is a starter but no serious app is running on it. Hence, a value of 1 is low risk.

Thanks for the feedback! I with the suggested approach 👌

Ivansete-status · 2025-07-01T21:09:59Z

waku/factory/conf_builder/waku_conf_builder.nim

+    builder.shardingConf = some(StaticSharding)
+  of AutoSharding:
+    builder.shardingConf = some(AutoSharding)
+    builder.numShardsInCluster = some(networkConf.shardingConf.numShardsInCluster)


This shouldn't be needed

Suggested change

builder.numShardsInCluster = some(networkConf.shardingConf.numShardsInCluster)

What do you mean it should not be needed? The number of shards in the network impact the autosharding protocol. If one node sets 8 and another sets 1024 then a given content topic foo will end up on different shards, meaning you won't be able to transmit messages.

Yes indeed, you are absolutely right. When I wrote my comment I was thinking about having 1024 by default to all nodes but, subscribing to 1024 shards doesn't seem like a great idea because I suspect it won't perform very well.
The static and auto shards generates a lot of confusion and is a source of possible discrepancies among nodes. For example, what happens if someone configures the following?

shards [1, 2, 3]
plus

numShardsInCluster=4

In that case, the foo content-topic can potentially fall in wrong shard (considering all nodes are in 1, 2, 3.)

Therefore, I think we need to infer the value of numShardsInCluster, or have a good default.

Another possible option would be to implement a hybrid solution between static and auto sharding: always have defined the shards list and, in case of using auto-sharding, infer the available shards from that available shards list. wdyt?

Let's take it from a user PoV. I am only covering when auto-sharding is enabled.

Library user/application developer: in this case, I don't think --shards should be used at all. They should use subscribe (contentTopic) to subscribe to a topic.
When sending a message, then we will have two choices when they are not subscribed to the right shard:

non-blocking logic: fallback to light push

blocking logic: error and require subscribe to be done first

Node operator for a given application: In this case, the operator want to support their specific application and run relay/store/etc for their shard. which means they would use --content-topics to subscribe to the right shards

Node operators - for Waku/incentivization/etc: In this instance, a node operator may want to support all shards, or specific shards where there is most traffic. Meaning --shards is most appropriate.

In conclusion

To setup relay shard subscription:

A. On the wakunode2 CLI, it is fine for a user to either both --shards or --content-topic to specify what shards to subscribe to, but it should not be both at the same time. Note that a partial content topic should be accepted (only first fields are needed for autosharding).
B. On the Waku API, then I would suggest that neither --shards nor --content-topic are available at node initialization, but instead, subscribe(contentTopic) is used.

When sending a message, in relay mode, on a shard not subscribed to. I suggest to error out for now. We can decide at a later stage to fallback to light push.

Therefore, I think we need to infer the value of numShardsInCluster, or have a good default.

The default nework conf value should be "TWN", so that this number is 8.
If someone want to roll out their own network, and own cluster id, then I agree we can propose a default value. Probably 1 or 4.
@jm-clius WDYT?

@Ivansete-status just so we are on the same page, you seem to say that numShardsInCluster is not needed, as well as wanting a good default, in lieu of the value.

Ultimately, it is needed because different network may have different shards. For example, we may grown the number of shards on Status network to 50 with the chat sdk. But someone bootstrapping a brand new network may prefer to stick to 1 until scaling is needed. So this parameter is needed.
Then the question of default value remain up to discussion, right?

A. and B. makes sense to me in this comment: #3468 (comment)
The following looks great to me as well:

When sending a message, in relay mode, on a shard not subscribed to. I suggest to error out for now.

If someone want to roll out their own network, and own cluster id, then I agree we can propose a default value. Probably 1 or 4.

Why not 8 too?

I think we need a good first default, 8, and then, define an scaling algorithm that makes the network breath, i.e., subscribe to more or less shards depending on bandwidth needed. That will need a consensus in the network.

For that, I believe we need to also establish a fixed consensus shard on every network, on every cluster. For example, we can enforce that every cluster will have the shard 0 to be reserved for only containing Waku consensus and coordination messages. (Another option is having waku/2/rs/consensus pubsubtopic and leave shard 0 for data.)

That consensus topic will help nodes to reach an agreement in:

Define the total number of shards in the network. Each node will subscribe/unsubscribe accordingly.

Determine which nodes will act as bridge or aggregator between shards.

When to start scaling up/down, i.e., when to start changing the current shard set.

Define the shards a certain node will subscribe to (we can do math considering the peerId.)

Ultimately, it is needed because different network may have different shards.

I think you are absolutely right with that but, on the other hand, I believe we need to implement a network algorithm that makes the network scale up/down accordingly. I strongly suggest following the consensus topic idea that I commented above.

Then the question of default value remain up to discussion, right?

Yes :) I'd suggest having a good default plus an algorithm that makes it adaptive as per network conditions.

I don't think at this stage we need a consensus algorithm to set a network definition. Consensus for the networks we define is reached by publishing an RFC. We have only one defined network, which is TWN for which number of shards is 8. If we need to scale, we'll publish a new version of the spec - TWN Gen 1 if you will - and autosharding is already defined in such a way that it will work with backwards compatibility.

As per #3468 (comment) I will proceed with the following changes:

autosharding by default

num of shards in cluster: 1

Ivansete-status · 2025-07-01T21:12:21Z

waku/factory/conf_builder/waku_conf_builder.nim

+  let shardingConf =
+    if builder.shardingConf.isSome():
+      case builder.shardingConf.get()
+      of StaticSharding:
+        ShardingConf(kind: StaticSharding)
+      of AutoSharding:
+        if builder.numShardsInCluster.isSome():
+          ShardingConf(
+            kind: AutoSharding, numShardsInCluster: builder.numShardsInCluster.get()
+          )
+        else:
+          return err(
+            "Number of shards in cluster must be specified with autosharding enabled"
+          )


I think this can be simplified if we apply good default for numShardsInCluster ( 1024 ) and avoid making shardingConf an Option

ShardingConf MUST remain an enum. Because numShardsInCluster does not make sense for a static sharding conf.

Ivansete-status · 2025-07-01T21:19:21Z

waku/factory/external_config.nim

    numShardsInNetwork* {.
-      desc: "Number of shards in the network",
+      desc:
+        "Enables autosharding and set number of shards in the cluster, set to `0` to use static sharding",


As I mentioned before, this is a great opportunity to remove this parameter :D

Ivansete-status · 2025-07-01T21:22:33Z

waku/factory/networks_config.nim

+  ShardingConf* = object
+    case kind*: ShardingConfKind
+    of AutoSharding:
+      numShardsInCluster*: uint16
+    of StaticSharding:
+      discard


This is beautiful indeed ❤️
Nevertheless, I think we can simplify it by having a good default for numShardsInCluster (1024) in which case, the ShardingConf wouldn't be needed.
Sorry for repeating this same simplification idea multiple times :D

Ivansete-status · 2025-07-01T21:38:53Z

waku/factory/node_factory.nim

+  # TODO: when using autosharding, the user should not be expected to pass any shards, but only content topics
+  # Hence, this joint logic should be removed in favour of an either logic:
+  # use passed shards (static) or deduce shards from content topics (auto)


I do agree with @gabrielmer in this point.

Autosharding seems as a technique for both distributing the traffic randomly across the available shards and to simplify the publish operation.

It seems that we will always have a combination of both static and automatic sharding. For example, TWN, and any other network, would not be able to operate with only automatic sharding, because the content-topic is only available when publishing a message and not when subscribing.

A `NetworkConf` is a Waku network configuration. # Conflicts: # tests/factory/test_waku_conf.nim # Conflicts: # tests/factory/test_waku_conf.nim

A smarter data types simplifies the logic.

it # Conflicts: # waku/factory/external_config.nim

some metrics error reporting were missing # Conflicts: # waku/waku_lightpush/protocol.nim

To make it clearer that these are the shards the node will subscribe to.

Also makes shards to subscribe to all shards in auto sharding, none in static sharding.

fryorcraken · 2025-07-04T06:15:42Z

Will merge when CI is green, we can always revisit with a PR. Description is updated, last change: a0c434f (#3468)

autosharding is enabled by default (when no preset is set), with 1 shard in network
new default for subscribe shards for the CLI
- all for autosharding
- none for static sharding (because it's not possible to define)

subscribe shards default for libwaku can be reviewed as part of the Waku API.

* Reserve `networkconfig` name to waku network related settings * Rename cluster conf to network conf A `NetworkConf` is a Waku network configuration. # Conflicts: # tests/factory/test_waku_conf.nim # Conflicts: # tests/factory/test_waku_conf.nim * Improve sharding configuration A smarter data types simplifies the logic. * Fixing tests * fixup! rename to endpointConf * wip: autosharding is a specific configuration state and treat it like it # Conflicts: # waku/factory/external_config.nim * refactor lightpush handler some metrics error reporting were missing # Conflicts: # waku/waku_lightpush/protocol.nim * test_node_factory tests pass * remove warnings * fix tests * Revert eager previous replace-all command * fix up build tools compilation * metadata is used to store cluster id * Mount relay routes in static sharding * Rename activeRelayShards to subscribeShards To make it clearer that these are the shards the node will subscribe to. * Remove unused msg var * Improve error handling * Set autosharding as default, with 1 shard in network Also makes shards to subscribe to all shards in auto sharding, none in static sharding.

github-actions bot assigned fryorcraken Jun 24, 2025

fryorcraken changed the title ~~Improve sharding configuration~~ chore: improve sharding configuration Jun 26, 2025

fryorcraken force-pushed the waku-api/autosharding-config branch from 9a9092b to b1b3673 Compare June 26, 2025 04:34

fryorcraken commented Jun 26, 2025

View reviewed changes

waku/waku_lightpush/protocol.nim Show resolved Hide resolved

fryorcraken force-pushed the waku-api/autosharding-config branch 2 times, most recently from b19c900 to 4a74835 Compare June 26, 2025 06:22

fryorcraken commented Jun 26, 2025

View reviewed changes

fryorcraken marked this pull request as ready for review June 26, 2025 06:32

fryorcraken requested review from a team and SionoiS June 26, 2025 06:32

This comment was marked as resolved.

Sign in to view

SionoiS approved these changes Jun 26, 2025

View reviewed changes

fryorcraken marked this pull request as draft June 27, 2025 00:04

fryorcraken force-pushed the waku-api/autosharding-config branch from 266ca3e to 03faad4 Compare June 27, 2025 05:20

fryorcraken changed the title ~~chore: improve sharding configuration~~ chore!: make sharding configuration explicit Jun 27, 2025

fryorcraken marked this pull request as ready for review June 27, 2025 06:45

fryorcraken requested a review from a team June 27, 2025 06:45

fryorcraken force-pushed the waku-api/autosharding-config branch from 03faad4 to 1791bca Compare June 30, 2025 04:17

fryorcraken mentioned this pull request Jun 30, 2025

fix: auto-sharding needs to know number of shards in network logos-messaging/logos-delivery-interop-tests#126

Merged

fryorcraken force-pushed the waku-api/autosharding-config branch from 1791bca to d800b5a Compare July 1, 2025 00:02

gabrielmer reviewed Jul 1, 2025

View reviewed changes

gabrielmer self-requested a review July 1, 2025 11:07

gabrielmer approved these changes Jul 1, 2025

View reviewed changes

Ivansete-status reviewed Jul 1, 2025

View reviewed changes

fryorcraken added 17 commits July 3, 2025 12:52

Reserve networkconfig name to waku network related settings

4693d41

Rename cluster conf to network conf

40db9d8

A `NetworkConf` is a Waku network configuration. # Conflicts: # tests/factory/test_waku_conf.nim # Conflicts: # tests/factory/test_waku_conf.nim

Improve sharding configuration

02e0bf5

A smarter data types simplifies the logic.

Fixing tests

b1c2666

fixup! rename to endpointConf

63d2963

wip: autosharding is a specific configuration state and treat it like

102d42e

it # Conflicts: # waku/factory/external_config.nim

refactor lightpush handler

7a54c50

some metrics error reporting were missing # Conflicts: # waku/waku_lightpush/protocol.nim

test_node_factory tests pass

3d5e657

remove warnings

d18ebef

fix tests

ea502ca

Revert eager previous replace-all command

9b33e04

fix up build tools compilation

218ed53

metadata is used to store cluster id

930a836

Mount relay routes in static sharding

06ae88f

Rename activeRelayShards to subscribeShards

7f13699

To make it clearer that these are the shards the node will subscribe to.

Remove unused msg var

63f69b2

Improve error handling

e15046a

fryorcraken force-pushed the waku-api/autosharding-config branch from 81cf25e to e15046a Compare July 3, 2025 02:56

fryorcraken mentioned this pull request Jul 3, 2025

feat!: Use Waku API for libwaku #3490

Closed

2 tasks

Set autosharding as default, with 1 shard in network

a0c434f

Also makes shards to subscribe to all shards in auto sharding, none in static sharding.

fryorcraken mentioned this pull request Jul 4, 2025

feat: CLI args improvements #3349

Closed

12 tasks

fryorcraken merged commit 16bda04 into master Jul 4, 2025
24 of 28 checks passed

fryorcraken deleted the waku-api/autosharding-config branch July 4, 2025 07:10

fryorcraken mentioned this pull request Jul 7, 2025

fix!: encoder does not have enough information to deduce pubsub topic logos-messaging/logos-delivery-js#2448

Closed

5 tasks

weboko mentioned this pull request Jul 14, 2025

chore: fix nwaku interop tests logos-messaging/logos-delivery-js#2486

Closed


		return netConfigRes

		# TODO: numShardsInNetwork should be mandatory with autosharding, and unneeded otherwise

		let msg =
		"Static sharding is used, relay subscriptions must specify a pubsub topic"

	proc withActiveRelayShards*(b: var WakuConfBuilder, shards: seq[uint16]) =
	proc withShards*(b: var WakuConfBuilder, shards: seq[uint16]) =

Conversation

fryorcraken commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Follow-up changes

Issue

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

SionoiS left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fryorcraken commented Jun 27, 2025

Uh oh!

fryorcraken commented Jun 30, 2025

Uh oh!

gabrielmer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fryorcraken Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabrielmer Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fryorcraken Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabrielmer left a comment

Choose a reason for hiding this comment

Uh oh!

Ivansete-status left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fryorcraken commented Jun 24, 2025 •

edited

Loading

github-actions bot commented Jun 26, 2025 •

edited

Loading

fryorcraken Jul 1, 2025 •

edited

Loading

gabrielmer Jul 1, 2025 •

edited

Loading

fryorcraken Jul 2, 2025 •

edited

Loading

fryorcraken Jul 2, 2025 •

edited

Loading