Investigate ZombieNet Integration Requirements

[danforbes]

The outcome of this Issue should be a well defined list of the APIs and integrations that Gossamer must implement in order to allow Gossamer nodes to be included in ZombieNet networks and testing pipelines.

[edwardmack]

Notes:

• ZombieNet has two primary functions: spawn and test.
  • Spawn starts ephemeral networks based on configurations defined by the network definition file (toml or json).
  • Test provides a way to conduct test/assertions on the nodes in the networks. Tests are constructed using the ZombieNet DSL which handles interacting with the using polkadot.js api, metrics and logs.

I've started investigating requirements necessary to spawn gossamer nodes. The network definition spec defines attributes for defining a nodes configuration. A network is defines a provider that will host the network: kubernetes, Podman or native. I've started testing using native provider, which is starting the network on local hardware.

When executing the spawn command ZombieNet, the default behavior is to call build-spec to build a raw (machine readable) chain spec (based on chain attribute in network definition) to use when starting the nodes. The attributes sent to the build spec command for polkadot nodes are different than those used with gossamer's build-spec command, so using the default behavior with gossamer nodes does not work. Using the chain_spec_path attribute overrides this behavior by using the chain spec defined by the file set in chain_spec_path and does not call the build spec command. Using this does work with gossamer nodes.

The default_command attribute defines the commands that will be executed to start nodes, the default_command also starts the node with a set of pre-defined default arguments that are passed into the default command. Some of these defaults arguments are arguments that the gossamer node does not recognize, so the gossamer node does not start when using the 'default_command' or 'command' attributes due to errors of unrecognized CLI arguments.

Using the attribute command_with_args solves this issue since using this command there are no default arguments that are included. Only the args that are defined with the attribute are passed to the command when starting.

Using chain_spec_path and command_with_args will start gossamer nodes, however ZombieNet does not realize the nodes have started, and therefore stops the nodes after a one minute timeout period. Starting ZombieNet with debugging logging set (using DEBUG=zombie* ./zombienet-linux-x64 spawn --provider native examples/0001-small-network-gossamer.toml) we can see that ZombieNet uses bash grep to inspect the node's logs searching for regular expression 'Listening for new connections|Running JSON-RPC' to determine if a nodes has started. Since gossamer nodes do not report that when it's started using either of these terms, ZombieNet assumes the nodes has not started. I've created issue #2998 suggesting we update gossamer logs at startup to log one of these terms to the log file.

[edwardmack]

Notes on modifications made to zombienet cmdGenerator to start gossamer nodes:

• defalut flag for polkadot --no-mdns for gossamer its: --nomdns
• default flag for polkadot --node-key for gossamer it's --key
  • the key value passed into this is generated by zombienet: TODO: learn how this works, test
  • currently testing with test values, alice, bob,...
• default flag --prometheus-port doesn't work with gossamer, TODO: determine flags for prometheus
• default flag --rpc-port for gosamer it's --rpcport
• default flag --ws-port for gossamer it's --wsport
• default flag --listen-addr doesn't work with gossamer
• default flag --base-path for gossamer it's --basepath
• default flag --rpc-cors' value all` doesn't work with gossamer
• default flag --unsafe-rpc-external doesn't work with gossamer
• default flag --rpc-methods value unsafe doesn't work with gossamer
• default flag --unsafe-ws-external doesn't work with gossamer
• default flag --validator for gossamer it's --roles value 4

[edwardmack]

Testing DSL with javascript, example script is failing with error -32000: rpc: can't find method "state.QueryStorageAt": null

[edwardmack]

Cross client testing, testing to run gossamer node with polkadot node, current configuration works to start different node, but the nodes do not communicate with each other. Zombienet uses --node-key flag to define lib p2p network addresses for the nodes, then uses that address to build parameters for --bootnodes and --listen-addr (polkadot cli argument to set listen address). Zombienet builds the addresses in the form

/ip4/127.0.0.1/tcp/45475/ws/p2p/12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm

however default local listen addresses are in the form:

/ip4/127.0.0.1/tcp/30333/p2p/12D3KooWRkZhiRhsqmrQ28rt73K7V3aCBpqKrLGSXmZ99PTcTZby

Note that the zombienet created address includes /ws/ after the port number, while the default addresses do not, this addition is handled by the --listen-addr flag since zombienet uses the address with the /ws/ when defining the new listen address. Gossamer nodes do not have the --listen-addr flag, so when the --bootnodes flag with the address that contains /ws/ is used the gossamer nodes are not aware of them.

Implementing the --listen-addr flag to gossamer would solve this issue, or zombienet could be modified to build different boot node addresses for different nodes.