Optimal direct messaging

[lukaszrzasik]

Hi everyone,

We use rust-libp2p in our project for both gossiping and direct messaging. We're trying to optimize as much as possible and have noticed an extra round trip between nodes when delivering a direct message. We use QUIC and the request/response behavior for direct messaging. It seems the extra round trip happens because the request/response behavior creates a separate substream for each request.

Is this understanding correct? Is this the reason for the extra round trip? If so, is there a way to optimize it away? Perhaps we can reconfigure the behavior or use a better one.

Any hints or insights would be very helpful! Thanks!

[dariusc93]

Hey @lukaszrzasik. While I haven't got the chance to look at the code yet, I would assume it would be request-response has 2-RTT (somebody who knows the code more might be able to comment better on it) since it is sending the request and awaiting for a response before closing the substream. However, if you dont need the response side of it, maybe oneshot might be better suited? If not, you could create a new behaviour that could reuse the substream when sending a request, which might be more ideal depending on your use-case and what youre trying to optimize for.

[elenaf9]

We use QUIC and the request/response behavior for direct messaging. It seems the extra round trip happens because the request/response behavior creates a separate substream for each request.

I just double-checked, and yes, with the default config the protocol negotiation for a new substream requires a separate round-trip. We also support upgrade::Version::V1Lazy, which allows to send early application data together with the protocol info. I actually thought this was enabled by default, but seems like V1 is the default.

You can try overriding the upgrade version in swarm::Config::with_substream_upgrade_protocol_override and see if the extra round-trip still happens.

[lukaszrzasik]

Thank you @elenaf9 and @dariusc93 for such quick responses!

V1Lazy solved it! I've run some of our tests and I can definitely confirm that with V1Lazy and connection already established the request / response now takes only one RTT. That's a substantial improvement for us!

Thanks!

[jxs]

Hi @lukaszrzasik I am curious, can you disclose your bandwidth savings switching to V1Lazy and avoid the extra RTT?

[dariusc93]

I just double-checked, and yes, with the default config the protocol negotiation for a new substream requires a separate round-trip. We also support upgrade::Version::V1Lazy, which allows to send early application data together with the protocol info. I actually thought this was enabled by default, but seems like V1 is the default.

I thought so as well. I would have to scan back through previous PRs but maybe it changed when SwarmBuilder was introduced?

[lukaszrzasik]

Hi @lukaszrzasik I am curious, can you disclose your bandwidth savings switching to V1Lazy and avoid the extra RTT?

It's more about latency. We're focused on delivering a message, not so much on receiving a response, so delivering a message is a one way trip now instead of 3. It means sending a message from, for example, the east US cost to the west US coast takes ~30 ms instead of ~90 ms.

[elenaf9]

I thought so as well. I would have to scan back through previous PRs but maybe it changed when SwarmBuilder was introduced?

I just checked, and it seems like V1 was always the Default for Version, but in some cases, like transport upgrade, we already use V1Lazy, e.g. when upgrading tcp with an authentication layer:

rust-libp2p/libp2p/src/builder/phase/tcp.rs

Lines 84 to 89 in 79fc6f1

	transport: libp2p_tcp::$path::Transport::new(tcp_config)
	.upgrade(libp2p_core::upgrade::Version::V1Lazy)
	.authenticate(
	security_upgrade.into_security_upgrade(&self.keypair)?,
	)
	.multiplex(multiplexer_upgrade.into_multiplexer_upgrade())

I'd guess that for network behaviors we don't default to V1Lazy because some networking behaviors support multiple different protocol versions, which won't work with V1Lazy.