Preloader gateways for js-ipfs

[daviddias]

e.g. through DNS?

No. There's currently four, in three locations, subject to change. Why?

[daviddias]

I'm looking for a way to know which Gateways exist so that js-ipfs can try to connect to them directly or through Circuit Relay. This will help us ensure that Circuit Relay is fully ready.

Hrm I guess we can create DNS records. These hosts can come and go at any time though, and even change their PeerID.

Is this something that DHT interop or at least delegated routing would solve?

[daviddias]

DNS records would be stellar. DHT interop and/or delegated routing would contribute to the solution as well.

[daviddias]

@lgierth Here is a more complete description of the idea.

Create a Peer Discovery module, named js-libp2p-railing-gateways, that grabs gateway.ipfs.io, resolves the DNS to check the TXT records for dnsaddr= to find the WebSockets and/or Circuit Relay Addrs for the online Gatways so that it can connect to them.

This is solution/step 1 of a 3 part plan to get full dynamic Peer & Content Routing -- ipfs/js-ipfs#1459 --, but getting the experience to users better right away so that we don't get this ipfs/js-ipfs#1459

The plan in ipfs/js-ipfs#1459 is a bit thin on details. The drawings might make perfect sense to someone who attended that meeting, but it's a bit incoherent to me :)

The browser node would have to connect to all gateway nodes (currently 8, more coming) -- that part is probably fine I guess. The actual problem though is that the PeerIDs of those gateway nodes change all the time. From a gateway architecture/scaling/resilience standpoint, we do not want to introduce stable PeerIDs that people start relying on.

---

Anyway, I'm not terribly happy getting back into these workarounds. Whenever we've gotten into these short-term "urgent" workarounds in the past, we've mistakenly put a checkmark on the issue and moved on, and a few months later it re-emerged as a big urgent fire. Thinking of websocket-star here, and of the old always-connected-to-gateways workaround that you're asking to reinstate here. That workaround was a huge pain on the infra side, and has blocked neccessary stability and performance improvements for months, until I finally pulled the plug on the workaround (with your consent).

We desperately need DHT interop -- who is working on that currently? It's not the workarounds that are urgently needed, it's DHT interop. Only if DHT interop work is blocked, the workarounds should be considered.

Ideally, browser nodes connect to well-known relay nodes (e.g. dnsaddr relay.libp2p.io) and announce their via-relay addresses to peer routing, be it DHT or delegated.

---

Sidenote: what about the delegated-routing plan (libp2p/js-libp2p#120) we agreed on in January?

Let's discuss sync on Monday or Tuesday? I'm very interested in resolving this issue complex. It'll be nice to have one thing less to worry about in the big connectivity category of things.

[daviddias]

we do not want to introduce stable PeerIDs that people start relying on.

Totally. The DNS addr can have the TXT record with the PeerIds as well so that it doesn't expect those to be fixed.

DHT interop -- who is working on that currently?

This is a multiple people effort and one that takes time given that it is interop between 2 IPFS implementations. For example, there were plenty of go-multiplex fixes that we had to wait 6++ weeks to see deployed and now that they have been released, we discovered yet another issue that a fix will only be released on 0.4.17, which relates to #415.

As for who owns it on the JS side, you can see on the OKRs spreadsheet. Here are screenshots

On libp2p

On IPFS

@hugomrdias is primarily focused on getting the "Connectivity Magic" feeling, which basically includes all of the 3 steps/solutions presented on ipfs/js-ipfs#1459

Anyway, I'm not terribly happy getting back into these workarounds.

I wouldn't call it a workaround and I can explain more Monday based on ipfs/js-ipfs#1459. IPFS should be able to be clever and try to connect in different scenarios, ideally automatically but sometimes it will require some cleverness to work around things.

We find ourselves fighting against time given that only now we have people fully focused on the Peer & Content Routing but the reality is that Circuit Relay can't yet be fully trusted until ipfs/interop has all tests enabled when go-ipfs 0.4.17 comes out.

That said, I'm confident that we can finish it all this quarter given the amount of eyeballs and the reduced OKR list that is focused on specifically this problems. Until then, let's give a smooth transition to the users.