Skip to content

The Circuit Relay Specification: The first iteration #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 10 commits into from
Closed

The Circuit Relay Specification: The first iteration #15

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
63897d6
first pass
daviddias Mar 28, 2017
4c5a51c
update addresses examples
daviddias Mar 28, 2017
b74a39e
add error table
daviddias Mar 28, 2017
5ad252d
update wire format section with why notes
daviddias Mar 28, 2017
d65d541
Feat/update relay spec (#16)
dryajov Mar 30, 2017
5105b41
reviewed Overview, Dramatization and Addressing
daviddias May 8, 2017
9ca982b
apply lars CR
daviddias May 9, 2017
1759516
review Wire protocol section from the relay spec
daviddias May 9, 2017
76dfb72
Merge pull request #18 from libp2p/review/relay/wire-protocol
daviddias Jul 8, 2017
a758d06
Merge branch 'master' into feat/update-relay-spec
daviddias Jul 8, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
267 changes: 168 additions & 99 deletions relay/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,69 +1,51 @@
# Circuit Relay

> Circuit Switching in libp2p
> Circuit Switching for libp2p, also known as TURN or Relay in Networking literature.

Implementations
- [js-libp2p-circuit](https://github.com/libp2p/js-libp2p-circuit) -- status: started
- [go-libp2p-circuit](https://github.com/libp2p/go-libp2p-circuit) -- status: ready-to-start
## Implementations

- [js-libp2p-circuit](https://github.com/libp2p/js-libp2p-circuit)
- [go-libp2p-circuit](https://github.com/libp2p/go-libp2p-circuit)

## Table of Contents

Table of Contents
- [Overview](#overview)
- [Dramatization](#dramatization)
- [Addressing](#addressing)
- [Wire protocol](#wire-protocol)
- [Interfaces](#interfaces)
- [Implementation Details](#implementation-details)
- [Removing existing relay protocol](#removing-existing-relay-protocol)

Note: this document is currently in the process of being double-checked and updated. Please see [libp2p/specs#15 (comment)](https://github.com/libp2p/specs/pull/15#issuecomment-299949023) before using it as the basis for an implementation.

Note: this document is currently in the process of being double-checked and updated. Please see [libp2p/specs#15 (comment)](https://github.com/libp2p/specs/pull/15) before using it as the basis for an implementation.

## Overview

The circuit relay is a means of establishing connectivity between
libp2p nodes (such as IPFS) that wouldn't otherwise be able to connect to each other.
The circuit relay is a means of establishing connectivity between libp2p nodes (such as IPFS) that wouldn't otherwise be able to establish a direct connection to each other.

This helps in situations where nodes are behind NAT or reverse proxies,
or simply don't support the same transports (e.g. go-ipfs vs. browser-ipfs).
libp2p already has modules for NAT ([go-libp2p-nat](https://github.com/libp2p/go-libp2p-nat)),
but these don't always do the job, just because NAT traversal is complicated.
That's why it's useful to have a simple relay protocol.
This helps in situations where nodes are behind NAT or reverse proxies, or simply don't support the same transports (e.g. go-ipfs vs. browser-ipfs). libp2p already has modules for NAT ([go-libp2p-nat](https://github.com/libp2p/go-libp2p-nat)), however piercing through NATs is not always an option due to their implementation differences. The circuit relay protocol exists to overcome those scenarios.

Unlike a transparent **tunnel**, where a libp2p peer would just proxy a
communication stream to a destination (the destination being unaware of the
original source), a circuit-relay makes the destination aware of the original
source and the circuit followed to establish communication between the two.
This provides the destination side with full knowledge of the circuit which,
if needed, could be rebuilt in the opposite direction.
Unlike a transparent **tunnel**, where a libp2p peer would just proxy a communication stream to a destination (the destination being unaware of the original source), a circuit-relay makes the destination aware of the original source and the circuit followed to establish communication between the two. This provides the destination side with full knowledge of the circuit which, if needed, could be rebuilt in the opposite direction.

Apart from that, this relayed connection behaves just like a regular
connection would, but over an existing swarm stream with another peer
(instead of e.g. TCP.): One node asks a relay node to connect to another node
on its behalf. The relay node shortcircuits its streams to the two nodes,
and they are then connected through the relay.
Apart from that, this relayed connection behaves just like a regular connection would, but over an existing swarm stream with another peer (instead of e.g. TCP.): One node asks a relay node to connect to another node on its behalf. The relay node shortcircuits its streams to the two nodes, and they are then connected through the relay.

Relayed connections are end-to-end encrypted just like regular connections.

The circuit relay is both a tunneled transport and a mounted swarm protocol.
The transport is the means of ***establishing*** and ***accepting*** connections,
and the swarm protocol is the means to ***relaying*** connections.
The transport is the means of ***establishing*** and ***accepting*** connections, and the swarm protocol is the means to ***relaying*** connections.

```
+-------+ /ip4/.../tcp/.../ws/p2p/QmRelay +---------+ /ip4/.../tcp/.../p2p/QmTwo +-------+
| QmOne | <------------------------------------> | QmRelay | <-----------------------------------> | QmTwo |
+-------+ (/libp2p/relay/circuit multistream) +---------+ (/libp2p/relay/circuit multistream) +-------+
^ +-----+ ^
| | | |
| /p2p-circuit/QmTwo | | |
+--------------------------------------------+ +-------------------------------------------+
+-----+ /ip4/.../tcp/.../ws/p2p/QmRelay +-------+ /ip4/.../tcp/.../p2p/QmTwo +-----+
|QmOne| <------------------------------------>|QmRelay|<----------------------------------->|QmTwo|
+-----+ (/libp2p/relay/circuit multistream) +-------+ (/libp2p/relay/circuit multistream) +-----+
^ +-----+ ^
| /p2p-circuit/QmTwo | | |
+-----------------------------------------+ +-----------------------------------------+
```

Note: we're using the `/p2p` multiaddr protocol instead of `/ipfs` in this document.
`/ipfs` is currently the canonical way of addressing a libp2p or IPFS node,
but given the growing non-IPFS usage of libp2p, we'll migrate to using `/p2p`.

Note: at the moment we're not including a mechanism for discovering relay nodes.
For the time being, they should be configured statically.
## Notes for the reader

1. We're using the `/p2p` multiaddr protocol instead of `/ipfs` in this document. `/ipfs` is currently the canonical way of addressing a libp2p or IPFS node, but given the growing non-IPFS usage of libp2p, we'll migrate to using `/p2p`.

## Dramatization

Expand All @@ -86,91 +68,173 @@ Scene 2:
- QmOne tries to connect via relaying, because it shares this transport with QmTwo.
- A lively and prolonged dialogue ensues.


## Addressing

`/p2p-circuit` multiaddrs don't carry any meaning of their own.
They need to encapsulate a `/p2p` address, or
be encapsulated in a `/p2p` address, or both.
`/p2p-circuit` multiaddrs don't carry any meaning of their own. They need to encapsulate a `/p2p` address, or be encapsulated in a `/p2p` address, or both.

As with all other multiaddrs, encapsulation of different protocols determines which metaphorical tubes to connect to each other.

A `/p2p-circuit` circuit address, is formated following:

`[<relay peer multiaddr>]/p2p-circuit/<destination peer multiaddr>`

As with all other multiaddrs, encapsulation of different protocols
determines which metaphorical tubes to connect to each other.
Examples:

- `/p2p-circuit/p2p/QmVT6GYwjeeAF5TR485Yc58S3xRF5EFsZ5YAF4VcP3URHt` - Arbitrary relay node
- `/ip4/127.0.0.1/tcp/5002/ipfs/QmdPU7PfRyKehdrP5A3WqmjyD6bhVpU1mLGKppa2FjGDjZ/p2p-circuit/p2p/QmVT6GYwjeeAF5TR485Yc58S3xRF5EFsZ5YAF4VcP3URHt` - Specific relay node

This opens the room for multiple hop relay, where the second relay is encapsulated in the first relay multiaddr, such as:

`<1st relay>/p2p-circuit/<2nd relay>/p2p-circuit/<dst multiaddr>`

A few examples:

Using any relay available:

- `/p2p-circuit/p2p/QmTwo`
- Dial QmTwo, through any available relay node.
- The relay node will use peer routing to find an address for QmTwo.
- Dial QmTwo, through any available relay node (or find one node that can relay).
- The relay node will use peer routing to find an address for QmTwo if it doesn't have a direct connection.
- `/p2p-circuit/ip4/../tcp/../p2p/QmTwo`
- Dial QmTwo, through any available relay node, but force the relay node to use the encapsulated `/ip4` multiaddr for connecting to QmTwo.

Specify a relay:

- `/p2p/QmRelay/p2p-circuit/p2p/QmTwo`
- Dial QmTwo, through QmRelay.
- Use peer routing to find an address for QmRelay.
- The relay node will also use peer routing, to find an address for QmTwo.
- `/p2p-circuit/ip4/../tcp/../p2p/QmTwo`
- Dial QmTwo, through any available relay node,
but force the relay node to use the encapsulated `/ip4` multiaddr for connecting to QmTwo.
- We'll probably not support forced addresses for now, just because it's complicated.
- `/ip4/../tcp/../p2p/QmRelay/p2p-circuit`
- Listen for connections relayed through QmRelay.
- Includes info for connecting to QmRelay.
- Also makes QmRelay available for relayed dialing, based on how listeners currently relate to dialers.
- `/p2p/QmRelay/p2p-circuit`
- Same as previous example, but use peer routing to find an address for QmRelay.
- `/p2p-circuit/p2p/QmTwo/p2p-circuit/p2p/QmThree`
- Dial QmThree, through a relayed connection to QmTwo.
- The relay nodes will use peer routing to find an address for QmTwo and QmThree.
- We'll probably not support nested relayed connections for now, there are edge cases to think of.
- `/ip4/../tcp/../p2p/QmRelay/p2p-circuit/p2p/QmTwo`
- Dial QmTwo, through QmRelay.
- Includes info for connecting to QmRelay.
- The relay node will use peer routing to find an address for QmTwo.
- `/p2p-circuit`
- Use relay discovery to find a suitable relay node. (Neither specified nor implemented.)
- Listen for relayed connections.
- Dial through the discovered relay node for any `/p2p-circuit` multiaddr.

TODO: figure out forced addresses.
TODO: figure out nested relayed connections.
Double relay:

- `/p2p-circuit/p2p/QmTwo/p2p-circuit/p2p/QmThree`
- Dial QmThree, through a relayed connection to QmTwo.
- The relay nodes will use peer routing to find an address for QmTwo and QmThree.
- We'll probably not support nested relayed connections for now, there are edge cases to think of.

## Wire format

The wire format (or codec) is named `/ipfs/relay/circuit` and is simple.
A variable-length header consisting of two length-prefixed multiaddrs
is followed by a bidirectional stream of arbitrary data,
and the eventual closing of the stream.
We start the description of the Wire format by illustrating a possible flow scenario and then describing them in detail by phases.

### High level overview of establishing a relayed connection

Setup:
- Peers involved, A, B, R
- A wants to connect to B, but needs to relay through R

Assumptions:
- A has connection to R, R has connection to B

Events:
- phase I: Open a request for a relayed stream (A to R)
- A dials a new stream `sAR` to R using protocol `/libp2p/relay/hop`
- A writes Bs multiaddr `/p2p/QmB` on `sAR`
- R receives stream `sAR` and reads `/p2p/QmB` from it.
- phase II: Open a stream to be relayed (R to B)
- R opens a new stream `sRB` to B using protocol `/libp2p/relay/stop`
- R writes `/p2p/QmA` on `sRB`
- R writes OK to `sAR`
- phase III: Streams are piped together, establishing a circuit
- B receives stream `sRB` and reads `/p2p/QmA` from it.
- B writes OK to `sRB`
- B passes stream to `NewConnHandler` to be handled like any other new incoming connection

### Under the microscope

Notes for the reader:
- `maddrA` is peer A's multiaddr
- `maddrR` is peer R's multiaddr
- `maddrB` is peer B's multiaddr
- `maxAddrLen` is arbitrarily 1024
- `readLpMaddr` is a function that does the following:

Read a Uvarint `V` from the stream. If the value is higher than `maxAddrLen`, writes an error code back, closes the stream and halts the relay attempt. If not, then reads `V` bytes from the stream and checks if its a valid multiaddr. Check for multiaddr validity and error back if not a valid formated multiaddr.

#### Phase I

Peer A opens a new stream to R on the '/libp2p/relay/hop' protocol and writes:

```
<uvarint len(maddrA)><madddrA><uvarint len(maddrB)><madddrB>
```

It then waits for a response in the form of:
```
<uvarint error code><uvarint msglength><message>
```

Once it receives that, it checks if the status code is `OK`. If it is, it passes the new connection to its `NewConnHandler`. Otherwise, it returns the error message to the caller.

Peer R receives a new stream on the '/libp2p/relay/hop' protocol.

It then calls `readLpMaddr` twice. The first value is `<src>` and the second is `<dst>`.

Peer R checks to make sure that `<src>` matches the remote peer of the stream its reading from. If it does not match, it writes an Error message, closes the stream and halts the relay attempt.

Peer R checks if `<dst>` refers to itself, if it does, it (writes an error back?) closes the stream and halts the relay attempt.

Peer R then checks if it has an open connection to the peer specified by `<dst>`.
If it does not, and the relay is not an "active" relay it (writes an error back) closes the stream, and halts the relay attempt.

If R does not have a connection to `<dst>`, and it *is* an "active" relay, it attempts to connect to `<dst>`.

If this connection succeeds it continues, otherwise it (writes back an error) closes the stream, and halts the relay attempt.


#### Phase II

R now opens a new stream to B with the '/libp2p/relay/stop' relay protocol multicodec and writes:

```
<uvarint len(maddrA)><madddrA><uvarint len(maddrB)><madddrB>
```

After this, R pipes the stream from A and the stream it opened to B together. R also writes OK back to A. R's job is complete.

#### Phase III

Peer B receives a new stream on the '/libp2p/relay/stop' protocol. It then calls `readLpMaddr` twice on this stream.

The first value is `<src>` and the second value is `<dst>`. Any error from those calls should be written back accordingly.

B now verifies that `<dst>` matches its peer ID. It then also checks that `<src>` is valid. It uses src as the
'remote addr' of the new 'incoming relay connection' it will create.

Peer B now writes back a message of the form:
```
<src><dst><data>

^ ^ ^
| | |
| | +-- bidirectional data stream
| | (usually /multistream-select in the case of /p2p multiaddrs)
| |
| +------- multiaddr of the listening node
|
+------------ multiaddr of the dialing node
<uvarint 'OK'><uvarint len(msg)><string "OK">
```

After getting a stream to the relay node from its libp2p swarm,
the dialing transport writes the header to the stream.
The relaying node reads the header, gets a stream to the destination node,
then writes the header to the destination stream and shortcircuits the two streams.
And passes the relayed connection into its `NewConnHandler`.

### Status codes table

Each relayed connection corresponds to two multistreams,
one between QmOne and QmRelay, the other between QmRelay and QmTwo.
This is a table of status codes and sample messages that may occur during a relay setup. Codes in the 200 range are returned by the relay node. Codes in the 300 range are returned by the destination node.

Implementation details:
- The relay node has the `Swarm.EnableRelaying` config option enabled
- The relay node allows only one relayed connection between any two nodes.
- The relay node validates the `src` header field.
- The listening node validates the `dst` header field.

| Code | Message | Meaning |
| ----- |:--------------------------------------------------|:----------:|
| 100 | OK | Relay was setup correctly |
| 220 | "src address too long" | |
| 221 | "dst address too long" | |
| 250 | "failed to parse src addr: no such protocol ipfs" | The `<src>` multiaddr in the header was invalid |
| 251 | "failed to parse dst addr: no such protocol ipfs" | The `<dst>` multiaddr in the header was invalid |
| 260 | "passive relay has no connection to dst" | |
| 261 | "active relay could not connect to dst: connection refused" | relay could not form new connection to target peer |
| 262 | "could not open new stream to dst: BAD ERROR" | relay has connection to dst, but failed to open a new stream |
| 270 | "<dst> does not support relay" | |
| 280 | "can't relay to itself" | The relay got its own address as destination |
| 320 | "src address too long" | |
| 321 | "dst address too long" | |
| 350 | "failed to parse src addr: no such protocol ifps" | The `<src>` multiaddr in the header was invalid |
| 351 | "failed to parse dst addr: no such protocol ifps" | The `<dst>` multiaddr in the header was invalid |

## Interfaces

As explained above, the relay is both a transport (`tpt.Transport`)
and a mounted stream protocol (`p2pnet.StreamHandler`).
In addition it provides a means of specifying relay nodes to listen/dial through.
As explained above, the relay is both a transport (`tpt.Transport`) and a mounted stream protocol (`p2pnet.StreamHandler`). In addition it provides a means of specifying relay nodes to listen/dial through.

TODO: the usage of p2pnet.StreamHandler is a little bit off, but it gets the point across.

Expand All @@ -194,8 +258,9 @@ type CircuitRelay interface {
fund NewCircuitRelay(h p2phost.Host)
```

## Implementation details

### Removing existing relay protocol
### Removing existing relay protocol in Go

Note that there is an existing swarm protocol colloqiually called relay.
It lives in the go-libp2p package and is named `/ipfs/relay/line/0.1.0`.
Expand All @@ -210,5 +275,9 @@ It lives in the go-libp2p package and is named `/ipfs/relay/line/0.1.0`.
- Capable of *accepting* connections, and *relaying* connections.
- Not capable of *connecting* via relaying.

Since the existing protocol is incomplete, insecure, and certainly not used,
we can safely remove it.
Since the existing protocol is incomplete, insecure, and certainly not used, we can safely remove it.

## Future work

- Multihop relay - With this specification, we are only enabling single hop relays to exist. Multihop relay will come at a later stage as Packet Switching.
- Relay discovery mechanism - At the moment we're not including a mechanism for discovering relay nodes. For the time being, they should be configured statically.