Skip to content

Commit 7a67642

Browse files
vasco-santosvasco-santos
committed
chore: add benchmarks
1 parent 8baf618 commit 7a67642

File tree

13 files changed

+675
-26
lines changed

13 files changed

+675
-26
lines changed

‎.aegir.js

Lines changed: 4 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,12 @@
33
const Libp2p = require('libp2p')
44
const { MULTIADDRS_WEBSOCKETS } = require('./test/fixtures/browser')
55
const Peers = require('./test/fixtures/peers')
6+
const docker = require('./mysql-local/docker')
67
const PeerId = require('peer-id')
78
const WebSockets = require('libp2p-websockets')
89
const Muxer = require('libp2p-mplex')
910
const { NOISE: Crypto } = require('libp2p-noise')
1011

11-
const delay = require('delay')
12-
const execa = require('execa')
13-
const pWaitFor = require('p-wait-for')
1412
const isCI = require('is-ci')
1513

1614
let libp2p
@@ -50,34 +48,17 @@ const before = async () => {
5048
return
5149
}
5250

53-
const procResult = execa.commandSync('docker run -p 3306:3306 -e MYSQL_ROOT_PASSWORD=test-secret-pw -e MYSQL_DATABASE=libp2p_rendezvous_db -d mysql:8 --default-authentication-plugin=mysql_native_password', {
54-
all: true
55-
})
56-
containerId = procResult.stdout
57-
58-
console.log(`wait for docker container ${containerId} to be ready`)
59-
60-
await pWaitFor(() => {
61-
const procCheck = execa.commandSync(`docker logs ${containerId}`)
62-
const logs = procCheck.stdout + procCheck.stderr // Docker/MySQL sends to the stderr the ready for connections...
63-
64-
return logs.includes('ready for connections')
65-
}, {
66-
interval: 5000
67-
})
68-
// Some more time waiting to guarantee the container is really ready
69-
await delay(12e3)
51+
containerId = await docker.start()
7052
}
7153

7254
const after = async () => {
7355
await libp2p.stop()
7456

75-
if (isCI) {
57+
if (isCI || !containerId) {
7658
return
7759
}
7860

79-
console.log('docker container is stopping')
80-
execa.commandSync(`docker stop ${containerId}`)
61+
docker.stop(containerId)
8162
}
8263

8364
module.exports = {

‎benchmarks/README.md

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
# Rendezvous benchmarks
2+
3+
This benchmark contains a simulator to stress test a rendezvous server and gather performance metrics from it.
4+
5+
## Running
6+
7+
For running the benchmarks, it is required to install the dependencies of the `libp2p-rendezvous`, as well as Docker. With those installed, you only need to run the `index.js` file as follows:
8+
9+
```sh
10+
$ npm install
11+
$ cd benchmarks
12+
$ node index.js
13+
```
14+
15+
While default values exist for benchmarking, you can use CLI parameters to configure how to run the benchmark.
16+
17+
It is worth mentioning that this benchmark runner will be stressing a rendezvous server running in a separate process. It will run the configured number of libp2p client nodes in parallel, including their Rendezvous operations. As a result, a massive number of clients might degrade the overall performance of the clients as they will all be running in the same machine and process.
18+
19+
Network Latency is not considered in this benchmark. The benchmark focus on sending rendezvous requests over the wire through local connections.
20+
21+
### Configuration
22+
23+
```sh
24+
// Usage: $0 [--nClients <number>] [--nNamespaces <number>] [--initialRegistrations <number>]
25+
// [--benchmarkRuns <number>] [--benchmarkType <TYPE>] [--outputFile <path>]
26+
// [--discoverLimit <number>] [--discoverInexistentNamespaces]
27+
```
28+
29+
### Metrics
30+
31+
The metrics that can be obtained from this benchmark setup are the following:
32+
33+
- Operations {Register, Discover}
34+
- Average response time
35+
- Maximum response time
36+
- Median response time
37+
- Server performance
38+
- CPU
39+
- Memory
40+
41+
The Response Times (RT) metrics are measured in milliseconds while the Memory (Mem) metrics are measured in MB. CPU usage is a % value.
42+
43+
## Created Performance testing scenarios
44+
45+
There are a few considerations that we need to have before observing the results:
46+
47+
- Massive number of clients might degrade the overall performance of the clients as they will all be running in the same machine and process.
48+
- Response times will be influenced by Node's event loop as a large number of asynchronous operations will happen on the client side.
49+
- Number of connections open will influence the overall memory consumption, specially with a large number of parallel operations
50+
- In a real world scenario, connections will be open and closed per Rendezvous operation, while this benchmark kept them open for faster results.
51+
52+
To ease performance evaluation on this repo, a benchmark shell script was created for running several combinations of inputs in the benchmark, according to the tables below.
53+
54+
### Register
55+
56+
Measure adding n registrations. Each operation in the following table
57+
58+
| Type | Clients | Io registrations | Operations | Namespaces |
59+
|------|---------|------------------|------------|------------|
60+
| `Register` | 5 | 0 | 500 | 10 |
61+
| `Register` | 5 | 1000 | 500 | 10 |
62+
| `Register` | 10 | 1000 | 500 | 10 |
63+
| `Register` | 100 | 1000 | 500 | 10 |
64+
| `Register` | 100 | 1000 | 1000 | 10 |
65+
| `Register` | 100 | 10000 | 500 | 10 |
66+
| `Register` | 100 | 10000 | 1000 | 10 |
67+
| `Register` | 50 | 100000 | 500 | 10 |
68+
| `Register` | 50 | 100000 | 1000 | 10 |
69+
| `Register` | 100 | 100000 | 500 | 10 |
70+
| `Register` | 100 | 100000 | 1000 | 10 |
71+
| `Register` | 200 | 100000 | 500 | 10 |
72+
| `Register` | 200 | 100000 | 1000 | 10 |
73+
| `Register` | 200 | 200000 | 1000 | 10 |
74+
75+
### Discover
76+
77+
1. Measure discover existing registrations in series with limit of 20
78+
79+
| Type | Clients | Io registrations | Operations | Namespaces |
80+
|------|---------|------------------|------------|------------|
81+
| `Discover` | 5 | 1000 | 500 | 10 |
82+
| `Discover` | 5 | 1000 | 500 | 100 |
83+
| `Discover` | 10 | 10000 | 500 | 10 |
84+
| `Discover` | 10 | 10000 | 500 | 100 |
85+
| `Discover` | 10 | 10000 | 1000 | 10 |
86+
| `Discover` | 10 | 10000 | 1000 | 100 |
87+
| `Discover` | 100 | 100000 | 500 | 10 |
88+
| `Discover` | 100 | 100000 | 500 | 100 |
89+
90+
2. Measure discover existing registrations in series with limit of 100
91+
92+
| Type | Clients | Io registrations | Operations | Namespaces |
93+
|------|---------|------------------|------------|------------|
94+
| `Discover` | 5 | 1000 | 500 | 10 |
95+
| `Discover` | 5 | 1000 | 500 | 100 |
96+
| `Discover` | 10 | 10000 | 500 | 10 |
97+
| `Discover` | 10 | 10000 | 500 | 100 |
98+
| `Discover` | 10 | 10000 | 1000 | 10 |
99+
| `Discover` | 10 | 10000 | 1000 | 100 |
100+
| `Discover` | 100 | 100000 | 500 | 10 |
101+
| `Discover` | 100 | 100000 | 500 | 100 |
102+
103+
3. Measure trying to discover peers on inexistent namespaces.
104+
105+
| Type | Clients | Io registrations | Operations | Namespaces |
106+
|------|---------|------------------|------------|------------|
107+
| `Discover` | 5 | 0 | 500 | 10 |
108+
| `Discover` | 5 | 0 | 1000 | 10 |
109+
| `Discover` | 10 | 0 | 1000 | 10 |
110+
| `Discover` | 10 | 0 | 1000 | 100 |
111+
| `Discover` | 100 | 0 | 10000 | 10 |
112+
| `Discover` | 100 | 0 | 10000 | 100 |
113+
| `Discover` | 10 | 10000 | 10000 | 100 |
114+
| `Discover` | 100 | 10000 | 10000 | 100 |
115+
| `Discover` | 10 | 100000 | 10000 | 100 |
116+
| `Discover` | 100 | 100000 | 10000 | 100 |
117+
118+
### Results obtained
119+
120+
Running in a Macbook with 2.6 GHz 6-Core Intel Core i7 and 16 GB 2400 MHz DDR4.
121+
122+
The Response Times (RT) metrics are measured in milliseconds while the Memory (Mem) metrics are measured in MB. CPU usage is a % value.
123+
124+
**Register**
125+
126+
| Type | Clients | Io Reg | Namespaces | Ops | Avg RT | Median RT | Max RT | Avg CPU | Median CPU | Max CPU | Avg Mem | Median Mem | Max Mem |
127+
|----------|---------|--------|------------|-----|--------|-----------|--------|---------|------------|---------|---------|------------|---------|
128+
| REGISTER | 5 | 100 | 10 | 500 | 16 | 15 | 26 | 31 | 42 | 61 | 98 | 98 | 106 |
129+
| REGISTER | 5 | 1000 | 10 | 500 | 14 | 14 | 26 | 34 | 37 | 67 | 113 | 112 | 114 |
130+
| REGISTER | 10 | 1000 | 10 | 500 | 23 | 22 | 55 | 34 | 23 | 60 | 143 | 141 | 149 |
131+
| REGISTER | 100 | 1000 | 10 | 500 | 221 | 224 | 308 | 23 | 22 | 48 | 133 | 132 | 136 |
132+
| REGISTER | 100 | 1000 | 10 | 1000 | 276 | 277 | 410 | 26 | 41 | 54 | 138 | 134 | 165 |
133+
| REGISTER | 100 | 10000 | 10 | 500 | 1900 | 282 | 8468 | 15 | 18 | 75 | 364 | 362 | 382 |
134+
| REGISTER | 100 | 10000 | 10 | 1000 | 1061 | 292 | 8017 | 15 | 19 | 76 | 393 | 392 | 397 |
135+
| REGISTER | 50 | 100000 | 10 | 500 | 23686 | 358 | 57365 | 11 | 8 | 88 | 2341 | 2334 | 2645 |
136+
| REGISTER | 50 | 100000 | 10 | 1000 | 10055 | 425 | 56977 | 10 | 0 | 89 | 2501 | 2543 | 2887 |
137+
| REGISTER | 100 | 100000 | 10 | 500 | 45273 | 674 | 55370 | 11 | 6 | 95 | 2691 | 2718 | 2787 |
138+
| REGISTER | 100 | 100000 | 10 | 1000 | 22849 | 870 | 55060 | 11 | 11 | 88 | 2166 | 2225 | 2522 |
139+
| REGISTER | 200 | 100000 | 10 | 500 | 24572 | 2456 | 108989 | 10 | 10 | 90 | 2468 | 2476 | 2655 |
140+
| REGISTER | 200 | 100000 | 10 | 1000 | 61448 | 2069 | 299530 | 10 | 3 | 87 | 2515 | 2545 | 2742 |
141+
| REGISTER | 200 | 200000 | 10 | 1000 | 168163 | 2485 | 830998 | 12 | 3 | 93 | 2814 | 2896 | 3286 |
142+
143+
The median Response Time keeps a value below 1000 milliseconds when the server is interacting in parallel with 100 (or less) clients. It increases with the increase of clients interacting in parallel. These results are also affected by running 200 clients in the same process/machine and would probably be better when running in different machines as the Event Loop would be considerably more available to each client.
144+
145+
As expected, with the increase of clients connected to a single server doing multiple Register operations in parallel the memory consumption increase. Regarding CPU usage, except for some spikes over time, the average and median usage is low.
146+
147+
**Discover with limit of 20**
148+
149+
| Type | Clients | Io Reg | Namespaces | Ops | Avg RT | Median RT | Max RT | Avg CPU | Median CPU | Max CPU | Avg Mem | Median Mem | Max Mem |
150+
|----------|---------|--------|------------|-----|--------|-----------|--------|---------|------------|---------|---------|------------|---------|
151+
| DISCOVER | 5 | 1000 | 10 | 500 | 4 | 4 | 12 | 24 | 34 | 59 | 115 | 115 | 116 |
152+
| DISCOVER | 5 | 1000 | 100 | 500 | 5 | 5 | 17 | 29 | 32 | 66 | 114 | 113 | 115 |
153+
| DISCOVER | 10 | 10000 | 10 | 500 | 144 | 7 | 5129 | 17 | 20 | 67 | 332 | 330 | 359 |
154+
| DISCOVER | 10 | 10000 | 100 | 500 | 177 | 7 | 6505 | 17 | 9 | 88 | 367 | 369 | 370 |
155+
| DISCOVER | 10 | 10000 | 10 | 1000 | 80 | 7 | 5721 | 15 | 9 | 86 | 330 | 330 | 331 |
156+
| DISCOVER | 10 | 10000 | 100 | 1000 | 94 | 7 | 6437 | 16 | 18 | 84 | 354 | 353 | 397 |
157+
| DISCOVER | 100 | 100000 | 10 | 500 | 26379 | 118 | 112840 | 10 | 0 | 92 | 2009 | 2045 | 2204 |
158+
| DISCOVER | 100 | 100000 | 100 | 500 | 30609 | 139 | 123132 | 11 | 8 | 93 | 2229 | 2276 | 2416 |
159+
160+
Like in the Register Response Times, the median response Times are fairly low. But, as more and more requests accumulate and the benchmark process Event Loop cannot handle efficiently all the client responses. In addition, memory and CPU usage also increased as more and more clients.
161+
162+
**Discover with limit of 100**
163+
164+
| Type | Clients | Io Reg | Namespaces | Ops | Avg RT | Median RT | Max RT | Avg CPU | Median CPU | Max CPU | Avg Mem | Median Mem | Max Mem |
165+
|----------|---------|--------|------------|-----|--------|-----------|--------|---------|------------|---------|---------|------------|---------|
166+
| DISCOVER | 5 | 1000 | 10 | 500 | 4 | 4 | 16 | 34 | 0 | 103 | 110 | 111 | 111 |
167+
| DISCOVER | 5 | 1000 | 100 | 500 | 5 | 5 | 16 | 29 | 0 | 89 | 111 | 113 | 113 |
168+
| DISCOVER | 10 | 10000 | 10 | 500 | 166 | 9 | 6168 | 16 | 20 | 94 | 322 | 319 | 346 |
169+
| DISCOVER | 10 | 10000 | 100 | 500 | 192 | 9 | 6658 | 18 | 18 | 93 | 352 | 352 | 353 |
170+
| DISCOVER | 10 | 10000 | 10 | 1000 | 80 | 9 | 5274 | 18 | 19 | 97 | 326 | 325 | 362 |
171+
| DISCOVER | 10 | 10000 | 100 | 1000 | 102 | 9 | 6701 | 15 | 18 | 98 | 320 | 315 | 346 |
172+
| DISCOVER | 100 | 100000 | 10 | 500 | 29002 | 118 | 119308 | 10 | 0 | 149 | 2062 | 2021 | 2292 |
173+
| DISCOVER | 100 | 100000 | 100 | 500 | 30290 | 114 | 127063 | 10 | 5 | 154 | 1995 | 2037 | 2120 |
174+
175+
The difference in results between the default interval of 20 and bigger interval of 100 was not significant in any of the evaluated metrics.
176+
177+
**Discover inexistent namespaces**
178+
179+
| Type | Clients | Io Reg | Namespaces | Ops | Avg RT | Median RT | Max RT | Avg CPU | Median CPU | Max CPU | Avg Mem | Median Mem | Max Mem |
180+
|----------|---------|--------|------------|-----|--------|-----------|--------|---------|------------|---------|---------|------------|---------|
181+
| DISCOVER | 5 | 0 | 10 | 500 | 16 | 15 | 56 | 24 | 39 | 42 | 96 | 96 | 103 |
182+
| DISCOVER | 5 | 0 | 10 | 1000 | 17 | 16 | 107 | 20 | 26 | 44 | 102 | 103 | 110 |
183+
| DISCOVER | 10 | 0 | 5 | 1000 | 19 | 18 | 88 | 65 | 65 | 65 | 90 | 90 | 90 |
184+
| DISCOVER | 10 | 0 | 100 | 1000 | 20 | 19 | 61 | 23 | 21 | 56 | 101 | 101 | 109 |
185+
| DISCOVER | 100 | 0 | 10 | 10000 | 241 | 228 | 1012 | 22 | 15 | 79 | 282 | 294 | 299 |
186+
| DISCOVER | 100 | 0 | 100 | 10000 | 176 | 172 | 569 | 22 | 3 | 68 | 261 | 270 | 283 |
187+
| DISCOVER | 10 | 10000 | 100 | 10000 | 37 | 26 | 7795 | 16 | 8 | 103 | 277 | 203 | 457 |
188+
| DISCOVER | 100 | 10000 | 100 | 10000 | 348 | 266 | 7664 | 21 | 13 | 73 | 301 | 261 | 428 |
189+
| DISCOVER | 10 | 100000 | 100 | 10000 | 291 | 26 | 162046 | 12 | 5 | 153 | 1956 | 1935 | 2191 |
190+
| DISCOVER | 100 | 100000 | 100 | 10000 | 1779 | 291 | 178066 | 12 | 4 | 150 | 2392 | 2618 | 2828 |
191+
192+
There were no different conclusions from observing these results compared to the previous ones.
193+
194+
**Final Remarks**
195+
196+
Generally, libp2p nodes who register namespaces aim to be found for providing a specific service to boost the network. This way, they will probably not be much interested in discovering other peers. On the other side, peers who will discover peers providing a given service will likely not provide themselves any service. Moreover, they will not be continuously trying to discover peers. This is not a rule, but a general expectation to consider the results obtained. For instance, this can be used to discover peers sharing a pubsub topics to improve a given topology.
197+
198+
Taking into account the above consideration, let's consider a network of 1 Million libp2p nodes interacting with a rendezvous server where the average requests per 24 Hours are 8 requests per peer. This means around:
199+
200+
- 8M requests per day
201+
- 333.333k requests per hour
202+
- 5.555k requests per minute
203+
- 92 requests per second
204+
205+
In this context, around 92 connections would be established per second to create a rendezvous request, wait for it to be processed and receive a response. This will be bigger if the network gets bigger, or if peers do more requests than the mentioned average. In addition, DoS attacks might also happen.
206+
207+
Comparing these numbers with the results obtained, we can see that for around 100 connections simultaneously interacting with the rendezvous server, the median Round Trip times, memory consumption and CPU usage are within acceptable intervals. However, as requests accumulate it is also likely that some requests will take longer to process, even with a client per machine state. It is important highlighting that running 100+ clients in a single machine & process will also be a bottleneck and results should be better in "real" environment.
208+
209+
The ideal scenario for a deployment of a Rendezvous Server will be to have clusters of rendezvous servers backed by a federated DB. This would guarantee that the server keeps healthy and with good response times. For the above example with 1M nodes, we should probably have a cluster with 3 rendezvous servers, which would receive an average of 30 requests per second. The average does not mean that in certain times they can get to 50 requests per second, or even more.

‎benchmarks/id.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"id": "12D3KooWPCDhKA2NoJCnxUqKmBNLVg6ekNzcw9GVsncLgfdbN2pm",
3+
"privKey": "CAESYDprk82zAJeNcIHxgj3seEWRLCOh+7e7yTBVx1IW38HnxsEgoQGbW5xJDd7GHml2Mb8LNxsdB+WgznkhDLYZL97GwSChAZtbnEkN3sYeaXYxvws3Gx0H5aDOeSEMthkv3g==",
4+
"pubKey": "CAESIMbBIKEBm1ucSQ3exh5pdjG/CzcbHQfloM55IQy2GS/e"
5+
}

0 commit comments

Comments
 (0)