Skip to content

Optimize scheduler#2067

Merged
bors[bot] merged 3 commits into
nervosnetwork:developfrom
driftluo:optimize-scheduler
Jun 19, 2020
Merged

Optimize scheduler#2067
bors[bot] merged 3 commits into
nervosnetwork:developfrom
driftluo:optimize-scheduler

Conversation

@driftluo

@driftluo driftluo commented May 12, 2020

Copy link
Copy Markdown
Contributor

Problems with the current scheduler:

  1. The calculation is too frequent
  2. Inability to adapt to complex network environments, e.g., increased network latency as the number of nodes increases, network latency exceeding expectations, etc.

This PR implements an adaptive scheduler based on past data, removing most of the redundant calculations. Translating the overall data into 4 intervals:

| fast | normal | penalty | double penalty |

The dividing line is the 1/3 position, 4/5 position, 1/10 position.

There is a 14/30 normal area, 1/10 penalty area, 1/10 double penalty area, 1/3 accelerated reward area.

Replace the 30-second clear-out strategy with a doubling penalty.

10%-20% increase in sync performance from testing, from 220-240 block/s to 250-270 block/s on a dual-core machine

the simulation test: https://github.com/nervosnetwork/sync-scheduler-simulation-test

@driftluo

Copy link
Copy Markdown
Contributor Author

benchmark

@nervos-bot-user

Copy link
Copy Markdown

Benchmark Result

  • TPS: 359.98
  • Samples Count: 51
  • CKB Version: 70b6265ae1d0a969e0559859ef35f8d0b04ac304
  • Instance Type: c5.xlarge
  • Instances Count: 3
  • Bench Type: 2in2out
  • CKB Logger Filter: info,ckb=debug

@driftluo

Copy link
Copy Markdown
Contributor Author

benchmark

@nervos-bot-user

Copy link
Copy Markdown

Benchmark Result

  • TPS: 294.03
  • Samples Count: 51
  • CKB Version: 25e539c75ceab25802cfbbd3bbdb70276100a5b8
  • Instance Type: c5.xlarge
  • Instances Count: 3
  • Bench Type: 2in2out
  • CKB Logger Filter: info,ckb=debug

@driftluo driftluo force-pushed the optimize-scheduler branch from 25e539c to 70b6265 Compare May 12, 2020 04:34
@driftluo

Copy link
Copy Markdown
Contributor Author

benchmark

@nervos-bot-user

Copy link
Copy Markdown

Benchmark Result

  • TPS: 354.71
  • Samples Count: 50
  • CKB Version: 70b6265ae1d0a969e0559859ef35f8d0b04ac304
  • Instance Type: c5.xlarge
  • Instances Count: 3
  • Bench Type: 2in2out
  • CKB Logger Filter: info,ckb=debug

@driftluo driftluo marked this pull request as ready for review May 12, 2020 05:05
@driftluo driftluo requested review from a team, doitian, quake and zhangsoledad May 12, 2020 05:05
@driftluo driftluo force-pushed the optimize-scheduler branch from 70b6265 to 1bf3a58 Compare May 12, 2020 09:17
Comment thread sync/src/synchronizer/mod.rs Outdated
Comment thread sync/src/types.rs Outdated
Comment thread sync/src/types.rs Outdated
Comment thread sync/src/types.rs Outdated
@driftluo driftluo force-pushed the optimize-scheduler branch 3 times, most recently from 16bf17d to 4790ced Compare May 21, 2020 05:26
@driftluo driftluo force-pushed the optimize-scheduler branch from 4790ced to a48ae6a Compare June 5, 2020 09:22
@doitian

doitian commented Jun 12, 2020

Copy link
Copy Markdown
Member

@quake plans to give a better distribution models in simulation test to find better parameters after @keroro520 has collected new metrics. Before that we can merge this PR when there's no problems on the algorithm itself.

quake
quake previously approved these changes Jun 12, 2020
Comment thread sync/src/lib.rs Outdated
Comment thread sync/src/types.rs Outdated
Comment thread sync/src/types.rs Outdated
Comment thread sync/src/types.rs Outdated
Comment thread sync/src/types.rs Outdated
Comment thread sync/src/types.rs Outdated
Comment thread sync/src/types.rs Outdated
@driftluo driftluo force-pushed the optimize-scheduler branch from f7375ae to 7f4e1d9 Compare June 19, 2020 02:38
@driftluo driftluo requested a review from doitian June 19, 2020 02:40
doitian
doitian previously approved these changes Jun 19, 2020
@driftluo driftluo requested a review from doitian June 19, 2020 09:52
@doitian

doitian commented Jun 19, 2020

Copy link
Copy Markdown
Member

Sorry that I misunderstood #2067 (comment) , the original code looks good.

@doitian

doitian commented Jun 19, 2020

Copy link
Copy Markdown
Member

bors r=quake,doitian

@bors

bors Bot commented Jun 19, 2020

Copy link
Copy Markdown
Contributor

Build succeeded:

@bors bors Bot merged commit 8b4af64 into nervosnetwork:develop Jun 19, 2020
@driftluo driftluo deleted the optimize-scheduler branch June 19, 2020 12:12
@doitian

doitian commented Jul 20, 2020

Copy link
Copy Markdown
Member

The latest test result. The unit is block counts per second, higher is better.

  avg median min max count
eu_33_80_90 696 704 454 798 200
eu_50_75_90 674 683 483 786 193
eu_33_75_90 670 683 464 793 192
ap_33_80_90 662 666 475 759 190
ap_50_75_90 648 648 475 764 186
ap_33_75_90 636 650 474 720 182

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants