Use peermanager scores for blocksync peers and don't error out on block mismatch by philipsu522 · Pull Request #162 · sei-protocol/sei-tendermint

philipsu522 · 2023-10-06T23:31:17Z

Describe your changes and provide context

Import peermanager scores to blocksync reactor and use that to determine which peers to pick for blocksync. We also don't error out on a mismatch of block height from peer who sent it

Testing performed to validate your change

Added unit tests
Deploy to autobake, stopped a validator and restarted - verified catch up still works
Tested on standalone rpc in pacific-1

yzang2019 · 2023-10-11T20:44:25Z

internal/blocksync/pool.go

 	"context"
 	"errors"
 	"fmt"
+	"github.com/tendermint/tendermint/internal/p2p"


Nit: fix lint issue

yzang2019 · 2023-10-11T23:16:58Z

internal/blocksync/pool.go

-	for _, peer := range pool.peers {
+	// Generate a sorted list
+	sortedPeers := pool.getSortedPeers(pool.peers)
+	fmt.Printf("PSUDEBUG - block sync with sorted peers: %v\n", sortedPeers)


Shall we remove this debug log?

yzang2019 · 2023-10-11T23:19:59Z

internal/blocksync/pool.go

+	fmt.Printf("PSUDEBUG - block sync with sorted peers: %v\n", sortedPeers)
+	for _, nodeId := range sortedPeers {
+		peer := pool.peers[nodeId]
+		pool.peerManager.Score(peer.id)


why do we get the score out here without using it?

yzang2019 · 2023-10-11T23:22:56Z

internal/blocksync/pool.go

+	// Generate a sorted list
+	sortedPeers := pool.getSortedPeers(pool.peers)
+	fmt.Printf("PSUDEBUG - block sync with sorted peers: %v\n", sortedPeers)
+	for _, nodeId := range sortedPeers {


A few further optimizations we can do:

We should probably avoid using the low score peers, for example the ones with score 0

We should do some random shuffling so that we don't always targeting the same few top peers

philipsu522 · 2023-10-23T16:36:43Z

internal/blocksync/pool.go

 )

-var peerTimeout = 15 * time.Second // not const so we can override with tests
+var peerTimeout = 3 * time.Second // not const so we can override with tests


waiting 15s for timeout is too long given we expect blocks much more frequently

yzang2019 · 2023-10-23T18:45:24Z

internal/blocksync/pool.go

 	} else {
 		err := errors.New("requester is different or block already exists")
-		pool.sendError(err, peerID)
+		// Original behavior is to error out when there is a mismatch, which shuts down the entire reactor.


Overall LGTM, one question: do we understand why the original behavior wants to error this out and shutdown the entire reactor? Is there going to be any side effect if we change this behavior?

let's revert this now, and when the standalone rpc falls behind again (currently it is healthly, and was upgraded to the new version), we can re-apply this patch and see if it helps

* Reduce noisy tendermint logs * Suppress logs for all entering Use peermanager scores for blocksync peers and don't error out on block mismatch (#162) * Use peermanager scores for blocksync peers * Add debug * Randomize * debug * use state to filter * debug * debug * debug * debug * add comments * don't err * revert timeout * Add missing param * Remove flaky test * fix nil * debug * debug * debug * debug --------- Co-authored-by: Yiming Zang <50607998+yzang2019@users.noreply.github.com> Perf: Increase buffer size for pubsub server to boost performance (#167) * Increase buffer size for pubsub server * Add more timeout for test failure * Add more timeout * Fix test split scripts * Fix test split * Fix unit test * Unit test * Unit test [P2P] Optimize block pool requester retry and peer pick up logic (#170) * P2P Improvements: Fix block sync reactor and block pool retry logic Fix block sync auto restart not working as expected (#175) Fix edge case for blocksync (#178) Replay events during restart to avoid tx missing (#211) Add a new config to speed up block sync (#244) * Never switch to consensus due to timeout * Add blocksync peer config to speed up block sync rate * Fix config parsing * Add some logs feat: Exclude unconditional peers when connection limit checking (#245) Improve Peer Score Algorithm (#248) * feat: improve peer scoring algo * debug * debug * more debug * debug TryDiaNext * remove log * fix score type * rever block sync logic * rever block sync logic * rever block sync logic * Add block request log * Add apply block latency * add processEpeerEvent log back * update unit test * update unit test --------- Co-authored-by: yzang2019 <zymfrank@gmail.com>

* Reduce noisy tendermint logs * Suppress logs for all entering Use peermanager scores for blocksync peers and don't error out on block mismatch (#162) * Use peermanager scores for blocksync peers * Add debug * Randomize * debug * use state to filter * debug * debug * debug * debug * add comments * don't err * revert timeout * Add missing param * Remove flaky test * fix nil * debug * debug * debug * debug --------- Co-authored-by: Yiming Zang <50607998+yzang2019@users.noreply.github.com> Perf: Increase buffer size for pubsub server to boost performance (#167) * Increase buffer size for pubsub server * Add more timeout for test failure * Add more timeout * Fix test split scripts * Fix test split * Fix unit test * Unit test * Unit test [P2P] Optimize block pool requester retry and peer pick up logic (#170) * P2P Improvements: Fix block sync reactor and block pool retry logic Fix block sync auto restart not working as expected (#175) Fix edge case for blocksync (#178) Replay events during restart to avoid tx missing (#211) Add a new config to speed up block sync (#244) * Never switch to consensus due to timeout * Add blocksync peer config to speed up block sync rate * Fix config parsing * Add some logs feat: Exclude unconditional peers when connection limit checking (#245) Improve Peer Score Algorithm (#248) * feat: improve peer scoring algo * debug * debug * more debug * debug TryDiaNext * remove log * fix score type * rever block sync logic * rever block sync logic * rever block sync logic * Add block request log * Add apply block latency * add processEpeerEvent log back * update unit test * update unit test --------- Co-authored-by: yzang2019 <zymfrank@gmail.com> Add logs for blocksync Add log to debug replay Fix Fix setBlock true false Add more log for dialing Add more logs Add logs for missing height Fix Fix log Fix log Fix log Fix bpr pool goroutine Fix bpr pool goroutine Fix goroutine leak Fix goroutine leak

philipsu522 added 2 commits October 6, 2023 16:30

Use peermanager scores for blocksync peers

5786063

Add debug

1ac69bc

philipsu522 requested a review from yzang2019 October 11, 2023 20:23

yzang2019 reviewed Oct 11, 2023

View reviewed changes

philipsu522 added 7 commits October 16, 2023 14:30

Randomize

fe9f6e0

debug

08f8163

use state to filter

2f4517f

debug

bfc637c

debug

533f759

debug

8fa8638

debug

1137ac4

philipsu522 commented Oct 23, 2023

View reviewed changes

add comments

2bc2fbb

philipsu522 enabled auto-merge (squash) October 23, 2023 16:48

philipsu522 changed the title ~~Use peermanager scores for blocksync peers~~ Use peermanager scores for blocksync peers and don't error out on block mismatch Oct 23, 2023

yzang2019 reviewed Oct 23, 2023

View reviewed changes

Merge branch 'main' into block-sync-prio-peers

3610170

yzang2019 approved these changes Oct 23, 2023

View reviewed changes

philipsu522 added 10 commits October 24, 2023 14:22

don't err

a1bd259

revert timeout

a83139b

Add missing param

ae4b8bc

Merge branch 'main' into block-sync-prio-peers

e398b04

Remove flaky test

3f06493

fix nil

a70ffd2

debug

e47270c

debug

36d31f9

debug

ecf6e7f

debug

a149386

philipsu522 disabled auto-merge October 31, 2023 21:46

philipsu522 merged commit dd79101 into main Oct 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use peermanager scores for blocksync peers and don't error out on block mismatch#162

Use peermanager scores for blocksync peers and don't error out on block mismatch#162
philipsu522 merged 21 commits intomainfrom
block-sync-prio-peers

philipsu522 commented Oct 6, 2023 •

edited

Loading

Uh oh!

yzang2019 Oct 11, 2023

Uh oh!

yzang2019 Oct 11, 2023

Uh oh!

yzang2019 Oct 11, 2023

Uh oh!

yzang2019 Oct 11, 2023

Uh oh!

philipsu522 Oct 23, 2023

Uh oh!

yzang2019 Oct 23, 2023 •

edited

Loading

Uh oh!

philipsu522 Oct 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

philipsu522 commented Oct 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

yzang2019 Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

yzang2019 Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

yzang2019 Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

yzang2019 Oct 11, 2023

Choose a reason for hiding this comment

Uh oh!

philipsu522 Oct 23, 2023

Choose a reason for hiding this comment

Uh oh!

yzang2019 Oct 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philipsu522 Oct 24, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

philipsu522 commented Oct 6, 2023 •

edited

Loading

yzang2019 Oct 23, 2023 •

edited

Loading