Improve raftReplDev creation

[yuwmao]

Use user-defined config(e.g. priority) to create raftReplDev.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 94.73684% with 1 line in your changes missing coverage. Please review.

Project coverage is 66.75%. Comparing base (1a0cef8) to head (9ed7ccf).
Report is 179 commits behind head on master.

Files with missing lines	Patch %	Lines
src/lib/replication/repl_dev/solo_repl_dev.h	0.00%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #695       +/-   ##
===========================================
+ Coverage   56.51%   66.75%   +10.24%     
===========================================
  Files         108      109        +1     
  Lines       10300    11519     +1219     
  Branches     1402     1574      +172     
===========================================
+ Hits         5821     7690     +1869     
+ Misses       3894     3076      -818     
- Partials      585      753      +168     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xiaoxichen]

Reading this doc https://github.com/eBay/NuRaft/blob/1c2587f159775a4924573db64ffac5c973230761/docs/leader_election_priority.md#L16

I am concerning of open the priority of int32 to CM, as without understanding the implementation detail it probabaly set < int32_max, 1, 0> which will be a nightmare for leader election.

Probabaly HO can still provide a list of replica_id_t and HS decide the priority based on the internal knowledge, we can make the K configurable where K = round of election timeout needed.

Decay = (0.8^k) , for example K =2 make it <10000, 6400, 4096 >. (each with 0.64 decay meaning 2 election timeout).

[yuwmao]

Reading this doc https://github.com/eBay/NuRaft/blob/1c2587f159775a4924573db64ffac5c973230761/docs/leader_election_priority.md#L16

I am concerning of open the priority of int32 to CM, as without understanding the implementation detail it probabaly set < int32_max, 1, 0> which will be a nightmare for leader election.

Probabaly HO can still provide a list of replica_id_t and HS decide the priority based on the internal knowledge, we can make the K configurable where K = round of election timeout needed.

Decay = (0.8^k) , for example K =2 make it <10000, 6400, 4096 >. (each with 0.64 decay meaning 2 election timeout).

Good point, and the gap should be bigger than 10. Regarding to decide the priority based on the internal knowledge, I think we can only infer who is the leader although it's enough in our NuObject scenario.
There are two thoughts:

1. Make strict control of the priority setting in HS, make sure they are valid and we can define K as max round the election timeout, leaving some flexibility for CM.
2. Simplify the contract with CM, change priority to a bool which means who is assigned to the leader(maybe not necessary in proto, but at least CM should keep the intent in its DB). And in HS's replace_member, the in_member inherit out_member's priority directly based on srv_config.

I think option2 is simpler and more controllable.

[xiaoxichen]

Yes we discussed this a bit in CM meeting, currently there is no use-case we want to control the secondary-leader.

• CM will nominate a leader, and CM will persist the leader in its DB as well as internal PG structure. So that CM is able to tell for any given PG if the leader has drifted.

• When CM send create pg request, it will carry the leader intent, either through priority or we change the grpc protocol. But at least for now it is clear the one who receive the PG_CREATE is the leader.

• HO/HS should decide the priority setting of NuRaft, based on the K configuration or other internal knowledge.

• When replace member, the in_member inherit the priority ---though for the sake of completeness I believe the in_member will have a flag should_be_leader indicating if it supposed to be leader.

So basically it is your approach 2

[xiaoxichen]

this is a nice formula