handle nagtive log batch size returned by follower #588
Conversation
JacksonYao287
force-pushed
the
add-log_entries_ext
branch
from
6e785e9 to
939e1b2
Compare
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## master #588 +/- ##
===========================================
+ Coverage 56.51% 67.33% +10.82%
===========================================
Files 108 109 +1
Lines 10300 10725 +425
Branches 1402 1467 +65
===========================================
+ Hits 5821 7222 +1401
+ Misses 3894 2801 -1093
- Partials 585 702 +117 ☔ View full report in Codecov by Sentry. |
| if (batch_size_hint_in_bytes < 0) end = start + 1; | ||
|
|
||
| // for the case where batch_size_hint_in_bytes >= 0, we do not take any size check here for now. | ||
| // TODO: limit the size of the returned entries by batch_size_hint_in_bytes int the future if necessary |
There was a problem hiding this comment.
@yamingk we discussed pressure_back , do you think we can use the batch_size_hint for this purpose?
i.e. follower can respond with batch_size_hint to leader through the response of appendEntries request.
There was a problem hiding this comment.
Good finding. yes, looks like follower can set the next_batch_size_hint in append_entries_resp and leader will set it in its next batch size and consume it when next batch happens:
https://github.com/eBay/NuRaft/blob/eabdeeda538a27370943f79a2b08b5738b697ac3/src/handle_append_entries.cxx#L1069~1072
when follower hits some error before appending log entries, it will set batch_size_hint_in_bytes to -1 to ask leader do not send more log entries in the next append_log_req. https://github.com/eBay/NuRaft/blob/eabdeeda538a27370943f79a2b08b5738b697ac3/src/handle_append_entries.cxx#L760 in nuobject case , if a new member is added to a raft group and it tries to append create_shard log entry , which will try to alllocate block from the chunks of the pg, before the create_pg log is committed , which will allocated chunks to this pg, and error will happen and the log batch containing create_shard log entry will be wholy rejected and set batch_size_hint_in_bytes to -1 in the response to leader. this pr aims to set the log count in the next batch sent to follower to 1, so that: if the create_pg and create_shard are in the same log batch , the pr will first reject this log batch and leader will send only create_pg in the next batch , which will be accepted by follower , since it will only create this pg. if if the create_pg and create_shard are not in the same log batch, and create_shard is trying to allocate block before the pg it created(chunks of this pg is alllocated), then , with this pr, follower will reject this batch so that it will give more time to creating pg. create_shard log will be resent in the next batch , and at that moment pg has probably already been successfully be created.
4 participants
when follower hits some error before appending log entries, it will set
batch_size_hint_in_bytesto -1 to ask leader do not send more log entries in the next append_log_req.https://github.com/eBay/NuRaft/blob/eabdeeda538a27370943f79a2b08b5738b697ac3/src/handle_append_entries.cxx#L760
in nuobject case , if a new member is added to a raft group and it tries to append create_shard log entry , which will try to alllocate block from the chunks of the pg, before the create_pg log is committed , which will allocated chunks to this pg, and error will happen and the log batch containing create_shard log entry will be wholy rejected and set
batch_size_hint_in_bytesto -1 in the response to leader.this pr aims to set the log count in the next batch sent to follower to 1, so that:
if the create_pg and create_shard are in the same log batch , the pr will first reject this log batch and leader will send only create_pg in the next batch , which will be accepted by follower , since it will only create this pg.
if the create_pg and create_shard are not in the same log batch, and create_shard is trying to allocate block before the pg it created(chunks of this pg is alllocated), then , with this pr, follower will reject this batch so that it will give more time to creating pg. create_shard log will be resent in the next batch , and at that moment pg has probably already been successfully created.