Fix Journal.ForceWriteThread.forceWriteRequests.put deadlock #2962
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dlg99
requested changes
Jan 3, 2022
| // the flush so until this marker is encountered we can skip the force write | ||
| if (enableGroupForceWrites) { | ||
| forceWriteRequests.put(createForceWriteRequest(req.logFile, 0, 0, null, false, true)); | ||
| forceWriteMarkerSent = forceWriteRequests.offer(createForceWriteRequest(req.logFile, 0, 0, null, false, true)); |
Contributor
There was a problem hiding this comment.
consider logging and incrementing a counter (for monitoring/alerting/troubleshooting) if forceWriteMarkerSent is false.
Contributor
There was a problem hiding this comment.
createForceWriteRequest uses pool; if offer failed we need to recycle the request to avoid leak.
Contributor
|
also please fix checkstyle error: |
…WriteRequests.put
kezhuw
force-pushed
the
fix-Journal.ForceWriteThread.forceWriteRequests.put-deadlock
branch
from
a050531 to
4365c30
Compare
Member
Author
|
@dlg99 I have pushed fixup commits to solve your concerns, could you please take another look ? |
dlg99
approved these changes
Jan 21, 2022
Contributor
dlg99
left a comment
dlg99
left a comment
There was a problem hiding this comment.
Looks good, thank you!
let's wait for the CI results + someone else to review.
Contributor
wolfstudy
pushed a commit
to wolfstudy/bookkeeper
that referenced
this pull request
Aug 2, 2022
Descriptions of the changes in this PR: ### Motivation `Journal.ForceWriteThread` could deadlock as it is the sole consumer of `Journal.forceWriteRequests` while it send group marker blocking using `BlockingQueue.put`. This PR try to fix this. ### Changes * Add testing code to deadlock `Journal.ForceWriteThread` on `forceWriteRequests.put`. * Send force write group marker non-blocking to avoid deadlock `ForceWriteThread`. Master Issue: apache#2948 Reviewers: Andrey Yegorov <None> This closes apache#2962 from kezhuw/fix-Journal.ForceWriteThread.forceWriteRequests.put-deadlock (cherry picked from commit 1b1e937)
wolfstudy
pushed a commit
to wolfstudy/bookkeeper
that referenced
this pull request
Aug 2, 2022
…e request !7) Cherry pick 4.15 bug changes to repo 1. cherry pick some 4.15.0 changes - apache#2962 - apache#2836 2. Bump new bookie version to 4.14.4.220419-SNAPSHOT
zymap
pushed a commit
that referenced
this pull request
Aug 24, 2022
…equently forceWrite (#3454) Fix group ForceWrite not take effect with forceWriteMarkerSent in while loop of ForceWriteThread ### Motivation Bookkeeper 4.15 has performance issue. There are too many journal sync. Then I found in ForceWriteThread, forceWriteMarkerSent is reset to false for every loop, and this can cause shouldForceWrite=true and trigger journal sync. This new logic is bringed by #2962 ### Changes move forceWriteMarkerSent=false out of while loop.
zymap
pushed a commit
that referenced
this pull request
Aug 24, 2022
…equently forceWrite (#3454) Fix group ForceWrite not take effect with forceWriteMarkerSent in while loop of ForceWriteThread ### Motivation Bookkeeper 4.15 has performance issue. There are too many journal sync. Then I found in ForceWriteThread, forceWriteMarkerSent is reset to false for every loop, and this can cause shouldForceWrite=true and trigger journal sync. This new logic is bringed by #2962 ### Changes move forceWriteMarkerSent=false out of while loop. (cherry picked from commit dcc9a41)
Ghatage
pushed a commit
to sijie/bookkeeper
that referenced
this pull request
Jul 12, 2024
Descriptions of the changes in this PR: ### Motivation `Journal.ForceWriteThread` could deadlock as it is the sole consumer of `Journal.forceWriteRequests` while it send group marker blocking using `BlockingQueue.put`. This PR try to fix this. ### Changes * Add testing code to deadlock `Journal.ForceWriteThread` on `forceWriteRequests.put`. * Send force write group marker non-blocking to avoid deadlock `ForceWriteThread`. Master Issue: apache#2948 Reviewers: Andrey Yegorov <None> This closes apache#2962 from kezhuw/fix-Journal.ForceWriteThread.forceWriteRequests.put-deadlock
Ghatage
pushed a commit
to sijie/bookkeeper
that referenced
this pull request
Jul 12, 2024
…equently forceWrite (apache#3454) Fix group ForceWrite not take effect with forceWriteMarkerSent in while loop of ForceWriteThread ### Motivation Bookkeeper 4.15 has performance issue. There are too many journal sync. Then I found in ForceWriteThread, forceWriteMarkerSent is reset to false for every loop, and this can cause shouldForceWrite=true and trigger journal sync. This new logic is bringed by apache#2962 ### Changes move forceWriteMarkerSent=false out of while loop.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Descriptions of the changes in this PR:
Motivation
Journal.ForceWriteThreadcould deadlock as it is the sole consumer ofJournal.forceWriteRequestswhile it send group marker blocking usingBlockingQueue.put.This PR try to fix this.
Changes
Journal.ForceWriteThreadonforceWriteRequests.put.ForceWriteThread.Master Issue: #2948