[fix](memory) Fix BE OOM when load -238 fail #12666
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zhannngchen
reviewed
Sep 16, 2022
| error->set_tablet_id(writers[i]->tablet_id()); | ||
| error->set_msg(err_msg); | ||
| _broken_tablets.insert(writers[i]->tablet_id()); | ||
| } |
Contributor
There was a problem hiding this comment.
If some error happened while flush_memtable_and_wait, we should not call wait_flush on that delta_writer in L260, in that situation, wait_flush might wait infinitely?
Contributor
Author
There was a problem hiding this comment.
wait_flush should not wait infinitely, and the flush token that fails to submit the flush will not actually wait.
Do not return error directly, pass the error to the sink to decide whether to terminate the load.
Contributor
Author
There was a problem hiding this comment.
done, _broken_tablets does not wait flush
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
13 tasks
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
When the flush is triggered when the load channel exceeds the mem limit, if the flush fails, an error message is returned and the load is terminated. Usually flush failure is -238 error code. Because the memtable is frequently flushed after the load channel exceeds the mem limit, the number of segments exceeds the max value.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
When the flush is triggered when the load channel exceeds the mem limit, if the flush fails, an error message is returned and the load is terminated. Usually flush failure is -238 error code. Because the memtable is frequently flushed after the load channel exceeds the mem limit, the number of segments exceeds the max value.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
When the flush is triggered when the load channel exceeds the mem limit, if the flush fails, an error message is returned and the load is terminated. Usually flush failure is -238 error code. Because the memtable is frequently flushed after the load channel exceeds the mem limit, the number of segments exceeds the max value.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
When the flush is triggered when the load channel exceeds the mem limit, if the flush fails, an error message is returned and the load is terminated. Usually flush failure is -238 error code. Because the memtable is frequently flushed after the load channel exceeds the mem limit, the number of segments exceeds the max value.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
When the flush is triggered when the load channel exceeds the mem limit, if the flush fails, an error message is returned and the load is terminated. Usually flush failure is -238 error code. Because the memtable is frequently flushed after the load channel exceeds the mem limit, the number of segments exceeds the max value.
FreeOnePlus
pushed a commit
to FreeOnePlus/doris
that referenced
this pull request
Oct 8, 2022
When the flush is triggered when the load channel exceeds the mem limit, if the flush fails, an error message is returned and the load is terminated. Usually flush failure is -238 error code. Because the memtable is frequently flushed after the load channel exceeds the mem limit, the number of segments exceeds the max value.
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #12661
Problem summary
When the flush is triggered when the load channel exceeds the mem limit, if the flush fails, an error message is returned and the load is terminated.
Usually flush failure is -238 error code. Because the memtable is frequently flushed after the load channel exceeds the mem limit, the number of segments exceeds the max value.
Before fix:
After fix:
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...