Skip to content

Conversation

@Yukang-Lian
Copy link
Collaborator

Pick [Fix](group commit) Fix group commit block queue mem estimate faule #35314

Proposed changes

Issue Number: close #xxx

Problem: When group commit=async_mode and NULL data is imported into a variant type column, it causes incorrect memory statistics for group commit backpressure, leading to a stuck issue. Cause: In group commit mode, blocks are first added to a queue in batches using add block, and then blocks are retrieved from the queue using get block. To track memory usage during backpressure, we add the block size to the memory statistics during add block and subtract the block size from the memory statistics during get block. However, for variant types, during the add block write to WAL, serialization occurs, which can merge types (e.g., merging int and bigint into bigint), thereby changing the block size. This results in a discrepancy between the block size during get block and add block, causing memory statistics to overflow.
Solution: Record the block size at the time of add block and use this recorded size during get block instead of the actual block size. This ensures consistency in the memory addition and subtraction.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Proposed changes

Issue Number: close #xxx

…pache#35314)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

**Problem:** When `group commit=async_mode` and NULL data is imported
into a `variant` type column, it causes incorrect memory statistics for
group commit backpressure, leading to a stuck issue.
**Cause:** In group commit mode, blocks are first added to a queue in
batches using `add block`, and then blocks are retrieved from the queue
using `get block`. To track memory usage during backpressure, we add the
block size to the memory statistics during `add block` and subtract the
block size from the memory statistics during `get block`. However, for
`variant` types, during the `add block` write to WAL, serialization
occurs, which can merge types (e.g., merging `int` and `bigint` into
`bigint`), thereby changing the block size. This results in a
discrepancy between the block size during `get block` and `add block`,
causing memory statistics to overflow.
**Solution:** Record the block size at the time of `add block` and use
this recorded size during `get block` instead of the actual block size.
This ensures consistency in the memory addition and subtraction.

## Further comments

If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.32% (9151/25194)
Line Coverage: 27.86% (74703/268115)
Region Coverage: 26.74% (38512/144033)
Branch Coverage: 23.45% (19521/83252)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c01253eb4b4d2ffd8b1987dda27c81fb7464124f_c01253eb4b4d2ffd8b1987dda27c81fb7464124f/report/index.html

@dataroaring dataroaring merged commit 7d423b3 into apache:branch-2.1 Jul 7, 2024
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants