Skip to content

[Improvement](segment iterator) Optimize column row reservation to reduce overhead #42060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 3 commits into from
Oct 22, 2024
Merged

[Improvement](segment iterator) Optimize column row reservation to reduce overhead #42060

airborne12 merged 3 commits into from
Oct 22, 2024

Conversation

@airborne12
Copy link
Member

@airborne12 airborne12 commented Oct 18, 2024 •
edited
Loading

Proposed changes

This PR improves the segment iterator by reducing the reserved rows for a column when the row bitmap is smaller than the block row. This optimization aims to enhance memory efficiency and improve performance by reducing unnecessary row allocations.

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12 airborne12 changed the title [Improve](segment iterator) reserve less rows for column when row bit… [Improvement](segment iterator) Optimize column row reservation to reduce overhead Oct 18, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Oct 18, 2024

clang-tidy review says "All clean, LGTM! 👍"

@airborne12
Copy link
Member Author

airborne12 commented Oct 18, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

TeamCity be ut coverage result:
Function Coverage: 37.47% (9711/25919)
Line Coverage: 28.74% (80642/280617)
Region Coverage: 28.17% (41713/148088)
Branch Coverage: 24.75% (21209/85700)
Coverage Report: http://coverage.selectdb-in.cc/coverage/820e3984e22499a150acfa8a322cfd59cfc815cc_820e3984e22499a150acfa8a322cfd59cfc815cc/report/index.html

eldenmoon
eldenmoon reviewed Oct 18, 2024
View reviewed changes
@airborne12
Copy link
Member Author

airborne12 commented Oct 21, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Oct 21, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Oct 21, 2024

TeamCity be ut coverage result:
Function Coverage: 37.47% (9708/25912)
Line Coverage: 28.75% (80651/280523)
Region Coverage: 28.18% (41718/148045)
Branch Coverage: 24.77% (21213/85656)
Coverage Report: http://coverage.selectdb-in.cc/coverage/48b8cd544d27ec4cc4715d0b7631a9a4f7fe57df_48b8cd544d27ec4cc4715d0b7631a9a4f7fe57df/report/index.html

@github-actions
Copy link
Contributor

github-actions bot commented Oct 22, 2024

clang-tidy review says "All clean, LGTM! 👍"

@airborne12
Copy link
Member Author

airborne12 commented Oct 22, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 22, 2024

TeamCity be ut coverage result:
Function Coverage: 37.45% (9703/25907)
Line Coverage: 28.71% (80547/280527)
Region Coverage: 28.14% (41656/148014)
Branch Coverage: 24.72% (21168/85644)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fb2fc0ee44681e1b5ec219f5009c1c7431dd82c8_fb2fc0ee44681e1b5ec219f5009c1c7431dd82c8/report/index.html

zzzxl1993
zzzxl1993 approved these changes Oct 22, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Oct 22, 2024

PR approved by anyone and no changes requested.

qidaye
qidaye approved these changes Oct 22, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Oct 22, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 22, 2024
@airborne12
Copy link
Member Author

airborne12 commented Oct 22, 2024

run p1

@airborne12
Copy link
Member Author

airborne12 commented Oct 22, 2024

run cloud_p1

@airborne12 airborne12 merged commit 8653616 into apache:master Oct 22, 2024
@airborne12 airborne12 deleted the improve-perf branch October 22, 2024 10:53
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 24, 2024
@airborne12
[Improvement](segment iterator) Optimize column row reservation to re…
b4b1b2c 
…duce overhead (apache#42060)

## Proposed changes

This PR improves the segment iterator by reducing the reserved rows for
a column when the row bitmap is smaller than the block row. This
optimization aims to enhance memory efficiency and improve performance
by reducing unnecessary row allocations.
@airborne12 airborne12 mentioned this pull request Oct 24, 2024
Merged
airborne12 added a commit that referenced this pull request Oct 24, 2024
@airborne12
[Improvement](segment iterator) Optimize column row reservation to re…
f96ac18 
…duce overhead #42060 (#42372)

cherry pick from #42060
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eldenmoon eldenmoon eldenmoon left review comments

@qidaye qidaye qidaye approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@airborne12 @doris-robot @qidaye @zzzxl1993 @eldenmoon