Skip to content

perf: Only add CopyExec if source of ScanExec is native_comet#1808

Closed
andygrove wants to merge 3 commits intoapache:mainfrom
andygrove:scan-buffer-reuse
Closed

perf: Only add CopyExec if source of ScanExec is native_comet#1808
andygrove wants to merge 3 commits intoapache:mainfrom
andygrove:scan-buffer-reuse

Conversation

@andygrove
Copy link
Copy Markdown
Member

@andygrove andygrove commented May 28, 2025

Which issue does this PR close?

Closes #1836

Rationale for this change

Avoid adding unnecessary copies. Thanks to @rluvaton for noticing this issue in #1793 (comment)

What changes are included in this PR?

Add a flag to the protobuf definition of Scan to indicate if the source of the scan reuses buffers.

How are these changes tested?

Existing tests.

@andygrove andygrove requested a review from parthchandra May 28, 2025 15:36
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 28, 2025

Codecov Report

Attention: Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 58.50%. Comparing base (f09f8af) to head (e16f02a).
Report is 315 commits behind head on main.

Files with missing lines Patch % Lines
.../scala/org/apache/comet/serde/QueryPlanSerde.scala 66.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1808      +/-   ##
============================================
+ Coverage     56.12%   58.50%   +2.37%     
- Complexity      976     1129     +153     
============================================
  Files           119      129      +10     
  Lines         11743    12582     +839     
  Branches       2251     2341      +90     
============================================
+ Hits           6591     7361     +770     
+ Misses         4012     4002      -10     
- Partials       1140     1219      +79     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

@parthchandra parthchandra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm so far. (pending ci and changing from draft status)

@andygrove
Copy link
Copy Markdown
Member Author

I am running into unexpected memory corruption issues with this PR, so it seems that copies are needed in more cases than I had realized.

@andygrove
Copy link
Copy Markdown
Member Author

I started this PR thinking it would be a trivial change, but I need to learn more about the behavior here and this is not a current priority, so closing the PR for now.

@andygrove andygrove closed this Jun 28, 2025
@andygrove andygrove deleted the scan-buffer-reuse branch July 10, 2025 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid unnecessary uses of CopyExec in native plan

3 participants