Optimize planning / stop cloning Strings / Fields so much (2-3% faster planning time) #18415
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alamb
changed the title
github-actions
bot
added
sql
alamb
commented
Oct 31, 2025
| metadata, | ||
| .. | ||
| }) => { | ||
| let field = expr.to_field(schema).map(|(_, f)| f.as_ref().clone())?; |
Contributor
Author
There was a problem hiding this comment.
these as_ref().clone() type methods are doing a deep clone on the Fields each time even when the name is not changing
Closed
Contributor
Author
|
🤖 |
alamb
commented
Oct 31, 2025
| /// Returns an immutable reference of a specific `Field` instance selected using an | ||
| /// offset within the internal `fields` vector | ||
| pub fn field(&self, i: usize) -> &Field { | ||
| pub fn field(&self, i: usize) -> &FieldRef { |
Contributor
Author
There was a problem hiding this comment.
the changes in this file are the public API changes -- they return a reference to the Arc rather than what is in the Arc meaning the callsite can clone the arcs when needed
Contributor
Author
|
🤖: Benchmark completed Details
|
Contributor
Author
|
🤖 |
Contributor
Author
|
🤖: Benchmark completed Details
|
alamb
force-pushed
the
alamb/stop_cloning_fields
branch
from
074e485 to
879d1bf
Compare
github-actions
bot
added
the
documentation
alamb
force-pushed
the
alamb/stop_cloning_fields
branch
from
879d1bf to
9dcfdfc
Compare
alamb
commented
Nov 11, 2025
| let filter_df_fields = filter_df_fields | ||
| .into_iter() | ||
| .map(|(qualifier, field)| { | ||
| (qualifier.cloned(), Arc::new(field.clone())) |
Contributor
Author
There was a problem hiding this comment.
this was always a deep clone that is now remved
Contributor
Author
|
🤖 |
Contributor
Author
|
🤖: Benchmark completed Details
|
Contributor
Author
|
🤖 |
Contributor
Author
|
🤖: Benchmark completed Details
|
alamb
changed the title
alamb
commented
Nov 12, 2025
| } | ||
|
|
||
| impl FieldExt for Field { | ||
| fn renamed(self, new_name: &str) -> Self { |
Contributor
Author
There was a problem hiding this comment.
I added several new methods to avoid cloning when it is unnecesary
| @@ -429,38 +430,36 @@ | |||
Contributor
Author
There was a problem hiding this comment.
this is called as part of Expr::nullable and Expr::nullable so it is a very hot path on planning
Contributor
Author
|
@Jefffrey do you happen to have time to review this PR? |
Jefffrey
approved these changes
Nov 26, 2025
datafusion/common/src/dfschema.rs
Outdated
| &self.inner.fields | ||
| } | ||
|
|
||
| /// Returns an immutable reference of a specific `Field` instance selected using an |
Contributor
There was a problem hiding this comment.
Perhaps need to update some of these method docstrings since they assume returning a Field
| /// let renamed_field = int_field.renamed("your_int"); | ||
| /// assert_eq!(renamed_field.name(), "your_int"); | ||
| /// ``` | ||
| fn renamed(self, new_name: &str) -> Self; |
Contributor
There was a problem hiding this comment.
I wonder for these new methods if there is existing documentation elsewhere we'd need to upgrade to make these new methods more visible? i.e. "use these new methods if possible", though not sure how relevant might be considering these mainly replace the old as_ref from Arc -> clone -> modify field -> wrap in new Arc loop 🤔
Contributor
Author
There was a problem hiding this comment.
it is a good idea -- I will review the existing docs and try to make it more obvious / clear how to use the new methdods
Contributor
Author
There was a problem hiding this comment.
I tried to update the documentation with some more details on how to discover the new traits
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
…on into alamb/stop_cloning_fields
Contributor
Author
Thanks @xudong963 -- the good news is I think there is some more significant improvement we could make (by not copying strings so much). I'll try and explore it when I have more time |
Contributor
Author
|
run benchmarks sql_planner |
Contributor
Author
|
🤖 Hi @alamb, thanks for the request (#18415 (comment)).
Please choose one or more of these with |
Contributor
Author
|
run benchmark sql_planner |
Contributor
Author
|
show benchmark queue |
Contributor
Author
|
🤖 Hi @alamb, you asked to view the benchmark queue (#18415 (comment)).
|
Closed
Contributor
Author
|
🤖 |
Contributor
Author
|
🤖: Benchmark completed Details
|
Contributor
Author
|
Thanks again @xudong963 and @Jefffrey |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Avoid a bunch of clones / String copies during planning
What changes are included in this PR?
Change several methods on DFSchema to return
&FieldRefrather than&Fieldwhich permitsArc::clonerather than a deepFieldcloneAre these changes tested?
yes by CI
I also ran benchmarks that show a small but consistent speedup in the planning benchmarks
Are there any user-facing changes?
Yes, there are several API changes in DFSchema that now return
FieldRefrather thanFieldwhich allows usingArc::clonerather thanclone. I have updated the upgrading guide too