-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[Variant] Make sure ObjectBuilder and ListBuilder to be finalized before its parent builder #7843
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
49982d1
dc09a48
9fc30d7
198c033
6be8dbe
c2d5e70
d6b2831
7c10c9e
933eb73
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -450,6 +450,7 @@ impl MetadataBuilder { | |
| /// obj.insert("a", 1); | ||
| /// obj.insert("a", 2); // duplicate field | ||
| /// | ||
| /// // When validation is enabled, finish will return an error | ||
| /// let result = obj.finish(); // returns Err | ||
| /// assert!(result.is_err()); | ||
| /// ``` | ||
|
|
@@ -495,10 +496,20 @@ impl VariantBuilder { | |
| .with_validate_unique_fields(self.validate_unique_fields) | ||
| } | ||
|
|
||
| /// Append a non-nested value to the builder. | ||
| /// | ||
| /// # Example | ||
| /// ``` | ||
| /// # use parquet_variant::{Variant, VariantBuilder}; | ||
| /// let mut builder = VariantBuilder::new(); | ||
| /// // most primitive types can be appended directly as they implement `Into<Variant>` | ||
| /// builder.append_value(42i8); | ||
| /// ``` | ||
| pub fn append_value<'m, 'd, T: Into<Variant<'m, 'd>>>(&mut self, value: T) { | ||
| self.buffer.append_non_nested_value(value); | ||
| } | ||
|
|
||
| /// Finish the builder and return the metadata and value buffers. | ||
| pub fn finish(self) -> (Vec<u8>, Vec<u8>) { | ||
| (self.metadata_builder.finish(), self.buffer.into_inner()) | ||
| } | ||
|
|
@@ -577,6 +588,7 @@ impl<'a> ListBuilder<'a> { | |
| self.offsets.push(element_end); | ||
| } | ||
|
|
||
| /// Finish the list, writing it to the parent buffer and consuming self. | ||
| pub fn finish(mut self) { | ||
| self.check_new_offset(); | ||
|
|
||
|
|
@@ -603,6 +615,14 @@ impl<'a> ListBuilder<'a> { | |
| } | ||
| } | ||
|
|
||
| /// Drop implementation for ListBuilder does nothing | ||
| /// as the `finish` method must be called to finalize the list. | ||
| /// This is to ensure that the list is always finalized before its parent builder | ||
| /// is finalized. | ||
| impl Drop for ListBuilder<'_> { | ||
| fn drop(&mut self) {} | ||
| } | ||
|
|
||
| /// A builder for creating [`Variant::Object`] values. | ||
| /// | ||
| /// See the examples on [`VariantBuilder`] for usage. | ||
|
|
@@ -694,9 +714,7 @@ impl<'a, 'b> ObjectBuilder<'a, 'b> { | |
| list_builder | ||
| } | ||
|
|
||
| /// Finalize object | ||
| /// | ||
| /// This consumes self and writes the object to the parent buffer. | ||
| /// Finalize the object, writing it to the parent buffer and consuming self. | ||
| pub fn finish(mut self) -> Result<(), ArrowError> { | ||
| self.check_pending_field(); | ||
|
|
||
|
|
@@ -756,6 +774,14 @@ impl<'a, 'b> ObjectBuilder<'a, 'b> { | |
| } | ||
| } | ||
|
|
||
| /// Drop implementation for ObjectBuilder does nothing | ||
| /// as the `finish` method must be called to finalize the object. | ||
| /// This is to ensure that the object is always finalized before its parent builder | ||
| /// is finalized. | ||
| impl Drop for ObjectBuilder<'_, '_> { | ||
| fn drop(&mut self) {} | ||
| } | ||
|
|
||
| /// Trait that abstracts functionality from Variant fconstruction implementations, namely | ||
| /// `VariantBuilder`, `ListBuilder` and `ObjectFieldBuilder` to minimize code duplication. | ||
| pub(crate) trait VariantBuilderExt<'m, 'v> { | ||
|
|
@@ -1490,6 +1516,9 @@ mod tests { | |
| "Invalid argument error: Duplicate field keys detected: [x]" | ||
| ); | ||
|
|
||
| inner_list.finish(); | ||
| outer_list.finish(); | ||
|
|
||
|
Comment on lines
+1519
to
+1521
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems a bug. Caught by the change. These builders were not called
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. awesome -- i also think it is a great example of how this change improves the variant builders
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems like the test is still broken/useless, if it lacks any checking to verify the builder actually produced what it was asked to build?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks like the test just wants to test field validation for object builders, one failure case, one success case. It doesn't care about what it actually builds. I said "bug" doesn't mean that the test is useless but it lacks the
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
it does seem like the empty drop helps this one usecase I will try and write a test that shows what happens when a builder isn't finishd and we can evaluate the issues |
||
| // Valid object should succeed | ||
| let mut list = builder.new_list(); | ||
| let mut valid_obj = list.new_object(); | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't actually remove
finishas there are still examples where the builders are in same scope with top-level builder. So explicit call offinishis still necessary for that.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's said, and, for the case these builders are in the same scope as the top-level builder, after this PR, if such a builder is not called
finish, it will be caught by the compiler now, e.g.,There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this sounds like a great idea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting! The compiler is normally willing to end the lifetime of the orphaned
outer_listjust before thebuilder.finishcall, but providing animpl Dropsomehow force-extends the lifetime?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No,
outer_list's lifetime normally will be ended (if nofinishcall) after the scope as usual. That's why the compiler complains because it borrowsbuilder. Sobuilder.finishcall cannot be happened before the end ofouter_listlifetime.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That only happens if there's an
impl Droptho:https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=fe049aba262d590948887ee08f43618c
Somehow, the compiler is able to shorten the lifetime of
outer_listifimpl Dropdoesn't get in the way.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't what this PR does?