[Variant] Support creating nested objects and object with lists

[friendlymatthew]

Which issue does this PR close?

• Closes [Variant] Support Nested Data in VariantBuilder #7696
• Related to [Variant] Support nested lists and object lists #7740

Rationale for this change

This PR adds a feature to Variant::ObjectBuilder that enables constructing nested objects and objects with lists.

Are there any user-facing changes?

Adds two public methods to the ObjectBuilder API, new_list and new_object

[friendlymatthew]

This is the second half of the work required to fully support nested object and list building. I'd appreciate your thoughts!

After this, I'll follow up with PRs that address @scovich's comments in #7740.

cc @alamb @scovich

[alamb]

Thank you @friendlymatthew -- I went through this carefully and I think it looks great

As you might expect by now I had some suggestions, and I coded them up as another PR for your consideration:

• pydantic#2

[alamb]

Looking at these examples, it seems to me like the current API requires callingfinish -- it seems like it might be a nicer API if that just happened on drop() -- so you could use the scope of the example, like you have here, and the dropping would just complete the object

This would be a great follow on PR

[alamb]

• Filed [Variant] Remove explicit ObjectBuilder::finish() and ListBuilder::finish and move to Drop impl #7780

[alamb]

Given that this PR basically follows the existing pattern I am going to merge it in to keep the code flowing;

@PinkCrow007 @scovich @mkarbo @carpecodeum or @Weijun-H -- please feel free to review this PR and comment (or make a PR) for anything else you might find / see could be improved

[harshmotw-db]

@friendlymatthew Why are objects and lists treated as "pending" and not eagerly added? I am adding a function to parse JSON into Variant, and because of this, I am seeing a behavioral difference between Spark's parse_json expression and the rust implementation

[friendlymatthew]

@friendlymatthew Why are objects and lists treated as "pending" and not eagerly added? I am adding a function to parse JSON into Variant, and because of this, I am seeing a behavioral difference between Spark's parse_json expression and the rust implementation

Hi, by eagerly added do you mean to the parent builder? We can't add an object or list to the parent builder's buffer until the object/list itself has been fully created because each nested object/list has its own header and we can't know what the header will be.

What sort of behavioral differences are you seeing? If you call .finish() before operating on the builder, the object should be exactly the same as if it were to be done eagerly.

[harshmotw-db]

The difference I am seeing is that when creating a Variant out of the JSON string {"numbers": [4, -3e0, 1.001], "null": null, "booleans": [true, false]}, I am seeing "null" get an ID of 0 and "numbers" get an ID of 1. In Spark, I would expect "numbers" to get an ID of zero since it is the first key.

Adding self.check_pending_field() to the beginning of insert(...) fixes this. Is the current behavior intentional?

This is not a logical issue by any means though.

[friendlymatthew]

The difference I am seeing is that when creating a Variant out of the JSON string {"numbers": [4, -3e0, 1.001], "null": null, "booleans": [true, false]}, I am seeing "null" get an ID of 0 and "numbers" get an ID of 1. In Spark, I would expect "numbers" to get an ID of zero since it is the first key.

Adding self.check_pending_field() to the beginning of insert(...) fixes this. Is the current behavior intentional?

This is not a logical issue by any means though.

Good catch! I just put up a PR that should fix the issue above. #7786