[Variant] Add VariantBuilder::new_with_buffers to write to existing buffers

[alamb]

Which issue does this PR close?

• closes [Variant] Support VariantBuilder to write to buffers owned by the caller #7805
• part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736
• part of Convert JSON to VariantArray without copying (8 - 32% faster) #7911

Rationale for this change

I would like to be able to write Variants directly into the target buffer when writing multiple variants However, the current VariantBuilder allocates a new bufffer for each variant

What changes are included in this PR?

1. Add VariantBuilder::new_with_buffers and docs and tests

You can see how this API can be used to write directly into a buffer in VariantArrayBuilder in this PR:

• Convert JSON to VariantArray without copying (8 - 32% faster) #7911

Are these changes tested?

Yes new tests

Are there any user-facing changes?

New API

[friendlymatthew]

this is a really cool test case

[alamb]

I would have loved to append the variant but I hit this bug:

• [Variant] Panic when appending nested objects to VariantBuilder #7907

[friendlymatthew]

If you apply this diff, it should give you what you need: #7914.

I don't love that we need to use a builder to reencode the object/list. I'm going to spend some time thinking about a more elegant solution.

I wonder if this PR with its existing buffers can help 🤔

[alamb]

If you apply this diff, it should give you what you need: #7914.

Thank you @friendlymatthew -- that looks great ❤️ -- I left a few comments

I don't love that we need to use a builder to reencode the object/list. I'm going to spend some time thinking about a more elegant solution.

I think potentially a more performant solution would be to optimize #7914 to copy the Variant bytes directly and then update the field_ids.

However, that will get tricky I think if the field_ids in the variant under construction are different sizes than the old one.

I suggest we start with the simple thing (what you have) and then we can optimize if/when we have benchmarks that show it would help

[alamb]

I originally split this PR out from #7911 thinking it would make it eaiser to review

However, @scovich has been providing great feedback directly on #7911 directly, and thus to avoid too many PRs to keep track of I plan to merge this one in and keep working on #7911