fix: apply SchemaAdapter in Updater

[westonpace]

This will ensure JSON columns are properly converted in paths like add_columns

[westonpace]

Leaving as draft until #5927 merges

[github-actions]

Code Review

Summary: This PR fixes JSON column handling in the add_columns path by applying SchemaAdapter in the Updater and preserving field metadata in merge_insert.

Analysis

The fix is correct: The changes address two issues where JSON extension metadata was being lost:

1. updater.rs: The Updater now uses SchemaAdapter.to_physical_batch() to convert Arrow JSON extension types to Lance's internal JSON format before writing. This ensures JSON columns added via add_columns are properly stored.

2. merge_insert/exec/write.rs: Changed from Field::new(name, type, nullable) to field.clone(). The original code discarded field metadata (including extension type info), causing JSON columns to lose their type information during merge_insert operations.

Notes

• The lazy initialization of schema_adapter (created on first batch) is appropriate since the data schema isn't known until the first batch arrives.
• The to_physical_batch method is a simple helper that checks requires_physical_conversion() before calling convert_json_columns(), avoiding unnecessary work for non-JSON data.
• Test coverage is comprehensive, covering append, compaction, add_columns, and merge_insert operations with JSON data.

No P0/P1 issues identified.

[codecov]

Codecov Report

❌ Patch coverage is 84.61538% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/updater.rs	87.50%	0 Missing and 1 partial ⚠️
rust/lance/src/dataset/utils.rs	80.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!