refactor: align blob behavior that write via file format version, read via layout

[Xuanwo]

This PR fixes a bug where users could encode blob v1 even when blob v2 was enabled. However, our decoder only reads the dataset's configuration lance.blob.version, which can lead to decoding issues.

In this PR, we changed the following:

• During write operations, we determine which blob layout to use based on file format version.
• During read operations, we decide which blob layout to decode based on its physical layout.

---

Parts of this PR were drafted with assistance from Codex (with gpt-5.2) and fully reviewed and edited by me. I take full responsibility for all changes.

[github-actions]

PR Review

Summary

This PR ensures that when blob v2 is enabled (file version >= 2.2), all blob-marked fields use the v2 encoder regardless of their input type (Binary/LargeBinary or Struct). The key addition is a normalize_blob_v2_input function that converts various input formats to the normalized v2 struct format.

Critical Issues (P0)

1. Potential panic from unchecked cast operations
In normalize_blob_v2_input, the code calls .as_binary::<i64>() and .as_string::<i32>() without checking if the actual array type matches. If the struct has a data field that is not LargeBinary or a uri field that is not Utf8, these will panic:

// blob.rs:455-457
let data_col = data_col.as_binary::<i64>();
let uri_col = uri_col.as_string::<i32>();

Consider using as_binary_opt and as_string_opt with proper error handling, similar to the pattern used in the Binary/LargeBinary branch.

Moderate Issues (P1)

2. Missing unit tests for normalize_blob_v2_input
The new normalize_blob_v2_input function handles several conversion paths (6-field struct passthrough, 2-field struct conversion, Binary, LargeBinary) but has no dedicated unit tests. Only the integration test in take.rs exercises one path. Add unit tests to cover:

• Invalid struct with wrong number of fields
• Struct with missing required fields
• Binary vs LargeBinary input normalization
• Null handling in various paths

3. Test assertion may be fragile
The test test_take_blob_v2_from_legacy_large_binary_on_v2_2 asserts exact field names and positions:

assert_eq!(struct_arr.fields()[0].name(), "kind");
assert_eq!(struct_arr.fields()[1].name(), "position");

Consider using column_by_name for field access instead of positional indexing, matching the pattern used in the encoder itself.

[codecov]

Codecov Report

❌ Patch coverage is 84.68750% with 49 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-encoding/src/encodings/logical/blob.rs	57.14%	18 Missing ⚠️
rust/lance/src/dataset/blob.rs	87.39%	13 Missing and 2 partials ⚠️
rust/lance/src/dataset/write/insert.rs	84.31%	5 Missing and 3 partials ⚠️
...ust/lance-encoding/src/encodings/logical/struct.rs	0.00%	5 Missing and 1 partial ⚠️
rust/lance-datafusion/src/projection.rs	88.88%	0 Missing and 1 partial ⚠️
rust/lance/src/dataset/fragment/write.rs	0.00%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[westonpace]

I think my initial question is why a user will specify v1 or v2?

[westonpace]

We should document this parameter. Make sure to clearly specify when a user would choose v1. Is it only if they need backwards compatibility with a legacy software? Are there any good reasons a user would choose v1?

[westonpace]

Can you drop a short comment explaining what this function is doing?

[Xuanwo]

I think my initial question is why a user will specify v1 or v2?

The original idea is to allow users to continue using blob v1 even after a new file version has been released. What do you think? Perhaps it would be better to require users to enable blob v2 on new datasets created by newer versions?

[westonpace]

Looks nice, thanks for bearing with the reviews!

[Xuanwo]

"index::vector::ivf::v2::tests::test_build_ivf_pq_4bit::case_3" is known to be a flaky test, just skip.