Panic with `assertion failed: chunk_bytes <= 16 * 1024` when writing nested structs with V2.1 format

[Xuanwo]

Our users hit a case that lance can panic with assertion failed: chunk_bytes <= 16 * 1024 when writing nested structs with V2.1 format.

Error Message

thread 'lance-cpu' panicked at rust/lance-encoding/src/encodings/logical/primitive.rs:3336:13:
assertion failed: chunk_bytes <= 16 * 1024

Reproduction

The issue occurs when ALL of the following conditions are met:

1. Using Lance V2.1 format (data_storage_version = LanceFileVersion::V2_1)
2. Data structure is nested: struct<field: list<struct<...>>>
3. Inner struct has narrow fields (≤4 bytes each, e.g., Float32, Int32)
4. Single row contains >65k structs, highly compressed lead to our rep/def exceed the limits.