feat: support add sub-column to struct col by wojiaodoubao · Pull Request #5126 · lance-format/lance

wojiaodoubao · 2025-11-03T12:50:24Z

Adding sub-columns to a struct data type column.

codecov-commenter · 2025-11-05T14:06:34Z

Codecov Report

❌ Patch coverage is 99.14163% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/schema_evolution.rs	99.13%	0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

wojiaodoubao · 2025-11-11T03:57:46Z

In this pr, adding sub-columns to struct column only supports versions 2.1 and later. Considering that the v2.0 format will gradually be phased out, I think it might be unnecessary to support adding sub-columns for v2.0.

If we need to support adding sub-columns for v2.0, the main challenge is: In v2.0, a header column is stored for nested columns (struct, list), and it assumes the same header column will not exist multiple times within a single fragment.

There are two possible approaches:

Approach 1: Keep the encoding and decoding unchanged. New files will include the new leaf columns and redundant header columns, and when reading the header column, always select the first header column found.
Approach 2: Modify the encoding and decoding. New files will only include the new leaf columns and will not include redundant header columns (this requires significant changes).

wojiaodoubao · 2025-11-11T03:58:11Z

The failures are caused by #5206.

wojiaodoubao · 2025-11-11T03:59:16Z

Hi @jackye1995 , could you help review when you have time, looking forward to your suggestions!

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2025-11-11T04:02:59Z

+    async fn test_add_sub_column_to_packed_struct_col(version: LanceFileVersion) {
+        let mut metadata = HashMap::new();
+        metadata.insert("lance-encoding:packed".to_string(), "true".to_string());
+        let mut dataset = prepare_initial_dataset_with_struct_col(version, metadata).await;


Import HashMap for new struct column tests

The added tests call HashMap::new() (and pass HashMap<String, String> to helpers) but the surrounding test module never imports std::collections::HashMap. As written these tests won’t compile (cannot find type HashMap in this scope) and will block CI. Add the missing import or qualify the type before using it.

Useful? React with 👍 / 👎.

wojiaodoubao · 2025-11-27T13:46:26Z

This is a prerequisite for the variant (#5238). Hi @jackye1995 , could you help review this when you have time, thanks very much~

Xuanwo · 2025-12-03T07:20:54Z

File format v2.1 has been stabilized, and we should not add any more features to it. Instead, please target file format 2.2.

wojiaodoubao · 2025-12-03T11:52:20Z

File format v2.1 has been stabilized, and we should not add any more features to it. Instead, please target file format 2.2.

@Xuanwo Thanks your reminding! Marked v2.1 as unsupported in adding sub-column.

wojiaodoubao · 2025-12-03T12:35:25Z

Fixed all comments. Hi @jackye1995 , please help review when you have time, thanks very much~

wojiaodoubao · 2025-12-04T08:33:43Z

I rebased this pr and it is ready for review now. Hi @jackye1995 , please help when you have time, thanks very much~.

jackye1995

minor error message comment, other things look good to me

jackye1995 · 2025-12-08T06:43:31Z

+        (DataType::FixedSizeList(fl, _), DataType::FixedSizeList(fr, _)) => {
+            check_field_conflict(fl, fr, version)
+        }
+        (_, _) => Err(Error::invalid_input(


sorry I guess this actually need 2 different error messages, one is this one and one is the original one, because if the type is the same then this message is not really right. could you update that?

Adding sub-columns to a struct data type column. --------- Co-authored-by: lijinglun <lijinglun@bytedance.com>

github-actions Bot added the enhancement New feature or request label Nov 3, 2025

wojiaodoubao force-pushed the add-sub-column branch 2 times, most recently from dd91de3 to b49887f Compare November 5, 2025 13:14

jackye1995 self-requested a review November 7, 2025 05:29

wojiaodoubao marked this pull request as draft November 10, 2025 13:41

wojiaodoubao force-pushed the add-sub-column branch from b49887f to a633fc0 Compare November 11, 2025 03:19

wojiaodoubao marked this pull request as ready for review November 11, 2025 03:58

chatgpt-codex-connector Bot reviewed Nov 11, 2025

View reviewed changes

wojiaodoubao force-pushed the add-sub-column branch 2 times, most recently from 9479118 to aaf562e Compare November 18, 2025 09:44

wojiaodoubao force-pushed the add-sub-column branch from aaf562e to 79ba8bb Compare November 27, 2025 13:44

zhangyue19921010 mentioned this pull request Nov 27, 2025

feat(schema evolution): support adding nested struct columns #5361

Closed

jackye1995 reviewed Dec 2, 2025

View reviewed changes

Comment thread rust/lance/src/dataset/schema_evolution.rs

jackye1995 reviewed Dec 2, 2025

View reviewed changes

Comment thread rust/lance/src/dataset/schema_evolution.rs

jackye1995 reviewed Dec 2, 2025

View reviewed changes

Comment thread rust/lance/src/dataset.rs Outdated

jackye1995 reviewed Dec 2, 2025

View reviewed changes

Comment thread rust/lance/src/dataset.rs Outdated

wojiaodoubao force-pushed the add-sub-column branch 2 times, most recently from ffb02f1 to f51d118 Compare December 3, 2025 11:51

wojiaodoubao force-pushed the add-sub-column branch from f51d118 to ef48b26 Compare December 3, 2025 12:04

wojiaodoubao force-pushed the add-sub-column branch 2 times, most recently from 7987045 to c2c0778 Compare December 4, 2025 07:02

github-actions Bot added the python label Dec 4, 2025

wojiaodoubao force-pushed the add-sub-column branch 2 times, most recently from 06ad2d6 to 062d6b3 Compare December 4, 2025 08:03

jackye1995 approved these changes Dec 8, 2025

View reviewed changes

lijinglun and others added 2 commits December 8, 2025 21:48

feat: support add sub-column to struct col

db93b87

fix

d7681a3

wojiaodoubao force-pushed the add-sub-column branch from 062d6b3 to d7681a3 Compare December 8, 2025 14:02

jackye1995 merged commit 2671a2b into lance-format:main Dec 10, 2025
27 checks passed

jackye1995 pushed a commit to jackye1995/lance that referenced this pull request Jan 21, 2026

feat: support add sub-column to struct col (lance-format#5126)

1f5dee4

Adding sub-columns to a struct data type column. --------- Co-authored-by: lijinglun <lijinglun@bytedance.com>

andrea-reale mentioned this pull request Mar 30, 2026

emilk/fix write starvation rerun-io/lance#12

Closed

Conversation

wojiaodoubao commented Nov 3, 2025

Uh oh!

codecov-commenter commented Nov 5, 2025 • edited by codecov Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

wojiaodoubao commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wojiaodoubao commented Nov 11, 2025

Uh oh!

wojiaodoubao commented Nov 11, 2025

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

wojiaodoubao commented Nov 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Xuanwo commented Dec 3, 2025

Uh oh!

wojiaodoubao commented Dec 3, 2025

Uh oh!

wojiaodoubao commented Dec 3, 2025

Uh oh!

wojiaodoubao commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

jackye1995 Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Nov 5, 2025 •

edited by codecov Bot

Loading

wojiaodoubao commented Nov 11, 2025 •

edited

Loading

wojiaodoubao commented Dec 4, 2025 •

edited

Loading