feat(blob_v2): add dedicated blob support by Xuanwo · Pull Request #5406 · lance-format/lance

Xuanwo · 2025-12-04T09:57:04Z

This PR will add dedicated blob support in lance.

Parts of this PR were drafted with assistance from Codex (with gpt-5.1-codex-max) and fully reviewed and edited by me. I take full responsibility for all changes.

Signed-off-by: Xuanwo <github@xuanwo.io>

github-actions · 2025-12-04T09:57:20Z

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

Signed-off-by: Xuanwo <github@xuanwo.io>

codecov · 2025-12-04T12:20:27Z

Codecov Report

❌ Patch coverage is 47.61905% with 198 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/blob.rs	6.52%	172 Missing ⚠️
rust/lance/src/dataset/fragment/write.rs	42.85%	6 Missing and 2 partials ⚠️
rust/lance-arrow/src/schema.rs	0.00%	7 Missing ⚠️
rust/lance-encoding/src/encodings/logical/blob.rs	95.34%	5 Missing and 1 partial ⚠️
rust/lance/src/dataset/write.rs	70.58%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

Signed-off-by: Xuanwo <github@xuanwo.io>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Signed-off-by: Xuanwo <github@xuanwo.io>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Signed-off-by: Xuanwo <github@xuanwo.io>

westonpace

I love the way this is shaping up. Great work!

westonpace · 2025-12-05T13:10:34Z


+const DEDICATED_THRESHOLD: usize = 4 * 1024 * 1024;
+
+pub struct BlobPreprocessor {


Document what this is?

westonpace · 2025-12-05T13:11:33Z

+
+    fn next_blob_id(&mut self) -> u32 {
+        let id = self.local_counter;
+        self.local_counter = self.local_counter.wrapping_add(1);


Instead of wrapping_add maybe checked_add and return an error? Also, why not use u64 for counter? Unlikely we will ever hit u32 limit but it is just barely conceivable and better to just not worry about it.

In our current design, the blob ID is unique per data file (fragement + field), so it shares the same upper limit as the data file rows: u32::MAX. It should be safe to use u32 here.

I will change this place to just +=1.

westonpace · 2025-12-05T13:13:25Z

+///
+/// Layout: `<base>/<data_file_key>/<blob_id>.raw`
+/// - `base` is typically the dataset's data directory.
+/// - `data_file_key` is the stem of the data file (without extension).


Interesting. I hadn't expected path to include data_file_key but I don't see any harm in it and it could maybe be useful for things like cleanup?

Yes! I picked up this design to make the GC much easier: we can just remove all blob files under the same data_file_key once we decide to remove that data file.

westonpace · 2025-12-05T13:17:31Z

+        Ok(path)
+    }
+
+    pub(crate) async fn preprocess_batch(&mut self, batch: &RecordBatch) -> Result<RecordBatch> {


This requires blobs to be top-level fields. E.g. we cannot have Struct or List. I think this is fine for now but we should make sure we eventually document this somewhere.

Oh yeah, nice catch. I have to admit I didn’t take enough consideration into this and I actually didn’t expect this. I’ll create a follow up.

westonpace · 2025-12-05T13:18:59Z

+            new_fields.push(Arc::new(
+                arrow_schema::Field::new(
+                    field.name(),
+                    ArrowDataType::Struct(child_fields.into()),
+                    field.is_nullable(),
+                )
+                .with_metadata(field.metadata().clone()),


Do we need to mark the field as packed?

No, this record batch only exists in memory and we won’t really encode it.

Signed-off-by: Xuanwo <github@xuanwo.io>

This PR will add packed blob support for lance. This PR is based on #5406, please review that first. --- **Parts of this PR were drafted with assistance from Codex (with `gpt-5.1-codex-max`) and fully reviewed and edited by me. I take full responsibility for all changes.** --------- Signed-off-by: Xuanwo <github@xuanwo.io>

This PR will add dedicated blob support in lance. --- **Parts of this PR were drafted with assistance from Codex (with `gpt-5.1-codex-max`) and fully reviewed and edited by me. I take full responsibility for all changes.** --------- Signed-off-by: Xuanwo <github@xuanwo.io>

This PR will add packed blob support for lance. This PR is based on lance-format#5406, please review that first. --- **Parts of this PR were drafted with assistance from Codex (with `gpt-5.1-codex-max`) and fully reviewed and edited by me. I take full responsibility for all changes.** --------- Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo added 2 commits December 3, 2025 12:54

Save work

deb1db4

Signed-off-by: Xuanwo <github@xuanwo.io>

save work

f4ac526

Signed-off-by: Xuanwo <github@xuanwo.io>

github-actions Bot added the enhancement New feature or request label Dec 4, 2025

Xuanwo added 4 commits December 4, 2025 18:29

Merge remote-tracking branch 'origin/main' into xuanwo/blobv2-dedicated

3f7334d

Signed-off-by: Xuanwo <github@xuanwo.io>

Fix build

24f391d

Signed-off-by: Xuanwo <github@xuanwo.io>

remove not needed checks

b3a6b17

Signed-off-by: Xuanwo <github@xuanwo.io>

refactor

438d3d4

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo changed the title ~~feat(blob_v2): add dedicated blob support~~ feat(blob_v2): add dedicated blob support Dec 4, 2025

Xuanwo added 4 commits December 4, 2025 19:04

refactor blob logic

d949ed9

Signed-off-by: Xuanwo <github@xuanwo.io>

refactor

e0d03d8

Signed-off-by: Xuanwo <github@xuanwo.io>

Remove not needed comment

b73d3b3

Signed-off-by: Xuanwo <github@xuanwo.io>

Fix build

151e4de

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo added 5 commits December 4, 2025 22:25

Fix build

b8a07d3

Signed-off-by: Xuanwo <github@xuanwo.io>

Fix existsing tests

51816ed

Signed-off-by: Xuanwo <github@xuanwo.io>

Add tests

6f522ba

Signed-off-by: Xuanwo <github@xuanwo.io>

Polish blob id

55d645e

Signed-off-by: Xuanwo <github@xuanwo.io>

polish

f24739b

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo commented Dec 4, 2025

View reviewed changes

Comment thread rust/lance-core/src/utils/blob.rs Outdated

Comment thread rust/lance/src/dataset/blob.rs

Xuanwo marked this pull request as ready for review December 4, 2025 16:11

chatgpt-codex-connector Bot reviewed Dec 4, 2025

View reviewed changes

Comment thread rust/lance/src/dataset/write.rs Outdated

Xuanwo marked this pull request as draft December 5, 2025 02:56

use better logic for blob path

0289d8d

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo marked this pull request as ready for review December 5, 2025 05:59

chatgpt-codex-connector Bot reviewed Dec 5, 2025

View reviewed changes

Comment thread rust/lance/src/dataset/blob.rs

Fix uri should use utf-8 instead of large-utf8

026a12c

Signed-off-by: Xuanwo <github@xuanwo.io>

Xuanwo mentioned this pull request Dec 5, 2025

feat(blob_v2): add packed blob support #5413

Merged

westonpace approved these changes Dec 5, 2025

View reviewed changes

Address comments

682fee5

Signed-off-by: Xuanwo <github@xuanwo.io>

Merge branch 'main' into xuanwo/blobv2-dedicated

93f1977

Xuanwo mentioned this pull request Dec 5, 2025

Tracking issues for Blob V2 #4947

Open

9 tasks

Xuanwo merged commit d452750 into main Dec 5, 2025
25 checks passed

Xuanwo deleted the xuanwo/blobv2-dedicated branch December 5, 2025 16:46

andrea-reale mentioned this pull request Mar 30, 2026

emilk/fix write starvation rerun-io/lance#12

Closed

DanielMao1 mentioned this pull request Apr 21, 2026

feat(blob_v2): add ref_id deduplication across Inline/Packed/Dedicated #6587

Open


		const DEDICATED_THRESHOLD: usize = 4 * 1024 * 1024;

		pub struct BlobPreprocessor {

Conversation

Xuanwo commented Dec 4, 2025

Uh oh!

github-actions Bot commented Dec 4, 2025

Uh oh!

codecov Bot commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Dec 4, 2025 •

edited

Loading