feat: strategized plan compaction by zhangyue19921010 · Pull Request #5233 · lance-format/lance

zhangyue19921010 · 2025-11-13T08:28:21Z

codecov-commenter · 2025-11-13T08:56:06Z

Codecov Report

❌ Patch coverage is 95.77465% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/optimize.rs	95.77%	1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

westonpace

A few (mostly minor) suggestions

westonpace · 2025-11-13T21:55:33Z

+    // get all fragments by default
+    fn get_fragments(&self, dataset: &Dataset, _options: &CompactionOptions) -> Vec<FileFragment> {
+        // get_fragments should be returning fragments in sorted order (by id)
+        // and fragment ids should be unique
+        dataset.get_fragments()
+    }
+
+    // no filter by default
+    async fn filter_fragments(
+        &self,
+        _dataset: &Dataset,
+        fragments: Vec<FileFragment>,
+        _options: &CompactionOptions,
+    ) -> Result<Vec<FileFragment>> {
+        Ok(fragments)
+    }


Do these really need to be trait methods? I think we can probably leave them out and just let individual implementations use them if they want to. It will keep the trait simpler.

westonpace · 2025-11-13T21:56:30Z

+        Ok(fragments)
+    }
+
+    async fn plan(&self, dataset: &Dataset, options: &CompactionOptions) -> Result<CompactionPlan>;


Let's document this method.

westonpace · 2025-11-13T21:57:55Z

+        Ok(fragments)
+    }
+
+    async fn plan(&self, dataset: &Dataset, options: &CompactionOptions) -> Result<CompactionPlan>;


Will CompactionOptions be flexible enough for all possible strategies? Should we maybe accept options as a JSON string or a Map<String, String>? This way different strategies can expose their own custom options. That would leave the API a little less defined but it would be more flexible.

Do we even need to take CompactionOptions? Maybe it should be a argument to the constructor of the individual structs. That way each could have their own arguments but also be strongly typed.

Will CompactionOptions be flexible enough for all possible strategies? Should we maybe accept options as a JSON string or a Map<String, String>? This way different strategies can expose their own custom options. That would leave the API a little less defined but it would be more flexible.

Hi @westonpace Thanks a lot for your review. Added Map<String, String> for flexible.

Do we even need to take CompactionOptions? Maybe it should be a argument to the constructor of the individual structs. That way each could have their own arguments but also be strongly typed.

Hi @wjones127 Thanks a lot for your review.

I have tried several ways to eliminate the CompactionOptions parameter in the plan method, but none of them are perfect :( The main contradiction is that during users start planning compaction based on the built planner, they may dynamically adjust the options parameters on certain conditions, such as

https://github.com/lancedb/lance/blob/254a8217ac26666585983aa7ec8c4234f4c3f99f/rust/lance/src/dataset/optimize.rs#L225

If options are passed in when building the planner, then after modifying the options subsequently, it must also be ensured that the options in the planner can be seen. Therefore, we need Arc + mutex and cannot use clone.

On the contrary, it might be simpler and more flexible to pass in the desired options each time the plan method is called here.

If options are passed in when building the planner, then after modifying the options subsequently, it must also be ensured that the options in the planner can be seen. Therefore, we need Arc + mutex and cannot use clone.

I don't understand this. The logic of validate() can live in the planner and be internal.

If I were to rewrite compact_files, I would do:

pub async fn compact_files( dataset: &mut Dataset, mut options: CompactionOptions, remap_options: Option<Arc<dyn IndexRemapperOptions>>, // These will be deprecated later ) -> Result<CompactionMetrics> { info!(target: TRACE_DATASET_EVENTS, event=DATASET_COMPACTING_EVENT, uri = &dataset.uri); // .validate() now happens inside of `from_options` let planner = DefaultCompactionPlanner::from_options(options); compact_files_with_planner(dataset, &planner, remap_options).await }

changed as you suggested ~

westonpace · 2025-11-13T21:59:06Z

 /// Compacts the files in the dataset without reordering them.
 ///
-/// This does a few things:
+/// By default, his does a few things:


Suggested change

/// By default, his does a few things:

/// By default, this does a few things:

westonpace · 2025-11-13T21:59:27Z

 pub async fn compact_files(
+    dataset: &mut Dataset,
+    options: CompactionOptions,
+    remap_options: Option<Arc<dyn IndexRemapperOptions>>, // These will be deprecated later


zhangyue19921010 · 2025-11-19T02:48:02Z

Hi @westonpace and @wjones127 Thanks a lot for your review. All comments are addressed. PTAL :)

wjones127

Sorry for the delay, I accidentally left my comments pending. I'm still not sure about the design. It seems like it could be simplified further.

wjones127 · 2025-11-17T18:23:48Z

+    // get all fragments by default
+    fn get_fragments(&self, dataset: &Dataset, _options: &CompactionOptions) -> Vec<FileFragment> {
+        // get_fragments should be returning fragments in sorted order (by id)
+        // and fragment ids should be unique
+        dataset.get_fragments()
+    }


Was already commented on, but do we need this? It seems like individual implementations can just call dataset.get_fragments() and then do whatever filtering they would like.

wjones127 · 2025-11-17T18:39:20Z

+        Ok(fragments)
+    }
+
+    async fn plan(&self, dataset: &Dataset, options: &CompactionOptions) -> Result<CompactionPlan>;


If options are passed in when building the planner, then after modifying the options subsequently, it must also be ensured that the options in the planner can be seen. Therefore, we need Arc + mutex and cannot use clone.

I don't understand this. The logic of validate() can live in the planner and be internal.

If I were to rewrite compact_files, I would do:

pub async fn compact_files( dataset: &mut Dataset, mut options: CompactionOptions, remap_options: Option<Arc<dyn IndexRemapperOptions>>, // These will be deprecated later ) -> Result<CompactionMetrics> { info!(target: TRACE_DATASET_EVENTS, event=DATASET_COMPACTING_EVENT, uri = &dataset.uri); // .validate() now happens inside of `from_options` let planner = DefaultCompactionPlanner::from_options(options); compact_files_with_planner(dataset, &planner, remap_options).await }

zhangyue19921010 · 2025-11-25T02:20:08Z

Sorry for the delay, I accidentally left my comments pending. I'm still not sure about the design. It seems like it could be simplified further.

No worried @westonpace . Really appreciate for your response. Also all comments are addressed. PTAL~

wjones127

Just got back from vacation, so sorry for the delay.

This is headed in the right direction. I still have some comments on the API.

wjones127 · 2025-12-04T14:59:11Z

+
+    fn get_compaction_options(&self) -> &CompactionOptions;


Why is this necessary? The CompactionPlan already contains the CompactionOptions, so isn't the return value of plan sufficient?

wjones127 · 2025-12-04T15:05:58Z

+    async fn plan(
+        &self,
+        dataset: &Dataset,
+        configs: HashMap<String, String>,


Why do we need configs? The point of my earlier suggestion was to get rid of string typed configurations in favor of proper constructors. For example:

pub trait CompactionPlanner: Send + Sync { async fn plan(&self, dataset: &Dataset) -> Result<CompactionPlan>; } pub struct DefaultCompactionPlanner { options: CompactionOptions, } impl CompactionPlanner for DefaultCompactionPlanner { async fn plan(&self, dataset: &Dataset) -> Result<CompactionPlan> { let tasks = todo!(); Ok(CompactionPlan { tasks, read_version: dataset.manifest.version, options: self.options.clone() }) } } pub struct CustomPlanner { config_a: i32, config_b: Duration, options: CompactionOptions } impl CustomPlanner { fn new(config_a: i32, config_b: Duration, options: CompactionOptions) -> { todo!() } } impl CompactionPlanner for CustomPlanner { async fn plan(&self, dataset: &Dataset) -> Result<CompactionPlan> { let tasks = todo!("Use config_a and config_b to plan tasks"); Ok(CompactionPlan { tasks, read_version: dataset.manifest.version, options: self.options.clone() }) } }

Notice how CustomPlanner:new() takes strongly-typed parameters. We no long have to validate string inputs, which makes the API safer and easier to use.

BTW we should probably distinguish from configurations for planning and configurations for executing. The reason we need CompactionOptions later is they are needed for execution. But the options that will vary for planners are the planning options.

BTW we should probably distinguish from configurations for planning and configurations for executing. The reason we need CompactionOptions later is they are needed for execution. But the options that will vary for planners are the planning options.

I quickly glanced through it, and there are quite a few parts that need to be modified. I will try to create a new PR to solve this problem ：）

zhangyue19921010 · 2025-12-05T14:03:50Z

Just got back from vacation, so sorry for the delay.

This is headed in the right direction. I still have some comments on the API.

Hi @wjones127. Thanks for your response and wish u have a fantastic vacation! I will address the comments asap :)

Copilot

Pull request overview

This PR introduces a strategy pattern for compaction planning in Lance, allowing users to implement custom compaction strategies via the new CompactionPlanner trait. The existing compaction logic has been refactored into a DefaultCompactionPlanner implementation, maintaining backward compatibility while enabling extensibility.

Adds CompactionPlanner trait for custom compaction strategies
Refactors existing compaction planning logic into DefaultCompactionPlanner
Introduces compact_files_with_planner API to enable custom planner usage

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated 3 comments.

File	Description
rust/lance/src/dataset/optimize.rs	Adds `CompactionPlanner` trait and `DefaultCompactionPlanner` struct; refactors `plan_compaction` logic into the default planner; adds `compact_files_with_planner` function; updates documentation; adds test for default planner
java/lance-jni/Cargo.lock	Updates `lance-namespace-impls` version from 1.0.0-beta.4 to 1.0.0-beta.7

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

zhangyue19921010 · 2025-12-09T07:44:53Z

Hi @wjones127 Really appreciate for your help. Comments are all addressed . PTAL ~

wjones127

This looks great. Thanks for working with me on this.

zhangyue19921010 · 2025-12-10T02:24:55Z

This looks great. Thanks for working with me on this.

Thanks @wjones127 :) Will optimize the compaction plan strategy, based on the horizontal and tiered vertical strategies we discuss before as next step ~

Close lance-format#5186

feat: strategized plan compaction

b26776b

github-actions Bot added the enhancement New feature or request label Nov 13, 2025

westonpace reviewed Nov 13, 2025

View reviewed changes

code review

18eea6d

Merge branch 'main' into feat_strategized_plan_compaction_v2

9cdedb4

wjones127 reviewed Nov 21, 2025

View reviewed changes

code review

53954c5

wjones127 reviewed Dec 4, 2025

View reviewed changes

code review

801bebe

Copilot AI review requested due to automatic review settings December 9, 2025 07:13

github-actions Bot added the java label Dec 9, 2025

Copilot started reviewing on behalf of zhangyue19921010 December 9, 2025 07:14 View session

Copilot AI reviewed Dec 9, 2025

View reviewed changes

Comment thread rust/lance/src/dataset/optimize.rs Outdated

Comment thread rust/lance/src/dataset/optimize.rs Outdated

Comment thread rust/lance/src/dataset/optimize.rs

zhangyue19921010 added 4 commits December 9, 2025 15:19

code review

983b3a6

code review

0d0fb13

Merge branch 'main' into feat_strategized_plan_compaction_v2

2360768

code review

f15d6bb

wjones127 approved these changes Dec 9, 2025

View reviewed changes

wjones127 merged commit b9129d8 into lance-format:main Dec 9, 2025
25 of 27 checks passed

jackye1995 pushed a commit to jackye1995/lance that referenced this pull request Jan 21, 2026

feat: strategized plan compaction (lance-format#5233)

1bafa4e

Close lance-format#5186

andrea-reale mentioned this pull request Mar 30, 2026

emilk/fix write starvation rerun-io/lance#12

Closed

	/// By default, his does a few things:
	/// By default, this does a few things:

Conversation

zhangyue19921010 commented Nov 13, 2025

Uh oh!

codecov-commenter commented Nov 13, 2025 • edited by codecov Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 commented Nov 19, 2025

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 commented Nov 25, 2025

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 commented Dec 5, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhangyue19921010 commented Dec 9, 2025

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhangyue19921010 commented Dec 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

codecov-commenter commented Nov 13, 2025 •

edited by codecov Bot

Loading