Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
29c7b93
rough cascading compaction impl
capistrant Dec 19, 2025
ae21c82
Renaming refactor
capistrant Jan 15, 2026
71830fc
add support for more config knobs in the cascading reindex spec
capistrant Jan 15, 2026
1940267
add some testing
capistrant Jan 15, 2026
bdb514e
stop using forbidden apis
capistrant Jan 16, 2026
0f71da2
working on some test coverage
capistrant Jan 17, 2026
acad478
simplify search interval creation and enhance embedded test for casca…
capistrant Jan 20, 2026
d49fbf4
fixup checkstyle
capistrant Jan 20, 2026
5efb4bf
temporary fixup to test
capistrant Jan 20, 2026
5a87759
fix checkstyle
capistrant Jan 20, 2026
0850a8f
remove native runner for one compaction supervisor test due to native…
capistrant Jan 21, 2026
3667456
refactorings from self review
capistrant Jan 22, 2026
a280690
Fixup naming to prefer reindexing over compaction
capistrant Jan 22, 2026
3108b61
fix up a javadoc with up to date design spec
capistrant Jan 22, 2026
1f671e1
Fill in UT gaps for the composing provider
capistrant Jan 22, 2026
3be0da1
refactor test class for inline rule provider
capistrant Jan 22, 2026
79ff44b
Self review refactorings
capistrant Jan 22, 2026
420f3b2
Trying to transform cascadingreindexingtemplate to a compaction state…
capistrant Jan 22, 2026
bf2e02d
refactor the location of the reindexing filter rule optimizer
capistrant Jan 23, 2026
5b4f3d2
Refactor this idea of additivity and how it works for building configs
capistrant Jan 23, 2026
6f5ead7
Add a missing test class
capistrant Jan 23, 2026
6d1fc6e
fix checkstyle
capistrant Jan 23, 2026
799db27
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant Jan 23, 2026
6853b02
clean up a javadoc
capistrant Jan 23, 2026
2200467
trivial fixes
capistrant Jan 23, 2026
a0d68eb
Prevent an edge case for a negative period
capistrant Jan 23, 2026
bbb5bbd
fix a nasty bug opportunity
capistrant Jan 24, 2026
4e2dcb0
Add support to filter on nested columns for MSQ reindexing using virt…
capistrant Jan 26, 2026
77fea05
fix forbidden misses
capistrant Jan 27, 2026
a39c7d3
Wait or data availability before querying more than segment counts
capistrant Jan 27, 2026
816c7f9
Try to clear up confusion by renaming period to olderThan to signify …
capistrant Jan 28, 2026
960353e
Rename filter to deleteWhere in ReindexingFilterRule to make it clear…
capistrant Jan 28, 2026
f275a98
Rename the ReindexingFilterRule to ReindexingDeletionRule to clarify …
capistrant Jan 28, 2026
62cc519
Make projections rule not be additive
capistrant Jan 28, 2026
d706539
fix a test now that projections rules are not additive
capistrant Jan 28, 2026
5c3b5c5
Add two new skip offset concepts to cascading reindexing template + i…
capistrant Jan 29, 2026
0fbbeb2
Split out segment and query granularity rules
capistrant Jan 29, 2026
0ea46b7
Refactor the non overlapping timeline build for cascading reindexing
capistrant Jan 30, 2026
4ad3a90
Add a forced segment gran field to supervisor and some tests
capistrant Jan 30, 2026
a97fd18
Remove unused method
capistrant Jan 30, 2026
b58f026
fix import
capistrant Feb 2, 2026
f8d3be8
Fixing imports
capistrant Feb 2, 2026
49bd2fc
fix a test
capistrant Feb 2, 2026
c3197ba
cleanup
capistrant Feb 2, 2026
89763bc
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant Feb 4, 2026
457d912
fix imports after merging master
capistrant Feb 4, 2026
b44882b
fixup test file
capistrant Feb 4, 2026
846ff7c
test updates
capistrant Feb 4, 2026
ce3e1f6
minor
capistrant Feb 4, 2026
d988908
Replace Selector with Equals and In with TypedIn
capistrant Feb 6, 2026
3f75762
Add strict enforcement of segment granularity changes
capistrant Feb 6, 2026
cbacb9f
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant Feb 8, 2026
2293e86
Fix bug introduced that was breaking MVD in compaction
capistrant Feb 9, 2026
91d4a5c
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant Feb 9, 2026
75c4329
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant Feb 10, 2026
7030054
Add ReindexingDataSchema rule that will replace other rule types
capistrant Feb 13, 2026
6f1bd02
Tear out rules that are now consolidated into data schema rules
capistrant Feb 13, 2026
f9d020f
checkpointed api for getting a rule timeline
capistrant Feb 13, 2026
ebf40cf
naming refactor
capistrant Feb 14, 2026
d510cb4
Refactors and UTs
capistrant Feb 14, 2026
091a0d3
bug fixes for timeline
capistrant Feb 16, 2026
3dd6069
Support dynamic discovery of skip intervals when skipOffsetFromLatest…
capistrant Feb 16, 2026
ceafdc7
Cleanup some ugly date parsing in supervisor timeline view
capistrant Feb 16, 2026
eb26cd1
fix a rendering bug for the UI component
capistrant Feb 16, 2026
66d6815
web console testing and fixes found while writing tests
capistrant Feb 16, 2026
bbb05b9
clean refactor of reindexing config builder
capistrant Feb 16, 2026
a39f222
clean up refactor of cascading reindexing template
capistrant Feb 16, 2026
6c842a7
self review cleanup
capistrant Feb 16, 2026
33fc482
add missing tests for supervisor resource api
capistrant Feb 16, 2026
ccd6c71
Refactor exception class for timeline to be more descriptive
capistrant Feb 16, 2026
c7ed4e1
remove redundant naming patterns
capistrant Feb 16, 2026
f6ef9a9
Add some java docs
capistrant Feb 16, 2026
83766fa
refactor how the final config customization works for cascading reind…
capistrant Feb 16, 2026
f07b97a
fix serde issues I was having
capistrant Feb 16, 2026
707904f
fixup random bits after CI ran
capistrant Feb 16, 2026
a24cd32
waiting for segments to be available should avoid issues with segs th…
capistrant Feb 17, 2026
bfff8de
Move ReindexingConfigBuilder to a hopefully better place
capistrant Feb 17, 2026
5fa4146
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant Feb 17, 2026
3f2747c
Remove web-console changes from this PR
capistrant Feb 17, 2026
0d0d0c5
Remove support for the reindexing timeline creation from app code
capistrant Feb 17, 2026
4b211a0
missed a few spots
capistrant Feb 17, 2026
387eb5a
fix silly jackson serde error
capistrant Feb 18, 2026
34d31ab
addressing review comments
capistrant Feb 19, 2026
30d2d77
Create strict allow list for segment granularity options
capistrant Feb 19, 2026
a381179
add some compaction transform serde tests out of paranoia
capistrant Feb 19, 2026
e560a97
All P0D older than to essentially make all data from referance and ol…
capistrant Feb 19, 2026
408208c
fix test that did not get updated when CompactionTransformSpec update…
capistrant Feb 19, 2026
16b25af
Fix improper hashing in the data schema rule
capistrant Feb 19, 2026
216d984
move rule tests to junit5
capistrant Feb 19, 2026
619a0d1
migrate the rule provider test files to junit5
capistrant Feb 19, 2026
b55f7d8
Add junit5 deps to indexing-service and migrate new test files to junit5
capistrant Feb 19, 2026
889d876
Add vintage engine so junit4 and 5 can coexist in indexing-service
capistrant Feb 19, 2026
8a14e85
Clean up ReindexingRule invalid input handling. Throw DruidException
capistrant Feb 20, 2026
d3e8917
Fix the transform spec equality check in compactionstatus
capistrant Feb 20, 2026
ede3596
Fixup checkstyle
capistrant Feb 20, 2026
54a4b6a
fix regression now that our check on transform spec is more involved
capistrant Feb 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1528,7 +1528,7 @@ public void testAutoCompactionDutyWithFilter(boolean useSupervisors) throws Exce
NO_SKIP_OFFSET,
null,
null,
new CompactionTransformSpec(new SelectorDimFilter("page", "Striker Eureka", null)),
new CompactionTransformSpec(new SelectorDimFilter("page", "Striker Eureka", null), null),
null,
false,
CompactionEngine.NATIVE
Expand Down

Large diffs are not rendered by default.

20 changes: 20 additions & 0 deletions indexing-service/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,26 @@
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-migrationsupport</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.easymock</groupId>
<artifactId>easymock</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import com.fasterxml.jackson.databind.module.SimpleModule;
import com.google.common.collect.ImmutableList;
import com.google.inject.Binder;
import org.apache.druid.indexing.compact.CascadingReindexingTemplate;
import org.apache.druid.indexing.compact.CompactionSupervisorSpec;
import org.apache.druid.indexing.overlord.supervisor.SupervisorStateManagerConfig;
import org.apache.druid.indexing.scheduledbatch.ScheduledBatchSupervisorSpec;
Expand All @@ -47,7 +48,8 @@ public List<? extends Module> getJacksonModules()
new SimpleModule(getClass().getSimpleName())
.registerSubtypes(
new NamedType(CompactionSupervisorSpec.class, CompactionSupervisorSpec.TYPE),
new NamedType(ScheduledBatchSupervisorSpec.class, ScheduledBatchSupervisorSpec.TYPE)
new NamedType(ScheduledBatchSupervisorSpec.class, ScheduledBatchSupervisorSpec.TYPE),
new NamedType(CascadingReindexingTemplate.class, CascadingReindexingTemplate.TYPE)
)
);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,14 @@ public CompactionConfigValidationResult validateCompactionTask(
Map<Interval, DataSchema> intervalDataSchemaMap
)
{
// Virtual columns in filter rules are not supported by native compaction
if (compactionTask.getTransformSpec() != null
&& compactionTask.getTransformSpec().getVirtualColumns() != null
&& compactionTask.getTransformSpec().getVirtualColumns().getVirtualColumns().length > 0) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could also consider using isEmpty()

return CompactionConfigValidationResult.failure(
"Virtual columns in filter rules are not supported by the Native compaction engine. Use MSQ compaction engine instead."
);
}
return CompactionConfigValidationResult.success();
}

Expand Down
Loading
Loading