-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Reindexing rule providers with cascading interval based reindexing #18939
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
capistrant
merged 96 commits into
apache:master
from
capistrant:reindexing-rule-providers-with-cascading-reindexing
Feb 20, 2026
Merged
Changes from all commits
Commits
Show all changes
96 commits
Select commit
Hold shift + click to select a range
29c7b93
rough cascading compaction impl
capistrant ae21c82
Renaming refactor
capistrant 71830fc
add support for more config knobs in the cascading reindex spec
capistrant 1940267
add some testing
capistrant bdb514e
stop using forbidden apis
capistrant 0f71da2
working on some test coverage
capistrant acad478
simplify search interval creation and enhance embedded test for casca…
capistrant d49fbf4
fixup checkstyle
capistrant 5efb4bf
temporary fixup to test
capistrant 5a87759
fix checkstyle
capistrant 0850a8f
remove native runner for one compaction supervisor test due to native…
capistrant 3667456
refactorings from self review
capistrant a280690
Fixup naming to prefer reindexing over compaction
capistrant 3108b61
fix up a javadoc with up to date design spec
capistrant 1f671e1
Fill in UT gaps for the composing provider
capistrant 3be0da1
refactor test class for inline rule provider
capistrant 79ff44b
Self review refactorings
capistrant 420f3b2
Trying to transform cascadingreindexingtemplate to a compaction state…
capistrant bf2e02d
refactor the location of the reindexing filter rule optimizer
capistrant 5b4f3d2
Refactor this idea of additivity and how it works for building configs
capistrant 6f5ead7
Add a missing test class
capistrant 6d1fc6e
fix checkstyle
capistrant 799db27
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant 6853b02
clean up a javadoc
capistrant 2200467
trivial fixes
capistrant a0d68eb
Prevent an edge case for a negative period
capistrant bbb5bbd
fix a nasty bug opportunity
capistrant 4e2dcb0
Add support to filter on nested columns for MSQ reindexing using virt…
capistrant 77fea05
fix forbidden misses
capistrant a39c7d3
Wait or data availability before querying more than segment counts
capistrant 816c7f9
Try to clear up confusion by renaming period to olderThan to signify …
capistrant 960353e
Rename filter to deleteWhere in ReindexingFilterRule to make it clear…
capistrant f275a98
Rename the ReindexingFilterRule to ReindexingDeletionRule to clarify …
capistrant 62cc519
Make projections rule not be additive
capistrant d706539
fix a test now that projections rules are not additive
capistrant 5c3b5c5
Add two new skip offset concepts to cascading reindexing template + i…
capistrant 0fbbeb2
Split out segment and query granularity rules
capistrant 0ea46b7
Refactor the non overlapping timeline build for cascading reindexing
capistrant 4ad3a90
Add a forced segment gran field to supervisor and some tests
capistrant a97fd18
Remove unused method
capistrant b58f026
fix import
capistrant f8d3be8
Fixing imports
capistrant 49bd2fc
fix a test
capistrant c3197ba
cleanup
capistrant 89763bc
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant 457d912
fix imports after merging master
capistrant b44882b
fixup test file
capistrant 846ff7c
test updates
capistrant ce3e1f6
minor
capistrant d988908
Replace Selector with Equals and In with TypedIn
capistrant 3f75762
Add strict enforcement of segment granularity changes
capistrant cbacb9f
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant 2293e86
Fix bug introduced that was breaking MVD in compaction
capistrant 91d4a5c
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant 75c4329
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant 7030054
Add ReindexingDataSchema rule that will replace other rule types
capistrant 6f1bd02
Tear out rules that are now consolidated into data schema rules
capistrant f9d020f
checkpointed api for getting a rule timeline
capistrant ebf40cf
naming refactor
capistrant d510cb4
Refactors and UTs
capistrant 091a0d3
bug fixes for timeline
capistrant 3dd6069
Support dynamic discovery of skip intervals when skipOffsetFromLatest…
capistrant ceafdc7
Cleanup some ugly date parsing in supervisor timeline view
capistrant eb26cd1
fix a rendering bug for the UI component
capistrant 66d6815
web console testing and fixes found while writing tests
capistrant bbb05b9
clean refactor of reindexing config builder
capistrant a39f222
clean up refactor of cascading reindexing template
capistrant 6c842a7
self review cleanup
capistrant 33fc482
add missing tests for supervisor resource api
capistrant ccd6c71
Refactor exception class for timeline to be more descriptive
capistrant c7ed4e1
remove redundant naming patterns
capistrant f6ef9a9
Add some java docs
capistrant 83766fa
refactor how the final config customization works for cascading reind…
capistrant f07b97a
fix serde issues I was having
capistrant 707904f
fixup random bits after CI ran
capistrant a24cd32
waiting for segments to be available should avoid issues with segs th…
capistrant bfff8de
Move ReindexingConfigBuilder to a hopefully better place
capistrant 5fa4146
Merge branch 'master' into reindexing-rule-providers-with-cascading-r…
capistrant 3f2747c
Remove web-console changes from this PR
capistrant 0d0d0c5
Remove support for the reindexing timeline creation from app code
capistrant 4b211a0
missed a few spots
capistrant 387eb5a
fix silly jackson serde error
capistrant 34d31ab
addressing review comments
capistrant 30d2d77
Create strict allow list for segment granularity options
capistrant a381179
add some compaction transform serde tests out of paranoia
capistrant e560a97
All P0D older than to essentially make all data from referance and ol…
capistrant 408208c
fix test that did not get updated when CompactionTransformSpec update…
capistrant 16b25af
Fix improper hashing in the data schema rule
capistrant 216d984
move rule tests to junit5
capistrant 619a0d1
migrate the rule provider test files to junit5
capistrant b55f7d8
Add junit5 deps to indexing-service and migrate new test files to junit5
capistrant 889d876
Add vintage engine so junit4 and 5 can coexist in indexing-service
capistrant 8a14e85
Clean up ReindexingRule invalid input handling. Throw DruidException
capistrant d3e8917
Fix the transform spec equality check in compactionstatus
capistrant ede3596
Fixup checkstyle
capistrant 54a4b6a
fix regression now that our check on transform spec is more involved
capistrant File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
400 changes: 338 additions & 62 deletions
400
...sts/src/test/java/org/apache/druid/testing/embedded/compact/CompactionSupervisorTest.java
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could also consider using
isEmpty()