Add configurable final stages to MSQ ingestion queries#16699
Merged
cryptoe merged 19 commits intoapache:masterfrom Aug 27, 2024
Merged
Add configurable final stages to MSQ ingestion queries#16699cryptoe merged 19 commits intoapache:masterfrom
cryptoe merged 19 commits intoapache:masterfrom
Conversation
github-actions
Bot
added
Area - Batch Ingestion
Area - Querying
Area - MSQ
10 tasks
cryptoe
reviewed
Jul 9, 2024
cryptoe
reviewed
Jul 15, 2024
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| * Creates a {@link TerminalStageSpec} which determines the final of a query. Currently, always returns a segment | ||
| * generation spec, but this can be used to configure a wide range of behaviours. | ||
| */ | ||
| public TerminalStageSpec createTerminalStageSpec(DruidQuery druidQuery, PlannerContext plannerContext) |
Check notice
Code scanning / CodeQL
Useless parameter
| * Creates a {@link TerminalStageSpec} which determines the final of a query. Currently, always returns a segment | ||
| * generation spec, but this can be used to configure a wide range of behaviours. | ||
| */ | ||
| public TerminalStageSpec createTerminalStageSpec(DruidQuery druidQuery, PlannerContext plannerContext) |
Check notice
Code scanning / CodeQL
Useless parameter
adarshsanjeev
changed the title
cryptoe
reviewed
Jul 22, 2024
| DataSourceMSQDestination destination = (DataSourceMSQDestination) querySpec.getDestination(); | ||
| TerminalStageSpec terminalStageSpec = destination.getTerminalStageSpec(); | ||
| if (terminalStageSpec instanceof SegmentGenerationStageSpec) { | ||
| return ((SegmentGenerationStageSpec) terminalStageSpec).constructFinalStage(queryId, queryDef, querySpec, jsonMapper); |
Contributor
There was a problem hiding this comment.
can this be a interface method and we pass query def only ?
Contributor
There was a problem hiding this comment.
We can inject the jsonMapper.
Contributor
Author
There was a problem hiding this comment.
We would also need the querySpec for the tuningConfig. Passing the jsonMapper here would also require SegmentGenerationStageSpec to not be a singleton class, and instead require a jsonMapper to be injected.
hevansDev
pushed a commit
to hevansDev/druid
that referenced
this pull request
Aug 29, 2024
* Add a segmentMorphFactory to MSQ. * Add test * Make argument nullable * Fix Guice issues * Merge with master * Remove extra information * Fix tests * Create a utils class * Refactor segment generation * Fix javadoc * Refactor * Refactor * Fix injection
10 tasks
edgar2020
pushed a commit
to edgar2020/druid
that referenced
this pull request
Sep 5, 2024
* Add a segmentMorphFactory to MSQ. * Add test * Make argument nullable * Fix Guice issues * Merge with master * Remove extra information * Fix tests * Create a utils class * Refactor segment generation * Fix javadoc * Refactor * Refactor * Fix injection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Modifies the
DataSourceMSQDestinationto have aTerminalStageSpec. This spec tells the controller how the segments are created. This allows queries to configure how segments are created after the queries have finished running.This will allow queries to add new
TerminalStageSpecwhich allow queries to have other results instead of generating segments.The PR also adds a
SegmentGenerationStageSpecto be used by default, which causes queries to create new segments with the query results.The SegmentGenerationTerminalStageSpec for ingestion is serialized as:This PR should not have a functional impact, and is meant to be used for other features in the future.
Backward Compatibility
The changes are backward compatible. The
DataSourceMSQDestinationcreated has a default value which uses theSegmentGenerationStageSpecby default, since normally all ingestions would require a segment generation stage. If indexers/MM are upgraded first, the stage spec automatically deserializes to segment generation.This PR has: