Skip to content

Make multi-stage-query (MSQ engine) a core capability of Druid rather than an extension#18391

Closed
kfaraz wants to merge 1 commit intoapache:masterfrom
kfaraz:move_msq_to_core
Closed

Make multi-stage-query (MSQ engine) a core capability of Druid rather than an extension#18391
kfaraz wants to merge 1 commit intoapache:masterfrom
kfaraz:move_msq_to_core

Conversation

@kfaraz
Copy link
Copy Markdown
Contributor

@kfaraz kfaraz commented Aug 12, 2025

Description

Druid MSQ engine offers a very convenient method of performing SQL-based batch ingestion, export, etc.

Given the importance and heavy adoption of MSQ, this patch converts MSQ into a core capability of Druid
rather than an extension, as was always intended.

Changes

  • Move all MSQ classes to services module without renaming the package
  • Move the module loader file org.apache.druid.guice.DruidModule to services/src/main/resources/META-INF/services
    • This effectively means that the modules listed in this file will always be loaded on all Druid services as desired
  • Update dependencies as needed
  • Remove mentions of extension from example common.runtime.properties files

Alternative

Instead of moving everything to services, the alternative was to simply move the
multi-stage-query module into a top-level module but that would
lead to cyclic dependencies between services and multi-stage-query since:
a) multi-stage-query depends on services for various classes
b) services is the entry point and must depend on all the core modules that need to be on the classpath

Pending

  • Once initial feedback is received, the docs will also be updated to remove mentions of the multi-stage-query extension.

Future

  • Over time, various MSQ classes may be moved out of services module into more appropriate modules such as
    processing, server, indexing-service or sql as applicable.

Release note

  • Multi-stage-query (MSQ engine) is now a core capability of Druid rather than an extension.
  • Remove druid-multi-stage-query from druid.extensions.loadList in common.runtime.properties.

Upgrade notes

While upgrading, the extension druid-multi-stage-query must be removed from druid.extensions.loadList.
Otherwise, Druid services would fail to start as they would not be able to locate this extension.


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@kfaraz kfaraz requested a review from gianm August 12, 2025 14:53
@kfaraz
Copy link
Copy Markdown
Contributor Author

kfaraz commented Aug 12, 2025

Closing in favor of #18394

@kfaraz kfaraz closed this Aug 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant