Skip to content

Allow Dart to query realtime tasks#18076

Merged
gianm merged 9 commits intoapache:masterfrom
adarshsanjeev:dart-realtime
Jun 6, 2025
Merged

Allow Dart to query realtime tasks#18076
gianm merged 9 commits intoapache:masterfrom
adarshsanjeev:dart-realtime

Conversation

@adarshsanjeev
Copy link
Copy Markdown
Contributor

@adarshsanjeev adarshsanjeev commented Jun 4, 2025

Add the ability for Dart to query real-time tasks.

Most of the changes are around including the realtime segments while slicing the inputs in the controller. There are also some small changes around handling worker counts in the case of no historical segments.

Additionally, cloneQueryMode is now respected by Dart.

Release Notes

  • MSQ Dart is now able to query real-time tasks by setting the query context parameter includeSegmentSource to realtime, in a similar way to MSQ tasks. Dart queries real-time tasks by default.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@github-actions github-actions Bot added Area - Batch Ingestion Area - Querying Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Jun 4, 2025
@adarshsanjeev adarshsanjeev marked this pull request as ready for review June 5, 2025 13:22
Copy link
Copy Markdown
Contributor

@gianm gianm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looking good. I don't see an IT - did you test it to make sure it works?

Btw, I don't think we have Dart integration tests yet, so I won't ask you to add an IT in this patch, but we should address that in the future.

@adarshsanjeev
Copy link
Copy Markdown
Contributor Author

Generally looking good. I don't see an IT - did you test it to make sure it works?

Yes, I have tested it locally, with only real-time segments as well as a mix.

I too noticed that there isn't a Dart IT. I will add one in another PR.

Comment thread docs/querying/query-context.md Outdated
.stream()
.filter(druidServerMetadata -> includeSegmentSource.getUsedServerTypes()
.contains(druidServerMetadata.getType()))
.filter(druidServerMetadata -> SegmentSource.REALTIME.getUsedServerTypes()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To mirror the suggestion for IndexerControllerContext, consider moving this to a constant in DartControllerContext.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed! This one instance should probably remain as REALTIME, not a default value. This part of the code is common to both and should only be called if realtime segments are queried.

@gianm gianm merged commit e8e548f into apache:master Jun 6, 2025
74 checks passed
@capistrant capistrant added this to the 34.0.0 milestone Jul 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area - Batch Ingestion Area - Documentation Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 Area - Querying

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants