Skip to content

Remove null and empty fields from native queries#12634

Merged
gianm merged 6 commits intoapache:masterfrom
paul-rogers:220610-json
Jun 16, 2022
Merged

Remove null and empty fields from native queries#12634
gianm merged 6 commits intoapache:masterfrom
paul-rogers:220610-json

Conversation

@paul-rogers
Copy link
Copy Markdown
Contributor

This is a subset of the changes for the planner test framework, split out to make reviews easier.

Druid native queries are often serialized to JSON. In the planner framework, we save the JSON as an "expected plan." To make the resulting files a bit smaller, this PR removes from the JSON those fields with default values: values that Jackson would infer even if the fields are not present. For example, rather than spelling out "foo": null, we can simply omit "foo" and get the same results. The same is true for "foo": {} and "foo": [] since most constructors will create an empty collection if passed a null.

The clean-up is done using the @JsonInclude annotation, sometimes with a custom filter class. Scan Query has a batch size which defaults to 4096 * 5 if the provided value is 0. Given this, added a filter which omits the value if it is the default, which avoids cluttering the deserialized JSON with magic numbers.

Some unit tests check for an exact serialization of the JSON, including the now-removed null fields. All such tests were updated with the revised format.

This change is purely cosmetic: the deserialized query objects do not change meaning. Tested via unit tests, including the new planner test framework.


This PR has:

  • been self-reviewed.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

@paul-rogers paul-rogers mentioned this pull request Jun 10, 2022
5 tasks
@lgtm-com
Copy link
Copy Markdown

lgtm-com Bot commented Jun 10, 2022

This pull request introduces 3 alerts when merging 1c70cf1 into 1ace733 - view on LGTM.com

new alerts:

  • 3 for Inconsistent equals and hashCode

@lgtm-com
Copy link
Copy Markdown

lgtm-com Bot commented Jun 12, 2022

This pull request introduces 3 alerts when merging e9a623c into 1ace733 - view on LGTM.com

new alerts:

  • 3 for Inconsistent equals and hashCode

@lgtm-com
Copy link
Copy Markdown

lgtm-com Bot commented Jun 13, 2022

This pull request introduces 3 alerts when merging 922087b into 1ace733 - view on LGTM.com

new alerts:

  • 3 for Inconsistent equals and hashCode

Comment thread processing/src/main/java/org/apache/druid/query/filter/SelectorDimFilter.java Outdated
Comment thread processing/src/main/java/org/apache/druid/query/groupby/GroupByQuery.java Outdated
@lgtm-com
Copy link
Copy Markdown

lgtm-com Bot commented Jun 14, 2022

This pull request introduces 3 alerts when merging a93bf0d into 27e8b43 - view on LGTM.com

new alerts:

  • 3 for Inconsistent equals and hashCode

@lgtm-com
Copy link
Copy Markdown

lgtm-com Bot commented Jun 14, 2022

This pull request introduces 3 alerts when merging ec67530 into 1f6e888 - view on LGTM.com

new alerts:

  • 3 for Inconsistent equals and hashCode

@lgtm-com
Copy link
Copy Markdown

lgtm-com Bot commented Jun 15, 2022

This pull request introduces 3 alerts when merging 2969d46 into 45e3111 - view on LGTM.com

new alerts:

  • 3 for Inconsistent equals and hashCode

Copy link
Copy Markdown
Contributor

@gianm gianm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gianm gianm merged commit 893759d into apache:master Jun 16, 2022
@abhishekagarwal87 abhishekagarwal87 added this to the 24.0.0 milestone Aug 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants