Skip to content

Improve the DruidRexExecutor w.r.t handling of numeric arrays#11968

Merged
abhishekagarwal87 merged 4 commits intoapache:masterfrom
LakshSingla:double-array-cast
Nov 23, 2021
Merged

Improve the DruidRexExecutor w.r.t handling of numeric arrays#11968
abhishekagarwal87 merged 4 commits intoapache:masterfrom
LakshSingla:double-array-cast

Conversation

@LakshSingla
Copy link
Copy Markdown
Contributor

@LakshSingla LakshSingla commented Nov 21, 2021

Description

Issue:
DruidRexExecutor while reducing Arrays, specially numeric arrays, doesn't convert the value from ExprResult's type to BigDecimal, which causes makeLiteral to cast the values. Also, if NaN or Infinite values are present in the array, the error is a generic NumberFormatException. For example:

SELECT ARRAY[1.11, 2.22] returns [1, 2]
SELECT SQRT(-1) throws a generic NumberFormatException instead of IAE

This PR introduces change to cast the numeric values to BigDecimal since Calcite's library understands that easily, and doesn't perform casts.


Key changed/added classes in this PR
  • DruidRexExecutor#reduce()

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

Copy link
Copy Markdown
Member

@clintropolis clintropolis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LakshSingla
Copy link
Copy Markdown
Contributor Author

Added the testcases, and fixed the older incorrect ones which were known.

Copy link
Copy Markdown
Contributor

@abhishekagarwal87 abhishekagarwal87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@abhishekagarwal87 abhishekagarwal87 merged commit b5a25f2 into apache:master Nov 23, 2021
.map(BigDecimal::valueOf)
.collect(Collectors.toList());
literal = rexBuilder.makeLiteral(resultAsBigDecimalList, constExp.getType(), true);
} else {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be better if you can add a check for double type here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An else case is required to stop the "Variable might not have been initialized" exception from cropping up, and I didn't want to provide a failsafe else block. In case when it's not an array (https://github.com/LakshSingla/druid/blob/4dcdb2d68563302a9b4d387cdbb220ec4f3a59b3/sql/src/main/java/org/apache/druid/sql/calcite/planner/DruidRexExecutor.java#L130), it was using the else case as well, that's why I omitted the else if.
One approach can be to initialize the literal to null and then including your suggestion. Let me know if that seems better, and I can add that in a separate commit.

@abhishekagarwal87 abhishekagarwal87 added this to the 0.23.0 milestone May 11, 2022
@clintropolis clintropolis mentioned this pull request Apr 12, 2023
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants