Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the DruidRexExecutor w.r.t handling of numeric arrays #11968

Merged
merged 4 commits into from
Nov 23, 2021

Conversation

LakshSingla
Copy link
Contributor

@LakshSingla LakshSingla commented Nov 21, 2021

Description

Issue:
DruidRexExecutor while reducing Arrays, specially numeric arrays, doesn't convert the value from ExprResult's type to BigDecimal, which causes makeLiteral to cast the values. Also, if NaN or Infinite values are present in the array, the error is a generic NumberFormatException. For example:

SELECT ARRAY[1.11, 2.22] returns [1, 2]
SELECT SQRT(-1) throws a generic NumberFormatException instead of IAE

This PR introduces change to cast the numeric values to BigDecimal since Calcite's library understands that easily, and doesn't perform casts.


Key changed/added classes in this PR
  • DruidRexExecutor#reduce()

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

Copy link
Member

@clintropolis clintropolis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LakshSingla
Copy link
Contributor Author

Added the testcases, and fixed the older incorrect ones which were known.

Copy link
Contributor

@abhishekagarwal87 abhishekagarwal87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@abhishekagarwal87 abhishekagarwal87 merged commit b5a25f2 into apache:master Nov 23, 2021
.map(BigDecimal::valueOf)
.collect(Collectors.toList());
literal = rexBuilder.makeLiteral(resultAsBigDecimalList, constExp.getType(), true);
} else {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be better if you can add a check for double type here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An else case is required to stop the "Variable might not have been initialized" exception from cropping up, and I didn't want to provide a failsafe else block. In case when it's not an array (https://github.com/LakshSingla/druid/blob/4dcdb2d68563302a9b4d387cdbb220ec4f3a59b3/sql/src/main/java/org/apache/druid/sql/calcite/planner/DruidRexExecutor.java#L130), it was using the else case as well, that's why I omitted the else if.
One approach can be to initialize the literal to null and then including your suggestion. Let me know if that seems better, and I can add that in a separate commit.

@abhishekagarwal87 abhishekagarwal87 added this to the 0.23.0 milestone May 11, 2022
@clintropolis clintropolis mentioned this pull request Apr 12, 2023
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants