Skip to content

Add some new expression functions and macros. #4442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fjy merged 10 commits into from
Jun 28, 2017
Merged

Add some new expression functions and macros. #4442

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
efcc855
Add some new expression functions and macros.
gianm Jun 22, 2017
ae34c4b
Add easymock to datasketches tests.
gianm Jun 22, 2017
a44bfb4
Add easymock to distinctcount tests.
gianm Jun 22, 2017
e58618e
Add easymock to virtual-columns tests.
gianm Jun 25, 2017
a15e5a1
Code review comments.
gianm Jun 26, 2017
35f86f9
Merge branch 'master' into se-new-expr-functions
gianm Jun 26, 2017
f3ad57d
Clean up code a bit.
gianm Jun 26, 2017
d6b1c8a
Add easymock to scan-query tests.
gianm Jun 26, 2017
cf411c4
Rework ExprMacros that have multiple impls.
gianm Jun 28, 2017
588224a
Improve test coverage.
gianm Jun 28, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
264 changes: 259 additions & 5 deletions common/src/main/java/io/druid/math/expr/Function.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

package io.druid.math.expr;

import com.google.common.base.Strings;
import io.druid.java.util.common.IAE;
import org.joda.time.DateTime;
import org.joda.time.format.DateTimeFormat;
Expand All @@ -41,7 +42,7 @@ abstract class SingleParam implements Function
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 1) {
throw new IAE("function '%s' needs 1 argument", name());
throw new IAE("Function[%s] needs 1 argument", name());
}
Expr expr = args.get(0);
return eval(expr.eval(bindings));
Expand All @@ -56,7 +57,7 @@ abstract class DoubleParam implements Function
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 2) {
throw new IAE("function '%s' needs 2 arguments", name());
throw new IAE("Function[%s] needs 2 arguments", name());
}
Expr expr1 = args.get(0);
Expr expr2 = args.get(1);
Expand Down Expand Up @@ -242,6 +243,27 @@ protected ExprEval eval(double param)
}
}

class Div extends DoubleParamMath
{
@Override
public String name()
{
return "div";
}

@Override
protected ExprEval eval(final long x, final long y)
{
return ExprEval.of(x / y);
}

@Override
protected ExprEval eval(final double x, final double y)
{
return ExprEval.of((long) (x / y));
Copy link
Copy Markdown
Member

@leventov leventov Jun 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And why even cast to long? There is ExprEval.of(double).

Copy link
Copy Markdown
Contributor Author

@gianm gianm Jun 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this function div is meant to be integer division.

Copy link
Copy Markdown
Member

@leventov leventov Jun 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't realize that. If so the original (long) x / (long) y was probably better, but I'm not sure. Up to you

Copy link
Copy Markdown
Contributor Author

@gianm gianm Jun 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think (long) (x / y) makes more sense to me. That way the division is done on the numbers in their 'native' form (which might be outside the range of long) and then the result is converted to the nearest integer form.

}
}

class Exp extends SingleParamMath
{
@Override
Expand Down Expand Up @@ -686,14 +708,78 @@ public String name()
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 3) {
throw new IAE("function 'if' needs 3 arguments");
throw new IAE("Function[%s] needs 3 arguments", name());
}

ExprEval x = args.get(0).eval(bindings);
return x.asBoolean() ? args.get(1).eval(bindings) : args.get(2).eval(bindings);
}
}

/**
* "Searched CASE" function, similar to {@code CASE WHEN boolean_expr THEN result [ELSE else_result] END} in SQL.
*/
class CaseSearchedFunc implements Function
{
@Override
public String name()
{
return "case_searched";
}

@Override
public ExprEval apply(final List<Expr> args, final Expr.ObjectBinding bindings)
{
if (args.size() < 2) {
throw new IAE("Function[%s] must have at least 2 arguments", name());
}

for (int i = 0; i < args.size(); i += 2) {
if (i == args.size() - 1) {
// ELSE else_result.
return args.get(i).eval(bindings);
} else if (args.get(i).eval(bindings).asBoolean()) {
// Matching WHEN boolean_expr THEN result
return args.get(i + 1).eval(bindings);
}
}

return ExprEval.of(null);
}
}

/**
* "Simple CASE" function, similar to {@code CASE expr WHEN value THEN result [ELSE else_result] END} in SQL.
*/
class CaseSimpleFunc implements Function
{
@Override
public String name()
{
return "case_simple";
}

@Override
public ExprEval apply(final List<Expr> args, final Expr.ObjectBinding bindings)
{
if (args.size() < 3) {
throw new IAE("Function[%s] must have at least 3 arguments", name());
}

for (int i = 1; i < args.size(); i += 2) {
if (i == args.size() - 1) {
// ELSE else_result.
return args.get(i).eval(bindings);
} else if (new BinEqExpr("==", args.get(0), args.get(i)).eval(bindings).asBoolean()) {
// Matching WHEN value THEN result
return args.get(i + 1).eval(bindings);
}
}

return ExprEval.of(null);
}
}

class CastFunc extends DoubleParam
{
@Override
Expand Down Expand Up @@ -728,7 +814,7 @@ public String name()
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 1 && args.size() != 2) {
throw new IAE("function '%s' needs 1 or 2 arguments", name());
throw new IAE("Function[%s] needs 1 or 2 arguments", name());
}
ExprEval value = args.get(0).eval(bindings);
if (value.type() != ExprType.STRING) {
Expand Down Expand Up @@ -786,10 +872,178 @@ public String name()
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 2) {
throw new IAE("function 'nvl' needs 2 arguments");
throw new IAE("Function[%s] needs 2 arguments", name());
}
final ExprEval eval = args.get(0).eval(bindings);
return eval.isNull() ? args.get(1).eval(bindings) : eval;
}
}

class ConcatFunc implements Function
{
@Override
public String name()
{
return "concat";
}

@Override
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() == 0) {
return ExprEval.of(null);
} else {
// Pass first argument in to the constructor to provide StringBuilder a little extra sizing hint.
final StringBuilder builder = new StringBuilder(Strings.nullToEmpty(args.get(0).eval(bindings).asString()));
for (int i = 1; i < args.size(); i++) {
final String s = args.get(i).eval(bindings).asString();
if (s != null) {
builder.append(s);
}
}
return ExprEval.of(builder.toString());
}
}
}

class StrlenFunc implements Function
{
@Override
public String name()
{
return "strlen";
}

@Override
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 1) {
throw new IAE("Function[%s] needs 1 argument", name());
}

final String arg = args.get(0).eval(bindings).asString();
return arg == null ? ExprEval.of(0) : ExprEval.of(arg.length());
}
}

class SubstringFunc implements Function
{
@Override
public String name()
{
return "substring";
}

@Override
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 3) {
throw new IAE("Function[%s] needs 3 arguments", name());
}

final String arg = args.get(0).eval(bindings).asString();

if (arg == null) {
return ExprEval.of(null);
}

// Behaves like SubstringDimExtractionFn, not SQL SUBSTRING
final int index = args.get(1).eval(bindings).asInt();
final int length = args.get(2).eval(bindings).asInt();

if (index < arg.length()) {
if (length >= 0) {
return ExprEval.of(arg.substring(index, Math.min(index + length, arg.length())));
} else {
return ExprEval.of(arg.substring(index));
}
} else {
return ExprEval.of(null);
}
}
}

class ReplaceFunc implements Function
{
@Override
public String name()
{
return "replace";
}

@Override
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 3) {
throw new IAE("Function[%s] needs 3 arguments", name());
}

final String arg = args.get(0).eval(bindings).asString();
final String pattern = args.get(1).eval(bindings).asString();
final String replacement = args.get(2).eval(bindings).asString();
return ExprEval.of(
Copy link
Copy Markdown
Member

@leventov leventov Jun 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the spirit of other functions should probably return null if the first arg is null

Copy link
Copy Markdown
Contributor Author

@gianm gianm Jun 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExprEval.of will do this. It includes Strings.emptyToNull(value) in its constructor.

Strings.nullToEmpty(arg).replace(Strings.nullToEmpty(pattern), Strings.nullToEmpty(replacement))
);
}
}

class TrimFunc implements Function
{
@Override
public String name()
{
return "trim";
}

@Override
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 1) {
throw new IAE("Function[%s] needs 1 argument", name());
}

final String arg = args.get(0).eval(bindings).asString();
return ExprEval.of(Strings.nullToEmpty(arg).trim());
}
}

class LowerFunc implements Function
{
@Override
public String name()
{
return "lower";
}

@Override
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 1) {
throw new IAE("Function[%s] needs 1 argument", name());
}

final String arg = args.get(0).eval(bindings).asString();
return ExprEval.of(Strings.nullToEmpty(arg).toLowerCase());
}
}

class UpperFunc implements Function
{
@Override
public String name()
{
return "upper";
}

@Override
public ExprEval apply(List<Expr> args, Expr.ObjectBinding bindings)
{
if (args.size() != 1) {
throw new IAE("Function[%s] needs 1 argument", name());
}

final String arg = args.get(0).eval(bindings).asString();
return ExprEval.of(Strings.nullToEmpty(arg).toUpperCase());
}
}
}
Loading