[FEATURE]Check date type to avoid scan all partitions #4756
Conversation
| rewrittenExpr = applyRuleRepeatedly(rewrittenExpr, rule, analyzer); | ||
| } | ||
| } while (oldNumChanges != numChanges_); | ||
| if (expr instanceof BinaryPredicate) { |
There was a problem hiding this comment.
Why not check the type in analysis state?
| if (expr instanceof BinaryPredicate) { | ||
| Expr valueExpr = ((BinaryPredicate) expr).getBinding(); | ||
| if(valueExpr != null && valueExpr.getType() == Type.DATETIME && valueExpr instanceof CastExpr) { | ||
| throw new AnalysisException("invalid date type :" + valueExpr.toSql()); |
There was a problem hiding this comment.
I think throwing a exception is not the best solution of this problem, may be rewrite the binary expr day = '2020-10-32' to something like contant expr BoolLiteral[false] in where statement is better. We should discuss it first.
There was a problem hiding this comment.
I think throwing a exception is not the best solution of this problem, may be rewrite the binary expr
day = '2020-10-32'to something like contant expr BoolLiteral[false] in where statement is better. We should discuss it first.
ok，I will modify it;
| public Expr apply(Expr expr, Analyzer analyzer) throws AnalysisException { | ||
| if (!(expr instanceof BinaryPredicate)) return expr; | ||
| Expr valueExpr = expr.getChild(1); | ||
| if(valueExpr.getType() == Type.DATETIME && valueExpr instanceof CastExpr) { |
There was a problem hiding this comment.
| if(valueExpr.getType() == Type.DATETIME && valueExpr instanceof CastExpr) { | |
| if(valueExpr.getType() == Type.DATETIME && valueExpr instanceof CastExpr && valueExpr.getChild(0).getType() == Type.VARCHAR) { |
There is something wrong with this If statement.
For example, where day = 20201030 will rewrite to where false, which is not right.
You should only support covert StringLiteral now:
where day = '20201032' -> where false
There was a problem hiding this comment.
The rule of FoldConstantsRule will convert where day = 20201030 into where day='2020-10-30 00:00:00', whic is DateLiteral type.
morningman
added
area/planner
#4877) This reverts commit c8df76a. This commit has some problem when handling predicate like: `k1 = "2020-10-10 10:00:00.000"` This is a valid predicate, and FE Datetime can not support milli or micro seconds, so it will treat it as invalid date time value. So we revert it, and may find some better solution later.
3 participants
Proposed changes
select day from test where day='2020-10-32', table 'test' is parititioned by day. In this case, '2020-10-32' will be taken as CastExpr not LiteralExpr, and condition "day='2020-10-32'" will not be recognized as partitionfilter. This case will scan all partitions. To avoid scall all partitions, it is better to filter invalid date value.
issue: #4755
Types of changes
What types of changes does your code introduce to Doris?
Put an
xin the boxes that applyChecklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...