DRILL-8190: Fix mongo project pushdown for queries with joins #2652
Conversation
jnturton
added
the
backport-to-stable
|
I don't have the expertise to spot any problems here so LGTM and I'll just ask some general questions here and there. |
| @Override | ||
| public RelOptCost computeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) { | ||
| return super.computeSelfCost(planner, mq).multiplyBy(0.1); | ||
| return super.computeLogicalAggCost(planner, mq).multiplyBy(0.1); |
There was a problem hiding this comment.
Are these Plugin*Rel RelNodes used when operations are pushed down to storage plugins?
There was a problem hiding this comment.
Yes, they represent part of the pushed-down query, so it is possible to find out the most optimal query for both Drill and actual storage where the query is pushed down.
| public @Nullable RelOptCost computeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) { | ||
| double rowCount = estimateRowCount(mq); | ||
| double columnCount = Utilities.isStarQuery(getRowType()) ? STAR_COLUMN_COST : getRowType().getFieldCount(); | ||
| double valueCount = rowCount * columnCount; |
There was a problem hiding this comment.
Was it only queries with a pushed down join that were affected by there being no cost saving for adding a plugin projection?
There was a problem hiding this comment.
I think more queries were affected, but probably it was simpler to reproduce it on queries with joins.
| RelNode intermediatePrel = new PluginIntermediatePrel( | ||
| in.getCluster(), | ||
| in.getTraitSet().replace(outTrait), | ||
| in.getTraitSet().replace(outTrait).plus(DrillDistributionTrait.SINGLETON), |
There was a problem hiding this comment.
What is the effect of adding this SINGLETON distribution trait on computed plans?
There was a problem hiding this comment.
In this case, Drill wouldn't add extra exchange operators, since this part of the plan is executed in a single drillbit.
| import java.util.List; | ||
| import java.util.stream.Collectors; | ||
|
|
||
| public class DynamicTypeResolverBuilder extends StdTypeResolverBuilder { |
There was a problem hiding this comment.
Did this new serde builder become necessary because of the changes made in this PR for pushing down projections to Mongo?
There was a problem hiding this comment.
Yes, fixes for Mongo (in the java-exec module) helped to find out that serde for enumerable plugins is broken, so fixed it here.
|
Thanks for the explanations +1. |
2 participants
DRILL-8190: Fix mongo project pushdown for queries with joins
Description
Improved costs calculation for vertex drel to consider resulting columns number in its cost to make plans with the pushed project more preferable.
Added
SINGLETONto rel nodes for which execution cannot be distributed to several drillbits.Fixed deserialization of
EnumerableSubScan.Documentation
NA
Testing
Added unit test, existing tests pass.