Skip to content

Decoupled planning: add support for unnest#17177

Merged
kgyrtkirk merged 2 commits intoapache:masterfrom
kgyrtkirk:decouple-unnest
Oct 2, 2024
Merged

Decoupled planning: add support for unnest#17177
kgyrtkirk merged 2 commits intoapache:masterfrom
kgyrtkirk:decouple-unnest

Conversation

@kgyrtkirk
Copy link
Copy Markdown
Member

@kgyrtkirk kgyrtkirk commented Sep 27, 2024

  • adds support for UNNEST expressions
  • introduces LogicalUnnestRule to transform a Correlate doing UNNEST into a LogicalUnnest
  • UnnestInputCleanupRule could move the final unnested expr into the LogicalUnnest itself (usually its an mv_to_array expression)
  • enhanced source unwrapping to utilize FilteredDataSource if it looks right

@kgyrtkirk kgyrtkirk marked this pull request as ready for review September 27, 2024 13:39
github-advanced-security[bot]
github-advanced-security AI found potential problems Sep 27, 2024
View reviewed changes
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/DruidUnnestRule.java Outdated
RelTrait out,
String descriptionPrefix)
{
super(clazz, in, out, descriptionPrefix);

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [ConverterRule.ConverterRule](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/LogicalUnnestRule.java
{
public LogicalUnnestRule()
{
super(operand(LogicalCorrelate.class, any()));

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [RelOptRule.operand](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/LogicalUnnestRule.java
{
public LogicalUnnestRule()
{
super(operand(LogicalCorrelate.class, any()));

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [RelOptRule.any](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/UnnestInputCleanupRule.java
Comment on lines +48 to +51
operand(
LogicalUnnest.class,
operand(Project.class, any())
)

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [RelOptRule.operand](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/UnnestInputCleanupRule.java
super(
operand(
LogicalUnnest.class,
operand(Project.class, any())

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [RelOptRule.operand](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/UnnestInputCleanupRule.java
super(
operand(
LogicalUnnest.class,
operand(Project.class, any())

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [RelOptRule.any](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/UnnestInputCleanupRule.java
this.replaceableIndex = replaceableIndex;
}

public RexNode visitInputRef(RexInputRef inputRef)

Check notice

Code scanning / CodeQL

Missing Override annotation

This method overrides [RexShuttle.visitInputRef](1); it is advisable to add an Override annotation.
Comment thread sql/src/test/java/org/apache/druid/sql/calcite/DecoupledExtension.java

QueryTestBuilder builder = new QueryTestBuilder(testConfig)
{
public QueryTestBuilder expectedQueries(List<Query<?>> expectedQueries)

Check notice

Code scanning / CodeQL

Missing Override annotation

This method overrides [QueryTestBuilder.expectedQueries](1); it is advisable to add an Override annotation.
github-advanced-security[bot]
github-advanced-security AI found potential problems Sep 27, 2024
View reviewed changes
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/DruidUnnestRule.java Fixed
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/LogicalUnnestRule.java
{
public LogicalUnnestRule()
{
super(operand(LogicalCorrelate.class, any()));

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [RelOptRule.operand](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/LogicalUnnestRule.java
{
public LogicalUnnestRule()
{
super(operand(LogicalCorrelate.class, any()));

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [RelOptRule.any](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/UnnestInputCleanupRule.java
Comment on lines +48 to +51
operand(
LogicalUnnest.class,
operand(Project.class, any())
)

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [RelOptRule.operand](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/UnnestInputCleanupRule.java
super(
operand(
LogicalUnnest.class,
operand(Project.class, any())

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [RelOptRule.operand](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/UnnestInputCleanupRule.java
super(
operand(
LogicalUnnest.class,
operand(Project.class, any())

Check notice

Code scanning / CodeQL

Deprecated method or constructor invocation

Invoking [RelOptRule.any](1) should be avoided because it has been deprecated.
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/UnnestInputCleanupRule.java Fixed
Comment thread sql/src/test/java/org/apache/druid/sql/calcite/DecoupledExtension.java Fixed
github-advanced-security[bot]
github-advanced-security AI found potential problems Sep 27, 2024
View reviewed changes
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/DruidUnnest.java Fixed
github-advanced-security[bot]
github-advanced-security AI found potential problems Sep 27, 2024
View reviewed changes
Comment thread sql/src/main/java/org/apache/druid/sql/calcite/rule/logical/DruidUnnest.java Outdated
return new SourceDesc(dataSource, outputRowSignature);
}

private DimFilter buildDimFilter(PlannerContext plannerContext, SourceDesc inputDesc, RowSignature filterRowSignature)

Check notice

Code scanning / CodeQL

Useless parameter

The parameter 'inputDesc' is never used.
@kgyrtkirk kgyrtkirk merged commit 65277b1 into apache:master Oct 2, 2024
@kfaraz kfaraz modified the milestone: 31.0.0 Oct 5, 2024
@adarshsanjeev adarshsanjeev added this to the 32.0.0 milestone Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@imply-cheddar imply-cheddar imply-cheddar approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Area - Querying

Projects

None yet

Milestone

32.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@kgyrtkirk @github-advanced-security @imply-cheddar @adarshsanjeev @kfaraz