Skip to content

Conversation

@Jefffrey
Copy link
Contributor

Which issue does this PR close?

Closes #5598

Rationale for this change

With new separate analyzer stage, split off from logical optimizer stage, should have output of these rules in verbose explain for debugging

What changes are included in this PR?

Example output now:

❯ explain verbose select * from part;
+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type                                                  | plan                                                                                                                                                                                                                                                                            |
+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| initial_logical_plan                                       | Projection: part.id, part.bool_col, part.tinyint_col, part.smallint_col, part.int_col, part.bigint_col, part.float_col, part.double_col, part.date_string_col, part.string_col, part.timestamp_col                                                                              |
|                                                            |   TableScan: part                                                                                                                                                                                                                                                               |
| logical_plan after inline_table_scan                       | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after type_coercion                           | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after count_wildcard_rule                     | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| analyzed_logical_plan                                      | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after simplify_expressions                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after unwrap_cast_in_comparison               | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after replace_distinct_aggregate              | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after decorrelate_where_exists                | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after decorrelate_where_in                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after scalar_subquery_to_join                 | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after extract_equijoin_predicate              | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after simplify_expressions                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after merge_projection                        | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after rewrite_disjunctive_predicate           | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_duplicated_expr               | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_filter                        | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_cross_join                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after common_sub_expression_eliminate         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_limit                         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after propagate_empty_relation                | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after filter_null_join_keys                   | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_outer_join                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after push_down_limit                         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after push_down_filter                        | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after single_distinct_aggregation_to_group_by | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after simplify_expressions                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after unwrap_cast_in_comparison               | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after common_sub_expression_eliminate         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after push_down_projection                    | Projection: part.id, part.bool_col, part.tinyint_col, part.smallint_col, part.int_col, part.bigint_col, part.float_col, part.double_col, part.date_string_col, part.string_col, part.timestamp_col                                                                              |
|                                                            |   TableScan: part projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col]                                                                                                                  |
| logical_plan after eliminate_projection                    | TableScan: part projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col]                                                                                                                    |
| logical_plan after push_down_limit                         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after simplify_expressions                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after unwrap_cast_in_comparison               | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after replace_distinct_aggregate              | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after decorrelate_where_exists                | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after decorrelate_where_in                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after scalar_subquery_to_join                 | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after extract_equijoin_predicate              | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after simplify_expressions                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after merge_projection                        | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after rewrite_disjunctive_predicate           | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_duplicated_expr               | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_filter                        | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_cross_join                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after common_sub_expression_eliminate         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_limit                         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after propagate_empty_relation                | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after filter_null_join_keys                   | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_outer_join                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after push_down_limit                         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after push_down_filter                        | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after single_distinct_aggregation_to_group_by | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after simplify_expressions                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after unwrap_cast_in_comparison               | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after common_sub_expression_eliminate         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after push_down_projection                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after eliminate_projection                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan after push_down_limit                         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| logical_plan                                               | TableScan: part projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col]                                                                                                                    |
| initial_physical_plan                                      | ParquetExec: limit=None, partitions={1 group: [[home/jeffrey/Code/arrow-datafusion/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col] |
|                                                            |                                                                                                                                                                                                                                                                                 |
| physical_plan after aggregate_statistics                   | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| physical_plan after join_selection                         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| physical_plan after PipelineFixer                          | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| physical_plan after repartition                            | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| physical_plan after global_sort_selection                  | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| physical_plan after EnforceDistribution                    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| physical_plan after EnforceSorting                         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| physical_plan after CombinePartialFinalAggregate           | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| physical_plan after coalesce_batches                       | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| physical_plan after PipelineChecker                        | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                              |
| physical_plan                                              | ParquetExec: limit=None, partitions={1 group: [[home/jeffrey/Code/arrow-datafusion/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col] |
|                                                            |                                                                                                                                                                                                                                                                                 |
+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Apr 15, 2023
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me -- thank you @Jefffrey

@alamb
Copy link
Contributor

alamb commented Apr 15, 2023

cc @mingmwang

@alamb alamb merged commit 99e39fe into apache:main Apr 16, 2023
@Jefffrey Jefffrey deleted the analyzer_explain branch April 16, 2023 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make analyzed plan visible in the Explain command

3 participants