Subsequent `JOIN` s don't match to correct row

[MidasLamb]

Describe the bug

I have a query where I JOIN three tables, and then have a WHERE clause on the last table ("twintag.owner"). If I give a value that exists there, all the results are returned (regardless of whether or not the "twintag.owner" matches the actual value), and if I give a value that doesn't exist, it returns nothing.

The query is as follows:

SELECT "75620d04a8b7b34c70c68f7af68953af".*
FROM "75620d04a8b7b34c70c68f7af68953af" 
JOIN "f4be58656cd5e54b9bffa1706c3c49ff" AS "linkedtwintag"
    ON "75620d04a8b7b34c70c68f7af68953af"."82ad7656c6b6be56e9abfac772c89b4c" = "linkedtwintag"."776ff32c7d7bfb1c1a49d5c0d94e0db8" 
JOIN "__bag" AS "linkedtwintag_twintag"
    ON "linkedtwintag"."dataScope" = "linkedtwintag_twintag"."storage_qid" 
WHERE 1 = 1 
AND "linkedtwintag_twintag"."owner" = '324ae4533db15dcb19e3473fea54177a' 
ORDER BY "75620d04a8b7b34c70c68f7af68953af"."qid" ASC

If I start from the intermediate table it works as expected and I get back the one result I'm looking for:

SELECT "f4be58656cd5e54b9bffa1706c3c49ff".*
FROM "f4be58656cd5e54b9bffa1706c3c49ff" 
JOIN "__bag" AS "twintag"
    ON "f4be58656cd5e54b9bffa1706c3c49ff"."dataScope" = "twintag"."storage_qid"
WHERE 1 = 1 
AND "twintag"."owner" = '324ae4533db15dcb19e3473fea54177a'  ORDER BY "f4be58656cd5e54b9bffa1706c3c49ff"."qid" ASC

To Reproduce

Create some tables where you can construct a JOIN from table A to B and from B to C.
Execute the (type of) query, where you use the intermediate table to join from table A to C, filtering on something in table C:

SELECT "75620d04a8b7b34c70c68f7af68953af".*
FROM "75620d04a8b7b34c70c68f7af68953af" 
JOIN "f4be58656cd5e54b9bffa1706c3c49ff" AS "linkedtwintag"
    ON "75620d04a8b7b34c70c68f7af68953af"."82ad7656c6b6be56e9abfac772c89b4c" = "linkedtwintag"."776ff32c7d7bfb1c1a49d5c0d94e0db8" 
JOIN "__bag" AS "linkedtwintag_twintag"
    ON "linkedtwintag"."dataScope" = "linkedtwintag_twintag"."storage_qid" 
WHERE 1 = 1 
AND "linkedtwintag_twintag"."owner" = '324ae4533db15dcb19e3473fea54177a' 
ORDER BY "75620d04a8b7b34c70c68f7af68953af"."qid" ASC

Expected behavior

I expect only the items which match the filter on table C to be returned.
Currently it returns either ALL items if an item matches, or NO items if there is no match found. In the example above I expect 1 match, but I get all the items from table A back.
If I change the value I'm looking for in table C to be non-existent there, I get back no results instead, which is expected.

Additional context

I'm also using datafusion-remote-table from seafowl (https://github.com/splitgraph/seafowl/tree/main/datafusion_remote_tables)

[alamb]

Thank you for the report @MidasLamb -- is it possible to share a reproducer (aka the data referred to above)? This seems like it may be data specific

[MidasLamb]

@alamb , I've created a reproduction repo here: https://github.com/MidasLamb/datafusion-multiple-join-bug-example

If you search for "b93" in the output that is shown, you'll see in the first test that when the WHERE clause is added, there suddenly are a lot more results which show up out of nowhere compared to the first show() without a WHERE clause

[MidasLamb]

It seems to be a bug in a default optimizer, because adding this:

    let ctx = SessionContext::with_state(
        state
            .with_optimizer_rules(vec![])
    );

gives back the results that I expect

[Dandandan]

Awesome - @MidasLamb . If possible, could you identify:

• Which optimizer rule is at fault?
• What is the change in logical / physical plan (EXPLAIN VERBOSE)?

[MidasLamb]

@Dandandan , I'm trying to find the default optimizer rules so I can start weeding them out, but I can't find them immediately, so that's why I already posted that little update

[Dandandan]

No worries, this is the entire list:

        let rules: Vec<Arc<dyn OptimizerRule + Sync + Send>> = vec![
            Arc::new(SimplifyExpressions::new()),
            Arc::new(UnwrapCastInComparison::new()),
            Arc::new(ReplaceDistinctWithAggregate::new()),
            Arc::new(EliminateJoin::new()),
            Arc::new(DecorrelatePredicateSubquery::new()),
            Arc::new(ScalarSubqueryToJoin::new()),
            Arc::new(ExtractEquijoinPredicate::new()),
            // simplify expressions does not simplify expressions in subqueries, so we
            // run it again after running the optimizations that potentially converted
            // subqueries to joins
            Arc::new(SimplifyExpressions::new()),
            Arc::new(MergeProjection::new()),
            Arc::new(RewriteDisjunctivePredicate::new()),
            Arc::new(EliminateDuplicatedExpr::new()),
            Arc::new(EliminateFilter::new()),
            Arc::new(EliminateCrossJoin::new()),
            Arc::new(CommonSubexprEliminate::new()),
            Arc::new(EliminateLimit::new()),
            Arc::new(PropagateEmptyRelation::new()),
            Arc::new(FilterNullJoinKeys::default()),
            Arc::new(EliminateOuterJoin::new()),
            // Filters can't be pushed down past Limits, we should do PushDownFilter after PushDownLimit
            Arc::new(PushDownLimit::new()),
            Arc::new(PushDownFilter::new()),
            Arc::new(SingleDistinctToGroupBy::new()),
            // The previous optimizations added expressions and projections,
            // that might benefit from the following rules
            Arc::new(SimplifyExpressions::new()),
            Arc::new(UnwrapCastInComparison::new()),
            Arc::new(CommonSubexprEliminate::new()),
            Arc::new(PushDownProjection::new()),
            Arc::new(EliminateProjection::new()),
            // PushDownProjection can pushdown Projections through Limits, do PushDownLimit again.
            Arc::new(PushDownLimit::new()),
        ];

[MidasLamb]

Seems to be a caused by these two rules, commenting one of them makes it works as expected:

state.with_optimizer_rules(vec![
            // Arc::new(SimplifyExpressions::new()),
            // Arc::new(UnwrapCastInComparison::new()),
            // Arc::new(ReplaceDistinctWithAggregate::new()),
            // Arc::new(EliminateJoin::new()),
            // Arc::new(DecorrelatePredicateSubquery::new()),
            // Arc::new(ScalarSubqueryToJoin::new()),
            Arc::new(ExtractEquijoinPredicate::new()),
            // Arc::new(SimplifyExpressions::new()),
            // Arc::new(MergeProjection::new()),
            // Arc::new(RewriteDisjunctivePredicate::new()),
            // Arc::new(EliminateDuplicatedExpr::new()),
            // Arc::new(EliminateFilter::new()),
            Arc::new(EliminateCrossJoin::new()),
            // Arc::new(CommonSubexprEliminate::new()),
            // Arc::new(EliminateLimit::new()),
            //
            // Arc::new(PropagateEmptyRelation::new()),
            // Arc::new(FilterNullJoinKeys::default()),
            // Arc::new(EliminateOuterJoin::new()),
            // Arc::new(PushDownLimit::new()),
            // Arc::new(PushDownFilter::new()),
            // Arc::new(SingleDistinctToGroupBy::new()),
            // Arc::new(SimplifyExpressions::new()),
            // Arc::new(UnwrapCastInComparison::new()),
            // Arc::new(CommonSubexprEliminate::new()),
            // Arc::new(PushDownProjection::new()),
            // Arc::new(EliminateProjection::new()),
            // Arc::new(PushDownLimit::new()),
        ]),

Individual plans

Plan without bug:

+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type                                        | plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| initial_logical_plan                             | Projection: service_article.qid, service_article.timestamp, service_article.createdDate, service_article.82ad7656c6b6be56e9abfac772c89b4c, metadata.dataScope, metadata.qid, metadata.timestamp, metadata.createdDate, metadata.776ff32c7d7bfb1c1a49d5c0d94e0db8, bag.owner, bag.status, bag.created_date, bag.edition, bag.subedition, bag.parent, bag.time_limit, bag.file_count, bag.file_count_limit, bag.capacity, bag.capacity_limit, bag.bandwidth, bag.bandwidth_incoming, bag.bandwidth_outgoing, bag.bandwidth_limit, bag.client_ip, bag.deletion_time, bag.credits, bag.suspended, bag.suspendedowner, bag.flagged, bag.blessed, bag.cleared, bag.added, bag.saved, bag.shared, bag.skipped, bag.celebrated, bag.scan_count, bag.bandwidth_all_incoming, bag.bandwidth_all_outgoing, bag.resourcelimit_email_flag, bag.edition_activated_date, bag.storage_qid, bag.customizationflags                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |   Filter: bag.storage_qid = Utf8("b936b63f271c21b4a00957b325cb9a22")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |     Inner Join:  Filter: bag.storage_qid = metadata.dataScope                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |       Inner Join:  Filter: metadata.776ff32c7d7bfb1c1a49d5c0d94e0db8 = service_article.82ad7656c6b6be56e9abfac772c89b4c                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|                                                  |         TableScan: service_article                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                                  |         TableScan: metadata                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                                                  |       TableScan: bag                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| logical_plan after inline_table_scan             | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| logical_plan after type_coercion                 | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| logical_plan after count_wildcard_rule           | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| analyzed_logical_plan                            | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| logical_plan after eliminate_cross_join          | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| logical_plan                                     | Projection: service_article.qid, service_article.timestamp, service_article.createdDate, service_article.82ad7656c6b6be56e9abfac772c89b4c, metadata.dataScope, metadata.qid, metadata.timestamp, metadata.createdDate, metadata.776ff32c7d7bfb1c1a49d5c0d94e0db8, bag.owner, bag.status, bag.created_date, bag.edition, bag.subedition, bag.parent, bag.time_limit, bag.file_count, bag.file_count_limit, bag.capacity, bag.capacity_limit, bag.bandwidth, bag.bandwidth_incoming, bag.bandwidth_outgoing, bag.bandwidth_limit, bag.client_ip, bag.deletion_time, bag.credits, bag.suspended, bag.suspendedowner, bag.flagged, bag.blessed, bag.cleared, bag.added, bag.saved, bag.shared, bag.skipped, bag.celebrated, bag.scan_count, bag.bandwidth_all_incoming, bag.bandwidth_all_outgoing, bag.resourcelimit_email_flag, bag.edition_activated_date, bag.storage_qid, bag.customizationflags                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |   Filter: bag.storage_qid = Utf8("b936b63f271c21b4a00957b325cb9a22")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |     Inner Join:  Filter: bag.storage_qid = metadata.dataScope                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |       Inner Join:  Filter: metadata.776ff32c7d7bfb1c1a49d5c0d94e0db8 = service_article.82ad7656c6b6be56e9abfac772c89b4c                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|                                                  |         TableScan: service_article                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                                  |         TableScan: metadata                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                                                  |       TableScan: bag                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| initial_physical_plan                            | ProjectionExec: expr=[qid@0 as qid, timestamp@1 as timestamp, createdDate@2 as createdDate, 82ad7656c6b6be56e9abfac772c89b4c@3 as 82ad7656c6b6be56e9abfac772c89b4c, dataScope@4 as dataScope, qid@5 as qid, timestamp@6 as timestamp, createdDate@7 as createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8@8 as 776ff32c7d7bfb1c1a49d5c0d94e0db8, owner@9 as owner, status@10 as status, created_date@11 as created_date, edition@12 as edition, subedition@13 as subedition, parent@14 as parent, time_limit@15 as time_limit, file_count@16 as file_count, file_count_limit@17 as file_count_limit, capacity@18 as capacity, capacity_limit@19 as capacity_limit, bandwidth@20 as bandwidth, bandwidth_incoming@21 as bandwidth_incoming, bandwidth_outgoing@22 as bandwidth_outgoing, bandwidth_limit@23 as bandwidth_limit, client_ip@24 as client_ip, deletion_time@25 as deletion_time, credits@26 as credits, suspended@27 as suspended, suspendedowner@28 as suspendedowner, flagged@29 as flagged, blessed@30 as blessed, cleared@31 as cleared, added@32 as added, saved@33 as saved, shared@34 as shared, skipped@35 as skipped, celebrated@36 as celebrated, scan_count@37 as scan_count, bandwidth_all_incoming@38 as bandwidth_all_incoming, bandwidth_all_outgoing@39 as bandwidth_all_outgoing, resourcelimit_email_flag@40 as resourcelimit_email_flag, edition_activated_date@41 as edition_activated_date, storage_qid@42 as storage_qid, customizationflags@43 as customizationflags] |
|                                                  |   FilterExec: storage_qid@42 = b936b63f271c21b4a00957b325cb9a22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                                  |     NestedLoopJoinExec: join_type=Inner, filter=BinaryExpr { left: Column { name: "storage_qid", index: 1 }, op: Eq, right: Column { name: "dataScope", index: 0 } }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |       NestedLoopJoinExec: join_type=Inner, filter=BinaryExpr { left: Column { name: "776ff32c7d7bfb1c1a49d5c0d94e0db8", index: 1 }, op: Eq, right: Column { name: "82ad7656c6b6be56e9abfac772c89b4c", index: 0 } }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                                  |         ParquetExec: file_groups={1 group: [[home/midas/test/fus/service_article.parquet/part-0.parquet]]}, projection=[qid, timestamp, createdDate, 82ad7656c6b6be56e9abfac772c89b4c]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
|                                                  |         ParquetExec: file_groups={1 group: [[home/midas/test/fus/metadata.parquet/part-0.parquet]]}, projection=[dataScope, qid, timestamp, createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                                                  |       ParquetExec: file_groups={1 group: [[home/midas/test/fus/bag.parquet/part-0.parquet]]}, projection=[owner, status, created_date, edition, subedition, parent, time_limit, file_count, file_count_limit, capacity, capacity_limit, bandwidth, bandwidth_incoming, bandwidth_outgoing, bandwidth_limit, client_ip, deletion_time, credits, suspended, suspendedowner, flagged, blessed, cleared, added, saved, shared, skipped, celebrated, scan_count, bandwidth_all_incoming, bandwidth_all_outgoing, resourcelimit_email_flag, edition_activated_date, storage_qid, customizationflags]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                                                  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| physical_plan after aggregate_statistics         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after join_selection               | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after PipelineFixer                | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after repartition                  | ProjectionExec: expr=[qid@0 as qid, timestamp@1 as timestamp, createdDate@2 as createdDate, 82ad7656c6b6be56e9abfac772c89b4c@3 as 82ad7656c6b6be56e9abfac772c89b4c, dataScope@4 as dataScope, qid@5 as qid, timestamp@6 as timestamp, createdDate@7 as createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8@8 as 776ff32c7d7bfb1c1a49d5c0d94e0db8, owner@9 as owner, status@10 as status, created_date@11 as created_date, edition@12 as edition, subedition@13 as subedition, parent@14 as parent, time_limit@15 as time_limit, file_count@16 as file_count, file_count_limit@17 as file_count_limit, capacity@18 as capacity, capacity_limit@19 as capacity_limit, bandwidth@20 as bandwidth, bandwidth_incoming@21 as bandwidth_incoming, bandwidth_outgoing@22 as bandwidth_outgoing, bandwidth_limit@23 as bandwidth_limit, client_ip@24 as client_ip, deletion_time@25 as deletion_time, credits@26 as credits, suspended@27 as suspended, suspendedowner@28 as suspendedowner, flagged@29 as flagged, blessed@30 as blessed, cleared@31 as cleared, added@32 as added, saved@33 as saved, shared@34 as shared, skipped@35 as skipped, celebrated@36 as celebrated, scan_count@37 as scan_count, bandwidth_all_incoming@38 as bandwidth_all_incoming, bandwidth_all_outgoing@39 as bandwidth_all_outgoing, resourcelimit_email_flag@40 as resourcelimit_email_flag, edition_activated_date@41 as edition_activated_date, storage_qid@42 as storage_qid, customizationflags@43 as customizationflags] |
|                                                  |   FilterExec: storage_qid@42 = b936b63f271c21b4a00957b325cb9a22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                                  |     RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
|                                                  |       NestedLoopJoinExec: join_type=Inner, filter=BinaryExpr { left: Column { name: "storage_qid", index: 1 }, op: Eq, right: Column { name: "dataScope", index: 0 } }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
|                                                  |         NestedLoopJoinExec: join_type=Inner, filter=BinaryExpr { left: Column { name: "776ff32c7d7bfb1c1a49d5c0d94e0db8", index: 1 }, op: Eq, right: Column { name: "82ad7656c6b6be56e9abfac772c89b4c", index: 0 } }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |           ParquetExec: file_groups={1 group: [[home/midas/test/fus/service_article.parquet/part-0.parquet]]}, projection=[qid, timestamp, createdDate, 82ad7656c6b6be56e9abfac772c89b4c]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|                                                  |           ParquetExec: file_groups={1 group: [[home/midas/test/fus/metadata.parquet/part-0.parquet]]}, projection=[dataScope, qid, timestamp, createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|                                                  |         ParquetExec: file_groups={1 group: [[home/midas/test/fus/bag.parquet/part-0.parquet]]}, projection=[owner, status, created_date, edition, subedition, parent, time_limit, file_count, file_count_limit, capacity, capacity_limit, bandwidth, bandwidth_incoming, bandwidth_outgoing, bandwidth_limit, client_ip, deletion_time, credits, suspended, suspendedowner, flagged, blessed, cleared, added, saved, shared, skipped, celebrated, scan_count, bandwidth_all_incoming, bandwidth_all_outgoing, resourcelimit_email_flag, edition_activated_date, storage_qid, customizationflags]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|                                                  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| physical_plan after global_sort_selection        | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after EnforceDistribution          | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after CombinePartialFinalAggregate | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after EnforceSorting               | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after coalesce_batches             | ProjectionExec: expr=[qid@0 as qid, timestamp@1 as timestamp, createdDate@2 as createdDate, 82ad7656c6b6be56e9abfac772c89b4c@3 as 82ad7656c6b6be56e9abfac772c89b4c, dataScope@4 as dataScope, qid@5 as qid, timestamp@6 as timestamp, createdDate@7 as createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8@8 as 776ff32c7d7bfb1c1a49d5c0d94e0db8, owner@9 as owner, status@10 as status, created_date@11 as created_date, edition@12 as edition, subedition@13 as subedition, parent@14 as parent, time_limit@15 as time_limit, file_count@16 as file_count, file_count_limit@17 as file_count_limit, capacity@18 as capacity, capacity_limit@19 as capacity_limit, bandwidth@20 as bandwidth, bandwidth_incoming@21 as bandwidth_incoming, bandwidth_outgoing@22 as bandwidth_outgoing, bandwidth_limit@23 as bandwidth_limit, client_ip@24 as client_ip, deletion_time@25 as deletion_time, credits@26 as credits, suspended@27 as suspended, suspendedowner@28 as suspendedowner, flagged@29 as flagged, blessed@30 as blessed, cleared@31 as cleared, added@32 as added, saved@33 as saved, shared@34 as shared, skipped@35 as skipped, celebrated@36 as celebrated, scan_count@37 as scan_count, bandwidth_all_incoming@38 as bandwidth_all_incoming, bandwidth_all_outgoing@39 as bandwidth_all_outgoing, resourcelimit_email_flag@40 as resourcelimit_email_flag, edition_activated_date@41 as edition_activated_date, storage_qid@42 as storage_qid, customizationflags@43 as customizationflags] |
|                                                  |   CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |     FilterExec: storage_qid@42 = b936b63f271c21b4a00957b325cb9a22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |       RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                                                  |         NestedLoopJoinExec: join_type=Inner, filter=BinaryExpr { left: Column { name: "storage_qid", index: 1 }, op: Eq, right: Column { name: "dataScope", index: 0 } }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|                                                  |           NestedLoopJoinExec: join_type=Inner, filter=BinaryExpr { left: Column { name: "776ff32c7d7bfb1c1a49d5c0d94e0db8", index: 1 }, op: Eq, right: Column { name: "82ad7656c6b6be56e9abfac772c89b4c", index: 0 } }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
|                                                  |             ParquetExec: file_groups={1 group: [[home/midas/test/fus/service_article.parquet/part-0.parquet]]}, projection=[qid, timestamp, createdDate, 82ad7656c6b6be56e9abfac772c89b4c]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                                                  |             ParquetExec: file_groups={1 group: [[home/midas/test/fus/metadata.parquet/part-0.parquet]]}, projection=[dataScope, qid, timestamp, createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                                                  |           ParquetExec: file_groups={1 group: [[home/midas/test/fus/bag.parquet/part-0.parquet]]}, projection=[owner, status, created_date, edition, subedition, parent, time_limit, file_count, file_count_limit, capacity, capacity_limit, bandwidth, bandwidth_incoming, bandwidth_outgoing, bandwidth_limit, client_ip, deletion_time, credits, suspended, suspendedowner, flagged, blessed, cleared, added, saved, shared, skipped, celebrated, scan_count, bandwidth_all_incoming, bandwidth_all_outgoing, resourcelimit_email_flag, edition_activated_date, storage_qid, customizationflags]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                                  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| physical_plan after PipelineChecker              | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan                                    | ProjectionExec: expr=[qid@0 as qid, timestamp@1 as timestamp, createdDate@2 as createdDate, 82ad7656c6b6be56e9abfac772c89b4c@3 as 82ad7656c6b6be56e9abfac772c89b4c, dataScope@4 as dataScope, qid@5 as qid, timestamp@6 as timestamp, createdDate@7 as createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8@8 as 776ff32c7d7bfb1c1a49d5c0d94e0db8, owner@9 as owner, status@10 as status, created_date@11 as created_date, edition@12 as edition, subedition@13 as subedition, parent@14 as parent, time_limit@15 as time_limit, file_count@16 as file_count, file_count_limit@17 as file_count_limit, capacity@18 as capacity, capacity_limit@19 as capacity_limit, bandwidth@20 as bandwidth, bandwidth_incoming@21 as bandwidth_incoming, bandwidth_outgoing@22 as bandwidth_outgoing, bandwidth_limit@23 as bandwidth_limit, client_ip@24 as client_ip, deletion_time@25 as deletion_time, credits@26 as credits, suspended@27 as suspended, suspendedowner@28 as suspendedowner, flagged@29 as flagged, blessed@30 as blessed, cleared@31 as cleared, added@32 as added, saved@33 as saved, shared@34 as shared, skipped@35 as skipped, celebrated@36 as celebrated, scan_count@37 as scan_count, bandwidth_all_incoming@38 as bandwidth_all_incoming, bandwidth_all_outgoing@39 as bandwidth_all_outgoing, resourcelimit_email_flag@40 as resourcelimit_email_flag, edition_activated_date@41 as edition_activated_date, storage_qid@42 as storage_qid, customizationflags@43 as customizationflags] |
|                                                  |   CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |     FilterExec: storage_qid@42 = b936b63f271c21b4a00957b325cb9a22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |       RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                                                  |         NestedLoopJoinExec: join_type=Inner, filter=BinaryExpr { left: Column { name: "storage_qid", index: 1 }, op: Eq, right: Column { name: "dataScope", index: 0 } }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|                                                  |           NestedLoopJoinExec: join_type=Inner, filter=BinaryExpr { left: Column { name: "776ff32c7d7bfb1c1a49d5c0d94e0db8", index: 1 }, op: Eq, right: Column { name: "82ad7656c6b6be56e9abfac772c89b4c", index: 0 } }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
|                                                  |             ParquetExec: file_groups={1 group: [[home/midas/test/fus/service_article.parquet/part-0.parquet]]}, projection=[qid, timestamp, createdDate, 82ad7656c6b6be56e9abfac772c89b4c]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                                                  |             ParquetExec: file_groups={1 group: [[home/midas/test/fus/metadata.parquet/part-0.parquet]]}, projection=[dataScope, qid, timestamp, createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                                                  |           ParquetExec: file_groups={1 group: [[home/midas/test/fus/bag.parquet/part-0.parquet]]}, projection=[owner, status, created_date, edition, subedition, parent, time_limit, file_count, file_count_limit, capacity, capacity_limit, bandwidth, bandwidth_incoming, bandwidth_outgoing, bandwidth_limit, client_ip, deletion_time, credits, suspended, suspendedowner, flagged, blessed, cleared, added, saved, shared, skipped, celebrated, scan_count, bandwidth_all_incoming, bandwidth_all_outgoing, resourcelimit_email_flag, edition_activated_date, storage_qid, customizationflags]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                                  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Plan with bug:

+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type                                        | plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| initial_logical_plan                             | Projection: service_article.qid, service_article.timestamp, service_article.createdDate, service_article.82ad7656c6b6be56e9abfac772c89b4c, metadata.dataScope, metadata.qid, metadata.timestamp, metadata.createdDate, metadata.776ff32c7d7bfb1c1a49d5c0d94e0db8, bag.owner, bag.status, bag.created_date, bag.edition, bag.subedition, bag.parent, bag.time_limit, bag.file_count, bag.file_count_limit, bag.capacity, bag.capacity_limit, bag.bandwidth, bag.bandwidth_incoming, bag.bandwidth_outgoing, bag.bandwidth_limit, bag.client_ip, bag.deletion_time, bag.credits, bag.suspended, bag.suspendedowner, bag.flagged, bag.blessed, bag.cleared, bag.added, bag.saved, bag.shared, bag.skipped, bag.celebrated, bag.scan_count, bag.bandwidth_all_incoming, bag.bandwidth_all_outgoing, bag.resourcelimit_email_flag, bag.edition_activated_date, bag.storage_qid, bag.customizationflags                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |   Filter: bag.storage_qid = Utf8("b936b63f271c21b4a00957b325cb9a22")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |     Inner Join:  Filter: bag.storage_qid = metadata.dataScope                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |       Inner Join:  Filter: metadata.776ff32c7d7bfb1c1a49d5c0d94e0db8 = service_article.82ad7656c6b6be56e9abfac772c89b4c                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|                                                  |         TableScan: service_article                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                                  |         TableScan: metadata                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                                                  |       TableScan: bag                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| logical_plan after inline_table_scan             | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| logical_plan after type_coercion                 | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| logical_plan after count_wildcard_rule           | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| analyzed_logical_plan                            | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| logical_plan after extract_equijoin_predicate    | Projection: service_article.qid, service_article.timestamp, service_article.createdDate, service_article.82ad7656c6b6be56e9abfac772c89b4c, metadata.dataScope, metadata.qid, metadata.timestamp, metadata.createdDate, metadata.776ff32c7d7bfb1c1a49d5c0d94e0db8, bag.owner, bag.status, bag.created_date, bag.edition, bag.subedition, bag.parent, bag.time_limit, bag.file_count, bag.file_count_limit, bag.capacity, bag.capacity_limit, bag.bandwidth, bag.bandwidth_incoming, bag.bandwidth_outgoing, bag.bandwidth_limit, bag.client_ip, bag.deletion_time, bag.credits, bag.suspended, bag.suspendedowner, bag.flagged, bag.blessed, bag.cleared, bag.added, bag.saved, bag.shared, bag.skipped, bag.celebrated, bag.scan_count, bag.bandwidth_all_incoming, bag.bandwidth_all_outgoing, bag.resourcelimit_email_flag, bag.edition_activated_date, bag.storage_qid, bag.customizationflags                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |   Filter: bag.storage_qid = Utf8("b936b63f271c21b4a00957b325cb9a22")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |     Inner Join: metadata.dataScope = bag.storage_qid                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |       Inner Join:  Filter: metadata.776ff32c7d7bfb1c1a49d5c0d94e0db8 = service_article.82ad7656c6b6be56e9abfac772c89b4c                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|                                                  |         TableScan: service_article                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                                  |         TableScan: metadata                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                                                  |       TableScan: bag                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| logical_plan after eliminate_cross_join          | Projection: service_article.qid, service_article.timestamp, service_article.createdDate, service_article.82ad7656c6b6be56e9abfac772c89b4c, metadata.dataScope, metadata.qid, metadata.timestamp, metadata.createdDate, metadata.776ff32c7d7bfb1c1a49d5c0d94e0db8, bag.owner, bag.status, bag.created_date, bag.edition, bag.subedition, bag.parent, bag.time_limit, bag.file_count, bag.file_count_limit, bag.capacity, bag.capacity_limit, bag.bandwidth, bag.bandwidth_incoming, bag.bandwidth_outgoing, bag.bandwidth_limit, bag.client_ip, bag.deletion_time, bag.credits, bag.suspended, bag.suspendedowner, bag.flagged, bag.blessed, bag.cleared, bag.added, bag.saved, bag.shared, bag.skipped, bag.celebrated, bag.scan_count, bag.bandwidth_all_incoming, bag.bandwidth_all_outgoing, bag.resourcelimit_email_flag, bag.edition_activated_date, bag.storage_qid, bag.customizationflags                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |   Filter: bag.storage_qid = Utf8("b936b63f271c21b4a00957b325cb9a22")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |     Inner Join: metadata.dataScope = bag.storage_qid                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |       CrossJoin:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|                                                  |         TableScan: service_article                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                                  |         TableScan: metadata                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                                                  |       TableScan: bag                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| logical_plan after extract_equijoin_predicate    | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| logical_plan after eliminate_cross_join          | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| logical_plan                                     | Projection: service_article.qid, service_article.timestamp, service_article.createdDate, service_article.82ad7656c6b6be56e9abfac772c89b4c, metadata.dataScope, metadata.qid, metadata.timestamp, metadata.createdDate, metadata.776ff32c7d7bfb1c1a49d5c0d94e0db8, bag.owner, bag.status, bag.created_date, bag.edition, bag.subedition, bag.parent, bag.time_limit, bag.file_count, bag.file_count_limit, bag.capacity, bag.capacity_limit, bag.bandwidth, bag.bandwidth_incoming, bag.bandwidth_outgoing, bag.bandwidth_limit, bag.client_ip, bag.deletion_time, bag.credits, bag.suspended, bag.suspendedowner, bag.flagged, bag.blessed, bag.cleared, bag.added, bag.saved, bag.shared, bag.skipped, bag.celebrated, bag.scan_count, bag.bandwidth_all_incoming, bag.bandwidth_all_outgoing, bag.resourcelimit_email_flag, bag.edition_activated_date, bag.storage_qid, bag.customizationflags                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |   Filter: bag.storage_qid = Utf8("b936b63f271c21b4a00957b325cb9a22")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |     Inner Join: metadata.dataScope = bag.storage_qid                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |       CrossJoin:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|                                                  |         TableScan: service_article                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                                                  |         TableScan: metadata                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                                                  |       TableScan: bag                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| initial_physical_plan                            | ProjectionExec: expr=[qid@0 as qid, timestamp@1 as timestamp, createdDate@2 as createdDate, 82ad7656c6b6be56e9abfac772c89b4c@3 as 82ad7656c6b6be56e9abfac772c89b4c, dataScope@4 as dataScope, qid@5 as qid, timestamp@6 as timestamp, createdDate@7 as createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8@8 as 776ff32c7d7bfb1c1a49d5c0d94e0db8, owner@9 as owner, status@10 as status, created_date@11 as created_date, edition@12 as edition, subedition@13 as subedition, parent@14 as parent, time_limit@15 as time_limit, file_count@16 as file_count, file_count_limit@17 as file_count_limit, capacity@18 as capacity, capacity_limit@19 as capacity_limit, bandwidth@20 as bandwidth, bandwidth_incoming@21 as bandwidth_incoming, bandwidth_outgoing@22 as bandwidth_outgoing, bandwidth_limit@23 as bandwidth_limit, client_ip@24 as client_ip, deletion_time@25 as deletion_time, credits@26 as credits, suspended@27 as suspended, suspendedowner@28 as suspendedowner, flagged@29 as flagged, blessed@30 as blessed, cleared@31 as cleared, added@32 as added, saved@33 as saved, shared@34 as shared, skipped@35 as skipped, celebrated@36 as celebrated, scan_count@37 as scan_count, bandwidth_all_incoming@38 as bandwidth_all_incoming, bandwidth_all_outgoing@39 as bandwidth_all_outgoing, resourcelimit_email_flag@40 as resourcelimit_email_flag, edition_activated_date@41 as edition_activated_date, storage_qid@42 as storage_qid, customizationflags@43 as customizationflags] |
|                                                  |   FilterExec: storage_qid@42 = b936b63f271c21b4a00957b325cb9a22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                                  |     HashJoinExec: mode=Partitioned, join_type=Inner, on=[(Column { name: "dataScope", index: 4 }, Column { name: "storage_qid", index: 33 })]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |       CrossJoinExec                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
|                                                  |         ParquetExec: file_groups={1 group: [[home/midas/test/fus/service_article.parquet/part-0.parquet]]}, projection=[qid, timestamp, createdDate, 82ad7656c6b6be56e9abfac772c89b4c]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
|                                                  |         ParquetExec: file_groups={1 group: [[home/midas/test/fus/metadata.parquet/part-0.parquet]]}, projection=[dataScope, qid, timestamp, createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                                                  |       ParquetExec: file_groups={1 group: [[home/midas/test/fus/bag.parquet/part-0.parquet]]}, projection=[owner, status, created_date, edition, subedition, parent, time_limit, file_count, file_count_limit, capacity, capacity_limit, bandwidth, bandwidth_incoming, bandwidth_outgoing, bandwidth_limit, client_ip, deletion_time, credits, suspended, suspendedowner, flagged, blessed, cleared, added, saved, shared, skipped, celebrated, scan_count, bandwidth_all_incoming, bandwidth_all_outgoing, resourcelimit_email_flag, edition_activated_date, storage_qid, customizationflags]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                                                  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| physical_plan after aggregate_statistics         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after join_selection               | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after PipelineFixer                | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after repartition                  | ProjectionExec: expr=[qid@0 as qid, timestamp@1 as timestamp, createdDate@2 as createdDate, 82ad7656c6b6be56e9abfac772c89b4c@3 as 82ad7656c6b6be56e9abfac772c89b4c, dataScope@4 as dataScope, qid@5 as qid, timestamp@6 as timestamp, createdDate@7 as createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8@8 as 776ff32c7d7bfb1c1a49d5c0d94e0db8, owner@9 as owner, status@10 as status, created_date@11 as created_date, edition@12 as edition, subedition@13 as subedition, parent@14 as parent, time_limit@15 as time_limit, file_count@16 as file_count, file_count_limit@17 as file_count_limit, capacity@18 as capacity, capacity_limit@19 as capacity_limit, bandwidth@20 as bandwidth, bandwidth_incoming@21 as bandwidth_incoming, bandwidth_outgoing@22 as bandwidth_outgoing, bandwidth_limit@23 as bandwidth_limit, client_ip@24 as client_ip, deletion_time@25 as deletion_time, credits@26 as credits, suspended@27 as suspended, suspendedowner@28 as suspendedowner, flagged@29 as flagged, blessed@30 as blessed, cleared@31 as cleared, added@32 as added, saved@33 as saved, shared@34 as shared, skipped@35 as skipped, celebrated@36 as celebrated, scan_count@37 as scan_count, bandwidth_all_incoming@38 as bandwidth_all_incoming, bandwidth_all_outgoing@39 as bandwidth_all_outgoing, resourcelimit_email_flag@40 as resourcelimit_email_flag, edition_activated_date@41 as edition_activated_date, storage_qid@42 as storage_qid, customizationflags@43 as customizationflags] |
|                                                  |   FilterExec: storage_qid@42 = b936b63f271c21b4a00957b325cb9a22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                                  |     HashJoinExec: mode=Partitioned, join_type=Inner, on=[(Column { name: "dataScope", index: 4 }, Column { name: "storage_qid", index: 33 })]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |       RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                                                  |         CrossJoinExec                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|                                                  |           ParquetExec: file_groups={1 group: [[home/midas/test/fus/service_article.parquet/part-0.parquet]]}, projection=[qid, timestamp, createdDate, 82ad7656c6b6be56e9abfac772c89b4c]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|                                                  |           ParquetExec: file_groups={1 group: [[home/midas/test/fus/metadata.parquet/part-0.parquet]]}, projection=[dataScope, qid, timestamp, createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|                                                  |       ParquetExec: file_groups={1 group: [[home/midas/test/fus/bag.parquet/part-0.parquet]]}, projection=[owner, status, created_date, edition, subedition, parent, time_limit, file_count, file_count_limit, capacity, capacity_limit, bandwidth, bandwidth_incoming, bandwidth_outgoing, bandwidth_limit, client_ip, deletion_time, credits, suspended, suspendedowner, flagged, blessed, cleared, added, saved, shared, skipped, celebrated, scan_count, bandwidth_all_incoming, bandwidth_all_outgoing, resourcelimit_email_flag, edition_activated_date, storage_qid, customizationflags]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                                                  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| physical_plan after global_sort_selection        | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after EnforceDistribution          | ProjectionExec: expr=[qid@0 as qid, timestamp@1 as timestamp, createdDate@2 as createdDate, 82ad7656c6b6be56e9abfac772c89b4c@3 as 82ad7656c6b6be56e9abfac772c89b4c, dataScope@4 as dataScope, qid@5 as qid, timestamp@6 as timestamp, createdDate@7 as createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8@8 as 776ff32c7d7bfb1c1a49d5c0d94e0db8, owner@9 as owner, status@10 as status, created_date@11 as created_date, edition@12 as edition, subedition@13 as subedition, parent@14 as parent, time_limit@15 as time_limit, file_count@16 as file_count, file_count_limit@17 as file_count_limit, capacity@18 as capacity, capacity_limit@19 as capacity_limit, bandwidth@20 as bandwidth, bandwidth_incoming@21 as bandwidth_incoming, bandwidth_outgoing@22 as bandwidth_outgoing, bandwidth_limit@23 as bandwidth_limit, client_ip@24 as client_ip, deletion_time@25 as deletion_time, credits@26 as credits, suspended@27 as suspended, suspendedowner@28 as suspendedowner, flagged@29 as flagged, blessed@30 as blessed, cleared@31 as cleared, added@32 as added, saved@33 as saved, shared@34 as shared, skipped@35 as skipped, celebrated@36 as celebrated, scan_count@37 as scan_count, bandwidth_all_incoming@38 as bandwidth_all_incoming, bandwidth_all_outgoing@39 as bandwidth_all_outgoing, resourcelimit_email_flag@40 as resourcelimit_email_flag, edition_activated_date@41 as edition_activated_date, storage_qid@42 as storage_qid, customizationflags@43 as customizationflags] |
|                                                  |   FilterExec: storage_qid@42 = b936b63f271c21b4a00957b325cb9a22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|                                                  |     HashJoinExec: mode=Partitioned, join_type=Inner, on=[(Column { name: "dataScope", index: 4 }, Column { name: "storage_qid", index: 33 })]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |       RepartitionExec: partitioning=Hash([Column { name: "dataScope", index: 4 }], 16), input_partitions=16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                                                  |         RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |           CrossJoinExec                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|                                                  |             ParquetExec: file_groups={1 group: [[home/midas/test/fus/service_article.parquet/part-0.parquet]]}, projection=[qid, timestamp, createdDate, 82ad7656c6b6be56e9abfac772c89b4c]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|                                                  |             ParquetExec: file_groups={1 group: [[home/midas/test/fus/metadata.parquet/part-0.parquet]]}, projection=[dataScope, qid, timestamp, createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                                                  |       RepartitionExec: partitioning=Hash([Column { name: "storage_qid", index: 33 }], 16), input_partitions=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |         ParquetExec: file_groups={1 group: [[home/midas/test/fus/bag.parquet/part-0.parquet]]}, projection=[owner, status, created_date, edition, subedition, parent, time_limit, file_count, file_count_limit, capacity, capacity_limit, bandwidth, bandwidth_incoming, bandwidth_outgoing, bandwidth_limit, client_ip, deletion_time, credits, suspended, suspendedowner, flagged, blessed, cleared, added, saved, shared, skipped, celebrated, scan_count, bandwidth_all_incoming, bandwidth_all_outgoing, resourcelimit_email_flag, edition_activated_date, storage_qid, customizationflags]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|                                                  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| physical_plan after CombinePartialFinalAggregate | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after EnforceSorting               | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan after coalesce_batches             | ProjectionExec: expr=[qid@0 as qid, timestamp@1 as timestamp, createdDate@2 as createdDate, 82ad7656c6b6be56e9abfac772c89b4c@3 as 82ad7656c6b6be56e9abfac772c89b4c, dataScope@4 as dataScope, qid@5 as qid, timestamp@6 as timestamp, createdDate@7 as createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8@8 as 776ff32c7d7bfb1c1a49d5c0d94e0db8, owner@9 as owner, status@10 as status, created_date@11 as created_date, edition@12 as edition, subedition@13 as subedition, parent@14 as parent, time_limit@15 as time_limit, file_count@16 as file_count, file_count_limit@17 as file_count_limit, capacity@18 as capacity, capacity_limit@19 as capacity_limit, bandwidth@20 as bandwidth, bandwidth_incoming@21 as bandwidth_incoming, bandwidth_outgoing@22 as bandwidth_outgoing, bandwidth_limit@23 as bandwidth_limit, client_ip@24 as client_ip, deletion_time@25 as deletion_time, credits@26 as credits, suspended@27 as suspended, suspendedowner@28 as suspendedowner, flagged@29 as flagged, blessed@30 as blessed, cleared@31 as cleared, added@32 as added, saved@33 as saved, shared@34 as shared, skipped@35 as skipped, celebrated@36 as celebrated, scan_count@37 as scan_count, bandwidth_all_incoming@38 as bandwidth_all_incoming, bandwidth_all_outgoing@39 as bandwidth_all_outgoing, resourcelimit_email_flag@40 as resourcelimit_email_flag, edition_activated_date@41 as edition_activated_date, storage_qid@42 as storage_qid, customizationflags@43 as customizationflags] |
|                                                  |   CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |     FilterExec: storage_qid@42 = b936b63f271c21b4a00957b325cb9a22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |       CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |         HashJoinExec: mode=Partitioned, join_type=Inner, on=[(Column { name: "dataScope", index: 4 }, Column { name: "storage_qid", index: 33 })]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |           CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|                                                  |             RepartitionExec: partitioning=Hash([Column { name: "dataScope", index: 4 }], 16), input_partitions=16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |               RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
|                                                  |                 CrossJoinExec                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |                   ParquetExec: file_groups={1 group: [[home/midas/test/fus/service_article.parquet/part-0.parquet]]}, projection=[qid, timestamp, createdDate, 82ad7656c6b6be56e9abfac772c89b4c]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|                                                  |                   ParquetExec: file_groups={1 group: [[home/midas/test/fus/metadata.parquet/part-0.parquet]]}, projection=[dataScope, qid, timestamp, createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |           CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|                                                  |             RepartitionExec: partitioning=Hash([Column { name: "storage_qid", index: 33 }], 16), input_partitions=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
|                                                  |               ParquetExec: file_groups={1 group: [[home/midas/test/fus/bag.parquet/part-0.parquet]]}, projection=[owner, status, created_date, edition, subedition, parent, time_limit, file_count, file_count_limit, capacity, capacity_limit, bandwidth, bandwidth_incoming, bandwidth_outgoing, bandwidth_limit, client_ip, deletion_time, credits, suspended, suspendedowner, flagged, blessed, cleared, added, saved, shared, skipped, celebrated, scan_count, bandwidth_all_incoming, bandwidth_all_outgoing, resourcelimit_email_flag, edition_activated_date, storage_qid, customizationflags]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
|                                                  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| physical_plan after PipelineChecker              | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| physical_plan                                    | ProjectionExec: expr=[qid@0 as qid, timestamp@1 as timestamp, createdDate@2 as createdDate, 82ad7656c6b6be56e9abfac772c89b4c@3 as 82ad7656c6b6be56e9abfac772c89b4c, dataScope@4 as dataScope, qid@5 as qid, timestamp@6 as timestamp, createdDate@7 as createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8@8 as 776ff32c7d7bfb1c1a49d5c0d94e0db8, owner@9 as owner, status@10 as status, created_date@11 as created_date, edition@12 as edition, subedition@13 as subedition, parent@14 as parent, time_limit@15 as time_limit, file_count@16 as file_count, file_count_limit@17 as file_count_limit, capacity@18 as capacity, capacity_limit@19 as capacity_limit, bandwidth@20 as bandwidth, bandwidth_incoming@21 as bandwidth_incoming, bandwidth_outgoing@22 as bandwidth_outgoing, bandwidth_limit@23 as bandwidth_limit, client_ip@24 as client_ip, deletion_time@25 as deletion_time, credits@26 as credits, suspended@27 as suspended, suspendedowner@28 as suspendedowner, flagged@29 as flagged, blessed@30 as blessed, cleared@31 as cleared, added@32 as added, saved@33 as saved, shared@34 as shared, skipped@35 as skipped, celebrated@36 as celebrated, scan_count@37 as scan_count, bandwidth_all_incoming@38 as bandwidth_all_incoming, bandwidth_all_outgoing@39 as bandwidth_all_outgoing, resourcelimit_email_flag@40 as resourcelimit_email_flag, edition_activated_date@41 as edition_activated_date, storage_qid@42 as storage_qid, customizationflags@43 as customizationflags] |
|                                                  |   CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |     FilterExec: storage_qid@42 = b936b63f271c21b4a00957b325cb9a22                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |       CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |         HashJoinExec: mode=Partitioned, join_type=Inner, on=[(Column { name: "dataScope", index: 4 }, Column { name: "storage_qid", index: 33 })]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |           CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|                                                  |             RepartitionExec: partitioning=Hash([Column { name: "dataScope", index: 4 }], 16), input_partitions=16                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                                                  |               RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
|                                                  |                 CrossJoinExec                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                                                  |                   ParquetExec: file_groups={1 group: [[home/midas/test/fus/service_article.parquet/part-0.parquet]]}, projection=[qid, timestamp, createdDate, 82ad7656c6b6be56e9abfac772c89b4c]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|                                                  |                   ParquetExec: file_groups={1 group: [[home/midas/test/fus/metadata.parquet/part-0.parquet]]}, projection=[dataScope, qid, timestamp, createdDate, 776ff32c7d7bfb1c1a49d5c0d94e0db8]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|                                                  |           CoalesceBatchesExec: target_batch_size=8192                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|                                                  |             RepartitionExec: partitioning=Hash([Column { name: "storage_qid", index: 33 }], 16), input_partitions=1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
|                                                  |               ParquetExec: file_groups={1 group: [[home/midas/test/fus/bag.parquet/part-0.parquet]]}, projection=[owner, status, created_date, edition, subedition, parent, time_limit, file_count, file_count_limit, capacity, capacity_limit, bandwidth, bandwidth_incoming, bandwidth_outgoing, bandwidth_limit, client_ip, deletion_time, credits, suspended, suspendedowner, flagged, blessed, cleared, added, saved, shared, skipped, celebrated, scan_count, bandwidth_all_incoming, bandwidth_all_outgoing, resourcelimit_email_flag, edition_activated_date, storage_qid, customizationflags]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
|                                                  |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+--------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[MidasLamb]

It seems to replace an Inner Join with a Cross Join for some reason, which (in my particular case) isn't actually correct

[MidasLamb]

I've found this piece of comment/code in the EliminateCrossJoin optimizer:

// The filter of inner join will lost, skip this rule.
// issue: https://github.com/apache/arrow-datafusion/issues/4844
if join.filter.is_some() {
    return Ok(None);
}

However in my scenario the filtered join is not the top level, so that checks seems to get skipped.

I've set up a test for this by doing this:

        let t1 = test_table_scan_with_name("t1")?;
        let t2 = test_table_scan_with_name("t2")?;
        let t3 = test_table_scan_with_name("t3")?;
        let plan = LogicalPlanBuilder::from(t1)
            .join(t2, JoinType::Inner, (Vec::<Column>::new(), Vec::<Column>::new()), Some(col("t1.a").eq(col("t2.a"))))?
            .join(
                t3,
                JoinType::Inner,
                (
                    vec![Column::from_qualified_name("t2.a")],
                    vec![Column::from_qualified_name("t3.a")],
                ),
                None,
            )?
            .filter(col("t2.c").lt(lit(20u32)))?
            .build()?;