Allow user defined ExecutionPlans to benefit from pushdown behavior

[alamb]

Is your feature request related to a problem or challenge?

#8073 adds support for projection pushdown in the physical optimizer.

However, its implementation directly checks for and special cases the built in ExecutionPlan nodes in DataFusion (which other optimizer passes do as well)

For example

        } else if let Some(output_req) = input.downcast_ref::<OutputRequirementExec>() {
            try_swapping_with_output_req(projection, output_req)?
        } else if input.is::<CoalescePartitionsExec>() {
            try_swapping_with_coalesce_partitions(projection)?
        } else if let Some(filter) = input.downcast_ref::<FilterExec>() {
            try_swapping_with_filter(projection, filter)?
        } else if let Some(repartition) = input.downcast_ref::<RepartitionExec>() {
            try_swapping_with_repartition(projection, repartition)?
        } else if let Some(sort) = input.downcast_ref::<SortExec>() {
            try_swapping_with_sort(projection, sort)?

Describe the solution you'd like

It would be nice if there was an abstraction somehow where ProjectionPushdown could be used by extension / user defined ExecutionPlan nodes

Describe alternatives you've considered

There is a really nice discussion between @crepererum and @ozankabak here #8073 (comment) that provides insight and various options.

Additional context

No response

[yyy1000]

I'd like to help it.
A question is when I looked at #8073 (comment) , I understand adding a function to ExecutionPlan and let every impl ExecutionPlan also has it. But I couldn't understand what does D: abstraction mean. 🤔

[berkaysynnada]

Hi @yyy1000, thank you for your willingness to help. I've been working on improving the projection optimizer rule to ensure the most efficient use of projections while minimizing table loads. I plan to complete my work in a few days and will open the PR then. That new rule will have a similar limitations, so I will seek suggestions on how to integrate the rule with the ExecutionPlan and PhysicalExpr APIs. I believe your assistance will be much more valuable then.