Optimization rule `CommonSubexprEliminate` creates invalid projections

[andygrove]

Describe the bug
I am trying to run some complex SQL and am seeing errors such as:

thread 'q11' panicked at 'index out of bounds: the len is 12 but the index is 12', /home/andy/.cargo/git/checkouts/arrow-datafusion-71ae82d9dec9a01c/9401d6d/datafusion/optimizer/src/projection_push_down.rs:159:39

With the changes in #2900 I see a slighter more informative error:

FAILURE: Plan("Projection has mismatch between number of expressions (12) and number of fields in schema (42)")

I can clearly see that some of the code in build_project_plan is incorrect. It is meant to be building the projection output schema but is including all the fields from the projection input. Unfortunately, I do not understand the code well enough to try and fix this issue.

    let mut schema = DFSchema::new_with_metadata(fields, HashMap::new())?;
    schema.merge(input.schema()); // <--- this is not correct

    Ok(LogicalPlan::Projection(Projection::try_new_with_schema(
        project_exprs,
        Arc::new(input),
        Arc::new(schema),
        None,
    )?))

To Reproduce
I do not have a repro to share currently. I will try and create one.

Expected behavior
Should not fail. Rules should avoid changing plans if there are errors rather than panic.

Additional context
I have created #2909 as a workaround so that we can at least fail gracefully and skip this rule when it produces an error and still apply the other rules.

[andygrove]

@waynexia fyi - if you have any time available maybe you can give some guidance on this?