Skip to content

Broken logical plan serialization for aggregation queries  #3555

@thinkharderdev

Description

@thinkharderdev

Describe the bug
A clear and concise description of what the bug is.

Some aggregation plans will fail to deserialize. This is caused by a rewrite done by the TypeCoercion optimizer.

To Reproduce
Steps to reproduce the behavior:

    #[tokio::test]
    async fn roundtrip_logical_plan_aggregation() -> Result<(), DataFusionError> {
        let ctx = SessionContext::new();

        let schema = Schema::new(vec![
            Field::new("a", DataType::Int64, true),
            Field::new("b", DataType::Decimal128(15,2), true)
        ]);

        ctx.register_csv("t1", "testdata/test.csv", CsvReadOptions::default().schema(&schema)).await?;

        let query = "SELECT a, SUM(b + 1) FROM t1 GROUP BY a";
        let plan = ctx.sql(query).await?.to_logical_plan()?;

        let bytes =
            logical_plan_to_bytes(&plan)?;
        let logical_round_trip =
            logical_plan_from_bytes(&bytes, &ctx)?;
        assert_eq!(
            format!("{:?}", plan),
            format!("{:?}", logical_round_trip)
        );
        Ok(())

This fails with

Error: SchemaError(FieldNotFound { qualifier: None, name: "SUM(t1.b + Int64(1))", valid_fields: Some(["t1.a", "SUM(t1.b + Decimal128(Some(100),23,2))", "t1.a", "t1.b"]) })

The problem is there is a projection after this which projects the original expression name. Typically this doesn't matter since the schema is unchanged and everything still aligns. But when we deserialize we have to build the plan again and it will fail to validate.

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions