Describe the bug
Enabling columnar shuffle by default PR #250 causes a lot of test failures in Spark org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite.
This is a tricky bug.
I found that it is because Comet sets incorrect logical link to CometShuffleExchangeExec. Because logical link is used in Spark AQE framework to do query planning, incorrect logical links cause test failures there.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Describe the bug
Enabling columnar shuffle by default PR #250 causes a lot of test failures in Spark
org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite.This is a tricky bug.
I found that it is because Comet sets incorrect logical link to
CometShuffleExchangeExec. Because logical link is used in Spark AQE framework to do query planning, incorrect logical links cause test failures there.Steps to reproduce
No response
Expected behavior
No response
Additional context
No response