Describe the bug
Trying to use CometScanExec with Spark 3.5.2 results in a runtime exception.
Steps to reproduce
Class not found error at runtime, but can also be replicated by trying to compile Comet with Spark 3.5.2:
[ERROR] /Users/abinford/projects/arrow-datafusion-comet/spark/src/main/scala/org/apache/spark/sql/comet/CometScanExec.scala:144: value OP_ID_TAG is not a member of object org.apache.spark.sql.catalyst.plans.QueryPlan
Expected behavior
Compilation and runtime succeed
Additional context
This is due to apache/spark@88582e1 which removed OP_ID_TAG and uses a different mechanism to track operator IDs.
Describe the bug
Trying to use
CometScanExecwith Spark 3.5.2 results in a runtime exception.Steps to reproduce
Class not found error at runtime, but can also be replicated by trying to compile Comet with Spark 3.5.2:
Expected behavior
Compilation and runtime succeed
Additional context
This is due to apache/spark@88582e1 which removed
OP_ID_TAGand uses a different mechanism to track operator IDs.