[GLUTEN-10524] Remove unnecessary outputAttributes from BasicScanExecTransformer#10525
[GLUTEN-10524] Remove unnecessary outputAttributes from BasicScanExecTransformer#10525philo-he merged 2 commits intoapache:mainfrom
outputAttributes from BasicScanExecTransformer#10525Conversation
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
| } else { | ||
| // Retrieve the original attributes based on expression ID so that capitalization matches. | ||
| val originalAttributes = AttributeMap(relation.output.map(a => a -> a)) | ||
| requestedAttributes.map(attr => originalAttributes.getOrElse(attr, attr)).distinct |
There was a problem hiding this comment.
How about add prunedOutput to AbstractHiveTableScanExec and update output function for AbstractHiveTableScanExec?
There was a problem hiding this comment.
It sounds we can override the output for AbstractHiveTableScanExec directly.
But it seems the other subclass of AbstractHiveTableScanExec will be added.
What about your opinion? @zhztheplayer
There was a problem hiding this comment.
@beliefer, AbstractHiveTableScanExec was introduced to fix code compatibility across different Spark versions. It resides in the shim layer to hold version-specific code for the corresponding Spark version. Therefore, I assume only HiveTableScanExecTransformer extends it, and no other classes will extend it in the future.
There was a problem hiding this comment.
@philo-he Thank you. Let's override the output for AbstractHiveTableScanExec directly.
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
philo-he
changed the title
outputAttributes from BasicScanExecTransformer
3 participants
What changes are proposed in this pull request?
This PR proposes to remove unnecessary
outputAttributesforBasicScanExecTransformer.Fixes #10524
How was this patch tested?
GA tests.