Spark3.4: Enable Native execution if ParquetReaderType is Comet#12709
Spark3.4: Enable Native execution if ParquetReaderType is Comet#12709huaxingao wants to merge 1 commit intoapache:mainfrom
Conversation
huaxingao
force-pushed
the
enableNativeExecution
branch
from
aac3c54 to
5204c46
Compare
huaxingao
force-pushed
the
enableNativeExecution
branch
from
5204c46 to
e29d5cc
Compare
| relocate 'org.apache.avro', 'org.apache.iceberg.shaded.org.apache.avro' | ||
| relocate 'avro.shaded', 'org.apache.iceberg.shaded.org.apache.avro.shaded' | ||
| relocate 'com.thoughtworks.paranamer', 'org.apache.iceberg.shaded.com.thoughtworks.paranamer' | ||
| relocate 'org.apache.parquet', 'org.apache.iceberg.shaded.org.apache.parquet' |
There was a problem hiding this comment.
Is there a side effect for downstream projects?
| // TODO Update doc example so that it can actually be run, modifications were required for this | ||
| // test suite to run | ||
| @Test | ||
| @Ignore |
| TypeUtil.convertToParquet( | ||
| new StructField("_deleted", DataTypes.BooleanType, false, Metadata.empty())), | ||
| false /* useDecimal128 = false */, | ||
| true /* useDecimal128 = true */, |
There was a problem hiding this comment.
Is there a UT to catch this change?
|
|
||
| for (ScanTask task : taskIterable) { | ||
| if (task instanceof FileScanTask) { | ||
| if (!((FileScanTask) task).deletes().isEmpty()) { |
There was a problem hiding this comment.
We already have TableScanUtil.hasDeletes
| } | ||
|
|
||
| this.tasks = plannedTasks; | ||
| if (hasDeletes) { |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
@huaxingao Is this PR still active? We are actually really looking forward to having this for comet! |
|
@jiayuasu Yes, I will be working on this. One issue I'm currently facing is the shading issue with Iceberg. Since Iceberg shades Parquet, I can't pass Parquet objects and types from Iceberg to Comet. However, I don't think I can simply unshading Parquet. I likely need to introduce a higher level of abstraction to work around this problem. |
3 participants
This PR has the following changes:
SparkScanimplementorg.apache.comet.parquet.SupportsComet, so on Comet side, it can checkSupportsComet.isCometEnabled()and turn on native execution.useDecimal128to true in a few places, to be consistent with theuseDecimal128value in Comet