Conversation
|
|
||
| ### Complex Type Support in Parquet Scans | ||
|
|
||
| Comet now supports complex types (Structs, Maps, and Arrays) when reading Parquet files. This functionality is not |
There was a problem hiding this comment.
should we clarify it is available through native_datafusion reader?
There was a problem hiding this comment.
@comphead users no longer have to choose the implementation. The 'auto' mode will use the best scan option given the schema.
There was a problem hiding this comment.
I have tried to clarify this in this PR to update the documentation - apache/datafusion-comet#1981
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
thanks @andygrove
Nice release notes, can't say any better
| <table> | ||
| <thead> | ||
| <tr> | ||
| <th>Module</th> |
There was a problem hiding this comment.
Wondering what Module means here...
There was a problem hiding this comment.
This is the header for the column containing values core-1, core-2, etc
|
|
||
| ### Improved Spark SQL Test Coverage | ||
|
|
||
| Comet now passes 97% of the Spark SQL test suite, with more than 24,000 tests passing (based on testing against |
There was a problem hiding this comment.
Is this 97% considering tests we legitimately skip?
There was a problem hiding this comment.
This is 97% of all tests
There was a problem hiding this comment.
I wanted to make sure we do not give an impression that we have 3% of incompatibility. Many of them are legitimately skipped because simply it does not apply for Comet.
There was a problem hiding this comment.
The next sentence was intended to address this:
The remaining ignored tests are mostly related to metric differences or tests irrelevant to
Comet, such as tests for whole-stage code generation.
There was a problem hiding this comment.
I have rewritten this section to clarify that the remaining 3% of tests are not ignored due to compatibility issues.
|
Thanks for the reviews @kazuyukitanimura @parthchandra @comphead. I'll plan on merging this tomorrow once the release vote passes. |
4 participants
Preview: https://datafusion.staged.apache.org/blog/2025/07/01/datafusion-comet-0.9.0/