Skip to content

fix: CometReader.loadVector should not overwrite dictionary ids#476

Merged
viirya merged 2 commits intoapache:mainfrom
viirya:fix_dictionary_array_import
May 28, 2024
Merged

fix: CometReader.loadVector should not overwrite dictionary ids#476
viirya merged 2 commits intoapache:mainfrom
viirya:fix_dictionary_array_import

Conversation

@viirya
Copy link
Copy Markdown
Member

@viirya viirya commented May 26, 2024

Which issue does this PR close?

Closes #475.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

@codecov-commenter
Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 0% with 25 lines in your changes are missing coverage. Please review.

Project coverage is 34.03%. Comparing base (9125e6a) to head (0eb2ca6).
Report is 4 commits behind head on main.

Files Patch % Lines
...scala/org/apache/arrow/c/CometSchemaImporter.scala 0.00% 9 Missing ⚠️
...in/java/org/apache/comet/parquet/ColumnReader.java 0.00% 8 Missing ⚠️
.../scala/org/apache/spark/sql/comet/util/Utils.scala 0.00% 4 Missing ⚠️
...ain/java/org/apache/comet/parquet/BatchReader.java 0.00% 2 Missing ⚠️
...ava/org/apache/comet/parquet/LazyColumnReader.java 0.00% 1 Missing ⚠️
.../src/main/java/org/apache/comet/parquet/Utils.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main     #476      +/-   ##
============================================
- Coverage     34.05%   34.03%   -0.02%     
  Complexity      859      859              
============================================
  Files           116      117       +1     
  Lines         38680    38696      +16     
  Branches       8568     8567       -1     
============================================
  Hits          13171    13171              
- Misses        22746    22762      +16     
  Partials       2763     2763              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@viirya
Copy link
Copy Markdown
Member Author

viirya commented May 26, 2024

cc @sunchao @andygrove

Comment thread common/src/main/scala/org/apache/arrow/c/CometSchemaImporter.scala Outdated
Comment thread common/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala Outdated
@viirya
Copy link
Copy Markdown
Member Author

viirya commented May 27, 2024

CometSchemaImporter now is implemented in Java.

@viirya viirya force-pushed the fix_dictionary_array_import branch from c5e4f91 to 4678304 Compare May 27, 2024 17:25
@viirya viirya force-pushed the fix_dictionary_array_import branch from 4678304 to 0bc4933 Compare May 27, 2024 17:26
@viirya
Copy link
Copy Markdown
Member Author

viirya commented May 27, 2024

@sunchao Any more comments? Thanks.

@viirya viirya merged commit 479a97a into apache:main May 28, 2024
@viirya
Copy link
Copy Markdown
Member Author

viirya commented May 28, 2024

Merged. Thanks @andygrove @sunchao

@viirya viirya deleted the fix_dictionary_array_import branch May 28, 2024 01:55
himadripal pushed a commit to himadripal/datafusion-comet that referenced this pull request Sep 7, 2024
…he#476)

* fix: CometReader.loadVector should not overwrite dictionary ids

* For review

(cherry picked from commit 479a97a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CometReader.loadVector should not overwrite dictionary ids

4 participants