feat: support with_row_id options for spark connector#3084
feat: support with_row_id options for spark connector#3084SaintBacchus wants to merge 6 commits intolance-format:mainfrom
Conversation
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3084 +/- ##
==========================================
+ Coverage 77.31% 78.63% +1.31%
==========================================
Files 240 243 +3
Lines 79322 82873 +3551
Branches 79322 82873 +3551
==========================================
+ Hits 61326 65165 +3839
- Misses 14821 14921 +100
+ Partials 3175 2787 -388
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| for field in res.clone().fields() { | ||
| if field.name() == ROW_ID { | ||
| let new_field = match field.data_type() { | ||
| DataType::UInt64 => { |
There was a problem hiding this comment.
Trying to understand why making this change
Arrow in Java also have DataType:UInt64 but Spark cannot convert UInt64 to SparkType?
Could we modify the Spark Type to Arrow Type conversion directly? Like having an extension??
|
|
|
@SaintBacchus Can we support generic type conversion? |
implement SupportsMetadataColumns for LanceDataset
As discussion in [PR](#3084), I had implement the _rowid meta column just in java package.

_rowidcolumn for uint64 type.uint64intoint64and change the schema of_rowidfromuint64intoint64