Update arrow 37 #5782
Update arrow 37 #5782
Conversation
github-actions
bot
added
core
| }) | ||
| .collect(); | ||
| .collect::<Vec<_>>(); | ||
| let arrays = arrays.iter().collect(); |
| .map(|b| b.column(i).data()) | ||
| .map(|b| b.column(i).to_data()) | ||
| .collect::<Vec<_>>(); | ||
| let arrays = arrays.iter().collect(); |
| statement ok | ||
| create table foo as select | ||
| arrow_cast(100, 'Decimal128(3,2)') as col_d128 | ||
| arrow_cast(100, 'Decimal128(5,2)') as col_d128 |
There was a problem hiding this comment.
This results from - apache/arrow-rs#3996
Which now complains (correctly) that 100 cannot be represented in an decimal with scale 2 and precision 3
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
This looks good to me -- I haven't quite made it through this PR yet but what I have seen so far is good.
I plan to run the planning benchmark against this PR later today to see if it improves things
| statement ok | ||
| create table foo as select | ||
| arrow_cast(100, 'Decimal128(3,2)') as col_d128 | ||
| arrow_cast(100, 'Decimal128(5,2)') as col_d128 |
| df_schema.fields.into_iter().map(|f| f.field).collect(), | ||
| df_schema.metadata, | ||
| ) | ||
| let fields: Fields = df_schema.fields.into_iter().map(|f| f.field).collect(); |
There was a problem hiding this comment.
🤮 at least this should be cheaper at runtime
|
I took the liberty of merging up from main to resolve a conflict |
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
I think it looks great. Thank you @tustvold
I also ran the sql_planning benchmark, and sadly this branch doesn't show any improvements.
Given that this is a non trivial change that will take time to percolate through the ecosystem I plan to merge this shortly
* Update arrow 37 * Fix avro * Fix sql_planner benchmark * Update arrow pin * Format * Fix test * Remove pin * Update version * Fix logical merge conflicts * Pyarrow clippy * More clippy * fixup --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2 participants
Which issue does this PR close?
Closes #.
Rationale for this change
Arrow 37 has some non-trivial breaking changes to schema handling, lets get ahead of this
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?