Upgrade to arrow 21

[avantgardnerio]

Which issue does this PR close?

Closes #3224.

Rationale for this change

We have to track the latest arrow-rs, or DataFusion becomes less appealing for users.

What changes are included in this PR?

Fix build for breaking changes in arrow 21.

Are there any user-facing changes?

No

[codecov-commenter]

Codecov Report

Merging #3225 (cc267ef) into master (7dedcb1) will decrease coverage by 0.20%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3225      +/-   ##
==========================================
- Coverage   85.95%   85.74%   -0.21%     
==========================================
  Files         294      294              
  Lines       53560    53738     +178     
==========================================
+ Hits        46037    46078      +41     
- Misses       7523     7660     +137     

Impacted Files	Coverage Δ
datafusion/core/src/physical_optimizer/pruning.rs	94.75% <ø> (ø)
datafusion/core/src/physical_plan/hash_join.rs	94.94% <ø> (ø)
datafusion/core/src/physical_plan/hash_utils.rs	40.05% <ø> (ø)
datafusion/core/src/test/mod.rs	83.48% <ø> (ø)
datafusion/core/tests/parquet_pruning.rs	99.43% <ø> (ø)
datafusion/physical-expr/src/expressions/cast.rs	98.48% <ø> (ø)
...tafusion/physical-expr/src/expressions/try_cast.rs	98.79% <ø> (ø)
datafusion/common/src/scalar.rs	84.65% <100.00%> (ø)
datafusion/core/tests/sql/select.rs	99.77% <100.00%> (+<0.01%)	⬆️
datafusion/physical-expr/src/aggregate/average.rs	94.96% <100.00%> (ø)
... and 20 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[avantgardnerio]

@andygrove @alamb @tustvold I wanted to ping you just in case this is useful to you. I accidentally had this working against master too, so there's build fixes in the commit log of this branch for Arrow 22 if you need those at some point.

[avantgardnerio]

Also, for some reason, github doesn't seem to be pulling commits from this branch anymore and running CI on them.

It's been 3 commits to the source branch since tests were run. I'm not sure why that would be.

[avantgardnerio]

Okay, CI is caught up, this is ready for merge as soon as Arrow 21 goes out.

[andygrove]

Okay, CI is caught up, this is ready for merge as soon as Arrow 21 goes out.

@avantgardnerio Arrow 21.0.0 is released now. Do you want to update this PR to point to the release so we can merge?

[avantgardnerio]

@andygrove GTG

[andygrove]

What was the reason for this change?

[avantgardnerio]

We store precision as a u8 now. 234 fits, 455 doesn't.

[ursabot]

Benchmark runs are scheduled for baseline = a32c408 and contender = fcd2275. fcd2275 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[alamb]

Thank you for this @avantgardnerio 🙏