datafusion.optimizer.repartition_file_scans enabled by default

[korowa]

Which issue does this PR close?

Closes #5125.

Rationale for this change

I guess, it's fine to enable repartitioning by default in 19.0.0 (or, more likely, first release candidate for 19.0.0)

What changes are included in this PR?

Default value of datafusion.optimizer.repartition_file_scans is true now.

Are these changes tested?

Covered by existing tests

Are there any user-facing changes?

Repartitioning of file scans will be enabled by default

[alamb]

I think it is a good idea. Thank you @korowa

[alamb]

What do you think @tustvold @Dandandan @andygrove -- any concerns about turning on automatic repartitioned file scans (which allows scanning a single large parquet file in parallel)

[ozankabak]

This should help with mitigating the "low-performance" impression -- many people will not dig deep to configuration options and simply try things out with OOTB defaults.

[alamb]

I will plan to merge this sometime over the weekend unless anyone else would like time to comment or offer more thoughts

[jackwener]

This should help with mitigating the "low-performance" impression -- many people will not dig deep to configuration options and simply try things out with OOTB defaults.

Look great to me.

[ursabot]

Benchmark runs are scheduled for baseline = cfbb14d and contender = 222205d. 222205d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java