ARROW-11895: [Rust][DataFusion] Add support for more column statistics

[Dandandan]

This PR

• adds max/min/distinct_count column statistics
• implements null count statistics for the parquet reader & table provider

This will allow implementing more advanced statistics based optimizations, such as outlined in this article.

https://databricks.com/blog/2017/08/31/cost-based-optimizer-in-apache-spark-2-2.html

[github-actions]

https://issues.apache.org/jira/browse/ARROW-11895

[codecov-io]

Codecov Report

Merging #9646 (b5bafa5) into master (bfa99d9) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #9646   +/-   ##
=======================================
  Coverage   82.49%   82.50%           
=======================================
  Files         245      245           
  Lines       57347    57375   +28     
=======================================
+ Hits        47311    47339   +28     
  Misses      10036    10036           

Impacted Files	Coverage Δ
rust/datafusion/src/datasource/datasource.rs	100.00% <ø> (ø)
rust/datafusion/src/datasource/memory.rs	85.15% <100.00%> (+1.04%)	⬆️
rust/datafusion/src/physical_plan/parquet.rs	88.08% <100.00%> (+0.24%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bfa99d9...b5bafa5. Read the comment docs.

[Dandandan]

FYI @andygrove I plan to use this later for some ANALYZE / COMPUTE STATISTICS support (would be useful for in memory data) and/or use parquet statistics to make the join order optimization more advanced.

[alamb]

BTW the test workspace check is also failing on master, so it may not be related to this PR. See more details on https://issues.apache.org/jira/browse/ARROW-11896

[andygrove]

LGTM. Thanks @Dandandan

[alamb]

FYI I merged #9653 / ARROW-11896 for the Rust CI checks which may affect this PR. If you see "Rust / AMD64 Debian 10 Rust stable test workspace" failing with a linker error or no logs, rebasing against master will hopefully fix the problem

[seddonm1]

FYI @andygrove I plan to use this later for some ANALYZE / COMPUTE STATISTICS support (would be useful for in memory data) and/or use parquet statistics to make the join order optimization more advanced.

This is really cool @Dandandan . And we don't have to make apply the silly limitation that Spark does where the table has to be registered in a Hive Metastore to allow you to calculate statistics!

[Dandandan]

FYI @andygrove I plan to use this later for some ANALYZE / COMPUTE STATISTICS support (would be useful for in memory data) and/or use parquet statistics to make the join order optimization more advanced.

This is really cool @Dandandan . And we don't have to make apply the silly limitation that Spark does where the table has to be registered in a Hive Metastore to allow you to calculate statistics!

I am also pretty excited about using this for data science / BI use cases where you could load all data in memory. I think that would be a pretty unique feature.

[alamb]

Just to be clear, this is the infrastructure to implement more sophisticated statistics, but nothing (yet) takes advantage of them. I think it is a good step in the right direction. Thanks @Dandandan