Skip to content

feat: add an arrow-stats crate with the ability to calculate basic stats on arrays#5967

Merged
westonpace merged 7 commits intolance-format:mainfrom
westonpace:feat/arrow-stats
Apr 1, 2026
Merged

feat: add an arrow-stats crate with the ability to calculate basic stats on arrays#5967
westonpace merged 7 commits intolance-format:mainfrom
westonpace:feat/arrow-stats

Conversation

@westonpace
Copy link
Copy Markdown
Member

@westonpace westonpace commented Feb 18, 2026

We currently have stats spread out in a few places and they all require Datafusion which is a kind of heavy dependency. With the addition of column statistics (#5639) we were going to get yet another implementation. Instead I have made a simple standalone stats crate which does not require Datafusion.

@github-actions github-actions Bot added the enhancement New feature or request label Feb 18, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 18, 2026

PR Review: arrow-stats crate

Good addition that consolidates statistics calculation with a lightweight dependency footprint (no DataFusion required).

P1: Missing README.md

The Cargo.toml references readme = "README.md" but no README file is included in the PR. This will cause issues when publishing the crate.

Fix: Either add a README.md or remove the readme line from Cargo.toml.

Observations (non-blocking)

  • Test coverage is excellent with both unit tests and property-based tests
  • The Float16 support via half crate is well-handled
  • The total_cmp usage for floats correctly handles -0.0 vs 0.0 ordering

@codecov
Copy link
Copy Markdown

codecov Bot commented Feb 18, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 91.48936% with 56 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/arrow-stats/src/lib.rs 91.70% 40 Missing and 14 partials ⚠️
rust/arrow-stats/src/nan.rs 71.42% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@westonpace westonpace mentioned this pull request Mar 18, 2026
Open
@HaochengLIU HaochengLIU self-requested a review March 19, 2026 01:20
HaochengLIU
HaochengLIU reviewed Mar 19, 2026
View reviewed changes
Comment thread rust/arrow-stats/Cargo.toml
HaochengLIU
HaochengLIU reviewed Mar 19, 2026
View reviewed changes
Comment thread rust/arrow-stats/src/lib.rs
//!
//! # List Types
//!
//! List types are supported. The `item_nulls` field will be set to the number of null items within list entries.
Copy link
Copy Markdown
Member

@HaochengLIU HaochengLIU Mar 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think nested list is also not supported?

HaochengLIU
HaochengLIU reviewed Mar 19, 2026
View reviewed changes
Comment thread rust/arrow-stats/src/lib.rs
HaochengLIU
HaochengLIU reviewed Mar 19, 2026
View reviewed changes
Comment thread rust/arrow-stats/Cargo.toml
@HaochengLIU
Copy link
Copy Markdown
Member

HaochengLIU commented Mar 19, 2026

@westonpace added some minor comments, overall code LGTM

@HaochengLIU
Copy link
Copy Markdown
Member

HaochengLIU commented Mar 30, 2026

@westonpace gentle ping

westonpace and others added 4 commits March 31, 2026 05:58
@westonpace @claude
Add README for lance-arrow-stats crate
5a571a1 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@westonpace @claude
Support nested list types in StatisticsAccumulator
d9b018f 
Recurse through nested list types (List, LargeList, FixedSizeList) to
compute min/max over leaf values and track item_nulls at every nesting
level. Previously nested lists fell through to the unsupported-type
fallback, yielding None for min/max.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@westonpace
Fix old reference to arrow-scalar which is now lance-arrow-scalar
2b1586f
@westonpace @claude
Fix clippy errors for Rust 2024 match ergonomics
21473c4 
Remove redundant `ref mut` and `ref` binding modifiers that are now
disallowed under the Rust 2024 default binding mode rules.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
HaochengLIU
HaochengLIU approved these changes Apr 1, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@HaochengLIU HaochengLIU left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm now

@HaochengLIU
Copy link
Copy Markdown
Member

HaochengLIU commented Apr 1, 2026

@westonpace good to go!

@westonpace @claude
Fix doctest import to use renamed crate name
8110632 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@westonpace westonpace merged commit 21ae66b into lance-format:main Apr 1, 2026
28 checks passed
@westonpace
Copy link
Copy Markdown
Member Author

westonpace commented Apr 1, 2026

Thanks for the review @HaochengLIU !

Next step: update the column statistics PR to use lance_arrow_scalar and lance_arrow_stats

@HaochengLIU
Copy link
Copy Markdown
Member

HaochengLIU commented Apr 1, 2026

Thanks for the review @HaochengLIU !

Next step: update the column statistics PR to use lance_arrow_scalar and lance_arrow_stats

sure thing! I've semi-settled in in my new environment, let me know if there is anything I can help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HaochengLIU HaochengLIU HaochengLIU approved these changes

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@westonpace @HaochengLIU