refactor: write bitmap index statistics in file instead

[Xuanwo]

Close #4620

This PR will write bitmap index statistics in file instead so we don't need to load the entire index file to calculate it.

---

This PR was primarily authored with Codex using GPT-5-Codex and then hand-reviewed by me. I AM responsible for every change made in this PR. I aimed to keep it aligned with our goals, though I may have missed minor issues. Please flag anything that feels off, I'll fix it quickly.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[codecov-commenter]

Codecov Report

❌ Patch coverage is 75.26882% with 23 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index.rs	85.13%	7 Missing and 4 partials ⚠️
rust/lance-index/src/scalar/bitmap.rs	33.33%	8 Missing and 2 partials ⚠️
rust/lance-index/src/scalar/bloomfilter.rs	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[wjones127]

I like the approach. I think we should figure out whether or not we want index_types() method or if we want something else. Weston is working on something in parallel in #5221

[wjones127]

Some suggestions to modernize io tracking

[westonpace]

Approve pending CI green.

I think we're good here on backwards / forwards compatibility?

If new code is reading an old index then if let Some(stats) = plugin.load_statistics(index_store.clone(), index_details.as_ref()).await? fails and we fallback to the old approach?

If old code is reading a new index then the old approach should still work since we didn't get rid of any information.

We should probably add "get statistics" to the forwards / backwards compatibility test suites at some point but perhaps can be in a follow-up?

[Xuanwo]

I think we're good here on backwards / forwards compatibility?

If new code is reading an old index then if let Some(stats) = plugin.load_statistics(index_store.clone(), index_details.as_ref()).await? fails and we fallback to the old approach?

If old code is reading a new index then the old approach should still work since we didn't get rid of any information.

Yes, we should be good on this, either way works.

We should probably add "get statistics" to the forwards / backwards compatibility test suites at some point but perhaps can be in a follow-up?

Makes sense to me.