fix: fts flat search drops rows when avg_doc_length < 1.0

[wjones127]

Summary

• MemBM25Scorer::avg_doc_length() used integer division (total_tokens / num_docs), truncating values < 1.0 to 0. Changed to float division.
• flat_bm25_search_stream reconstructed total_tokens by casting the float avg back to u64 (avg_doc_length() as u64 * num_docs), losing precision. Added total_tokens() accessor to IndexBM25Scorer to pass the exact value through.

Fixes #5871

Test plan

• New test test_fts_unindexed_data_with_stop_words — indexes 4 single-word rows (3 stop words) so avg_doc_length = 0.25, appends 10 unindexed rows, and asserts all 10 are returned by FTS query. Verified it fails without the fix (returns 7) and passes with it.
• Existing test_fts_unindexed_data still passes
• lance-index tests pass

🤖 Generated with Claude Code

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[github-actions]

Code Review

Summary: Clean bug fix for FTS flat search dropping rows when avg_doc_length < 1.0.

No P0/P1 issues identified. The fix is correct:

1. Integer division bug (scorer.rs:59): Changing (total_tokens / num_docs) as f32 to total_tokens as f32 / num_docs as f32 correctly preserves fractional averages.

2. Precision loss bug (index.rs:2541): Adding total_tokens() accessor avoids the lossy round-trip avg_doc_length() as u64 * num_docs.

3. Test coverage: The new test directly exercises the edge case (avg_doc_length = 0.25) and validates the fix.

LGTM ✓

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!