feat: support multiple index segments for FTS indices

[wjones127]

Summary

• FTS indices now support multiple segments via optimize_indices, matching IVF_PQ vector index behavior
• optimize_indices(append()) creates a new segment for unindexed data without rebuilding existing segments
• optimize_indices(merge(N)) merges the N most recent segments into one
• All FTS query types (Match, Phrase, Boolean, Boost) search across all segments with unified cross-segment BM25 scoring (IDF reweighting), so multi-segment results are identical to single-segment

Previously, the scalar branch of merge_indices hardcoded indices_merged = 1, always merging everything into one index. Now it respects OptimizeOptions::num_indices_to_merge.

Test plan

• test_fts_append_creates_separate_segment — append creates 2 disjoint segments
• test_fts_multi_segment_score_equivalence — identical BM25 scores for 6 queries across single vs multi-segment
• test_fts_merge_combines_segments — merge(2) reduces 2→1 segment, results unchanged
• test_fts_three_segments_partial_merge — 3 segments → merge(2) → 2 segments
• test_fts_multi_segment_with_unindexed_data — 2 segments + unindexed flat scan
• test_fts_multi_segment_phrase_query — phrase queries across segments
• All existing inverted index tests pass

🤖 Generated with Claude Code

[github-actions]

PR Review: feat: support multiple index segments for FTS indices

P0: Prefilter built from incomplete metadata (correctness bug)

In both MatchQueryExec::execute and PhraseQueryExec::execute, after calling load_all_fts_segments() (which internally loads all segment metadata), the code calls load_scalar_index() again and passes the result to build_prefilter(). But load_scalar_index returns Option of IndexMetadata -- at most ONE segment metadata. The .into_iter().collect() produces a Vec of 0 or 1 elements. So the prefilter is only built from the first segment fragment bitmap, not all segments.

DatasetPreFilter::new() uses the metadata fragment_bitmap to determine which fragments need deletion masking. With multi-segment indices, fragments covered by non-first segments will not have their deletion vectors loaded, so deleted rows from those fragments could appear in query results.

Fix: Have load_all_fts_segments return the Vec of IndexMetadata alongside the indices (it already loads this internally), and pass the full metadata list to build_prefilter. This also eliminates the redundant load_scalar_index call (P1 below).

P1: Redundant index metadata loading

load_all_fts_segments internally calls load_scalar_index + load_indices_by_name, loading all metadata. Then both MatchQueryExec and PhraseQueryExec call load_scalar_index again. This doubles the metadata load cost. Returning the metadata from load_all_fts_segments fixes both the P0 and this.

Minor: Redundant frag_bitmap OR in merge path

In append.rs, the merge branch (num_to_merge > 0) does frag_bitmap |= effective_old_frags inside the match arm. Then after the match, the generic code at line 302-306 ORs the same fragments again via removed_indices. Since bitmap OR is idempotent this is correct, but the duplication could cause confusion for future readers.

Tests

Good coverage of append, merge, partial merge, unindexed data, and phrase queries across segments. The score equivalence test is a nice touch. However, none of the tests exercise the deletion path across multi-segment indices (delete rows, then query), which is exactly where the P0 bug above would manifest. Consider adding a test that deletes rows from data covered by a non-first segment and verifies they do not appear in results.