refactor: move DatasetIndexExt out of lance-index

[Xuanwo]

This moves DatasetIndexExt and the dataset-facing index segment types out of lance-index and into lance, so the public dataset index management API lives in the dataset layer instead of the lower-level index implementation crate.

Close #6221

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[github-actions]

Review

Clean refactoring — no P0/P1 issues found.

Reviewed the full diff across all 55 files. The trait and type definitions in the new lance/src/index/api.rs are faithful copies from lance-index, and all workspace-internal import paths have been updated consistently. Nice catch on the "indies" → "indices" typo fix in the doc comment.

One minor note: lance-index is a published crate (no publish = false in Cargo.toml), so removing DatasetIndexExt, IndexSegment, and IndexSegmentPlan from its public API is a semver-breaking change for any external consumers. The PR title already signals this, but worth confirming that downstream users (e.g. Python/Java bindings, lance-namespace-impls) are all accounted for — which they appear to be from the diff.

LGTM.

[codecov]

Codecov Report

❌ Patch coverage is 96.73913% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/api.rs	96.73%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!