Summary
DatasetIndexExt currently lives in lance-index, which forces IndexSegment to live there too even though it is not really part of the index implementation layer.
In #6209, this showed up as an awkward layering issue:
IndexSegment is only used as an API type for dataset-level index commit flowslance-index does not otherwise consume it internally- the main reason it lives in
lance-index today is that DatasetIndexExt is also defined there
Problem
This blurs the boundary between:
- index implementation concerns (
lance-index)
- dataset-facing API concerns (
lance)
As a result, types that conceptually belong to dataset index management end up living in the lower-level crate only because the trait is there.
Follow-up
We should move DatasetIndexExt into lance (or otherwise restructure the trait boundary), and then move IndexSegment with it so the API surface lives in the more natural layer.
This should be handled as follow-up work instead of extending the scope of #6209.
Summary
DatasetIndexExtcurrently lives inlance-index, which forcesIndexSegmentto live there too even though it is not really part of the index implementation layer.In #6209, this showed up as an awkward layering issue:
IndexSegmentis only used as an API type for dataset-level index commit flowslance-indexdoes not otherwise consume it internallylance-indextoday is thatDatasetIndexExtis also defined thereProblem
This blurs the boundary between:
lance-index)lance)As a result, types that conceptually belong to dataset index management end up living in the lower-level crate only because the trait is there.
Follow-up
We should move
DatasetIndexExtintolance(or otherwise restructure the trait boundary), and then moveIndexSegmentwith it so the API surface lives in the more natural layer.This should be handled as follow-up work instead of extending the scope of #6209.