feat: cleanup only scan managed files

[majin1102]

Close #5294

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[codecov]

Codecov Report

❌ Patch coverage is 72.72727% with 21 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/cleanup.rs	76.56%	13 Missing and 2 partials ⚠️
rust/lance/src/dataset.rs	40.00%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

[majin1102]

Hi @westonpace, this PR is ready for review.

Please take a look if you have time.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[yanghua]

Overall, LGTM, only a minor question.

[majin1102]

Thanks for reviewing, I will wait 2 days to see if there are any other comments.
cc @westonpace @jackye1995

[westonpace]

Overdue review but wanted to thank you for the contribution and good idea.

[westonpace]

There's some inconsistency with all these dir constants. Some are here, others are in dataset.rs. I don't think it really matters but it might be nice sometime to just put them all in datasets.rs and either make them all pub (which I think we've done here) or refer to them all from the Dataset accessor functions.

[majin1102]

Thanks for the suggestion.

I was planning to go with the latter approach, but I didn’t want to be too aggressive. I will carry this forward in other PRs