feat: simple auto cleanup

[dsgibbons]

Closes #2100.

This PR introduces:

• A hook called auto_cleanup_hook. If config keys lance.auto_cleanup.interval and lance.auto_cleanup.older_than are both set, then, every n_versions % lance.auto_cleanup.interval, cleanup_old_versions will automatically be called.
• A new write param auto_cleanup: Option<AutoCleanupParams>. If Some, new datasets are configured with the necessary lance.auto_cleanup config keys. By default this sets the interval=20 and older_than=14.
• A test to ensure hook is automatically invoked. Some older tests have been fixed as they either do not assume autoclaving will take place, or assume the initial default config is empty.

This PR is Rust-only. I can add Python bindings in a future PR if desired.

From #2100:

• Add configuration key for "auto_cleanup.interval", which is number of versions to have between running.
• Add a configuration key for "auto_cleanup.older_than", which is the older_than parameter for cleanup_old_versions.
• Double check documented default value of two weeks. Maybe it should be seven days. (two weeks seems reasonable to me?)
• Add hook to do this in the commit path.
• Document how to manually call cleanup_old_versions (not done yet - where would you like this documenting?)
• Document how to configure auto cleanup and how to disable it. (done in the doc comment for AutoCleanupParams - let me know if you want this documented elsewhere)

[dsgibbons]

Finally ready for review @wjones127 - better late than never? :)

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 86.30952% with 23 lines in your changes missing coverage. Please review.

Project coverage is 78.40%. Comparing base (be688a9) to head (17b7223).
Report is 245 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/cleanup.rs	86.88%	15 Missing and 1 partial ⚠️
rust/lance/src/dataset/write/insert.rs	83.33%	5 Missing ⚠️
rust/lance/src/io/commit.rs	71.42%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3572      +/-   ##
==========================================
+ Coverage   78.38%   78.40%   +0.01%     
==========================================
  Files         261      261              
  Lines       99344    99507     +163     
  Branches    99344    99507     +163     
==========================================
+ Hits        77871    78014     +143     
- Misses      18356    18375      +19     
- Partials     3117     3118       +1     

Flag	Coverage Δ
unittests	78.40% <86.30%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wjones127]

This is great work. Have a few small tweaks, and then I think this will be ready.

[dsgibbons]

@wjones127 I've rebased on main and removed those logging tests. Ready for review.

[wjones127]

My one remaining concern is whether this is the right unit. Days seems a little coarse.

One option is to user a finer time unit, like minutes or hours.

But another nice option might be to take a standard duration format, such as 3d 2h 5m. 8601 has a standard for this, but it's a little ugly (P3Y6M4DT12H30M5S). Even just accepting one number and one unit seems like a welcome improvement, though, and I think I prefer that over a fixed unit, since it makes the configuration more readable.

[dsgibbons]

(Hopefully) resolved by 205f611. I added humantime::parse_duration for human-friendly duration parsing. While this adds an additional dependency to Cargo.toml, it turns out that Cargo.lock contains this dependency as part of object_store (see here). I've expanded out the test case to include a custom duration of "1month 2days 2h 42min 6sec" and it seems to work fine.

In WriteParams, I've updated the older_than argument to be of type chrono::TimeDelta (as opposed to usize). humantime::parse_duration works on std::time::Duration, not chrono::TimeDelta, but I chose chrono::TimeDelta because this then aligns with the type signature in cleanup_old_versions, at the cost of an additional conversion from chromo::TimeDelta to std::time::Duration.

[dsgibbons]

See RUSTSEC-2025-0014 for reference.

[dsgibbons]

@wjones127 anything else needed to get this over the line?

[wjones127]

Nice work here! Sorry for the delay in my final review.

[yanghua]

@dsgibbons Does it mean the auto cleanup takes effect by default? Is there a switch for this?

[dsgibbons]

@yanghua That's right. I do wonder whether enabling destructive actions should be the default. You can set auto_cleanup: None in your WriteParams if you want to disable this behaviour. A couple of the existing unit tests were altered in this way by this PR to disable auto cleanup.

[yanghua]

Can we set None as the default value? Users may not know this default behavior and may not expect it.

[dsgibbons]

Sounds good to me. Thoughts @wjones127?

[yanghua]

@wjones127 Any thoughts?

[wjones127]

@yanghua let's discuss this in a dedicated issue, so the discussion is easier to find. Most people don't go looking in comment on closed PRs.

I've started a discussion here:
#3854