test: create framework for testing memory

[wjones127]

Adds a new integration test resource_test which contains a utility for making assertions about memory use with a span. This needs to be in a separate integration tests because it hooks into the global memory allocator, and doing that for all tests would probably make them quite slow.

Here's an example of how this can be used:

let alloc_tracker = AllocTracker::new();
{
    let _guard = alloc_tracker.enter();
    // Do something that allocates, even in async tasks
}
let stats = alloc_tracker.stats();
assert_lt!(stats.max_bytes_allocated, 100_000_000, "Used more than 100MB");
assert_eq!(stats.total_bytes_allocated, stats.total_bytes_deallocated, "No memory leaked");

Closes #4761

[codecov-commenter]

Codecov Report

❌ Patch coverage is 91.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.69%. Comparing base (9994f69) to head (26af669).
⚠️ Report is 26 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/lance-core/src/cache.rs	83.33%	1 Missing ⚠️
rust/lance-datafusion/src/datagen.rs	90.90%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4921      +/-   ##
==========================================
+ Coverage   81.38%   81.69%   +0.31%     
==========================================
  Files         333      334       +1     
  Lines      130056   132747    +2691     
  Branches   130056   132747    +2691     
==========================================
+ Hits       105847   108454    +2607     
+ Misses      20654    20649       -5     
- Partials     3555     3644      +89     

Flag	Coverage Δ
unittests	81.69% <91.66%> (+0.31%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wkalt]

I wonder if it would make sense to wrap this up in a macro. Would that promote more widespread usage?

#[maxallocs=10*1024*1024]
{
        let tmp_dir = tempfile::tempdir().unwrap();
        let tmp_path = tmp_dir.path().to_str().unwrap();
        let _dataset = InsertBuilder::new(tmp_path)
            .execute_stream(data)
            .await
            .unwrap();
}

[wjones127]

We can add that later if we find it simplifies things. Will be easier to see once we have more tests in here and see what patterns are repeating.

[wjones127]

Had to disable this for MacOS. We can implement a version that is cross platform in a future PR.