Move rate limiting from ScanScheduler into the object store layer

[westonpace]

Parent Issue

Part of OSS-636 (Implement AIMD Adaptive Retry)

Depends on OSS-639 (Create a proxy ObjectStore that applies the AIMD throttle)

Summary

Move the existing concurrency-based rate limiting out of ScanScheduler and into the object store layer (via the proxy ObjectStore from OSS-639). The I/O buffer rate limiting will remain in ScanScheduler.

Details

The current concurrency throttling in Lance has several weaknesses:

1. Read-path only: It is applied through ScanScheduler, which only affects the read path and even then does not cover all reads
2. Misplaced scope: It is both per-scan and global. The per-scan throttle is too local (why should we care about how many concurrent requests a single scan makes) and the global throttle is too global (what if a user has multiple storage accounts or buckets)
3. Not universally applicable: Throttling may not make sense for all object stores (e.g., when using NVMe with extremely high IOPS rates, just trying to obtain a semaphore can be a bottleneck)

By moving rate limiting into the object store layer:

• It will apply to both read and write paths
• It can be scoped per storage account/bucket rather than globally
• It can be disabled for backends where it doesn't make sense (e.g., local NVMe)
• ScanScheduler retains only I/O buffer management responsibility, simplifying its role