fix: memory-limited string BTREE index building

[wjones127]

Closes #4881

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.29%. Comparing base (f48bbd9) to head (ca03048).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5175      +/-   ##
==========================================
- Coverage   82.29%   82.29%   -0.01%     
==========================================
  Files         344      344              
  Lines      144765   144757       -8     
  Branches   144765   144757       -8     
==========================================
- Hits       119137   119128       -9     
- Misses      21709    21710       +1     
  Partials     3919     3919              

Flag	Coverage Δ
unittests	82.29% <100.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wjones127]

Memory benchmark

Creating a BTree index on a 1GB string column with LANCE_MEM_POOL_SIZE set to 20MB.

Script

# LD_PRELOAD=/usr/local/lib/libbytehound.so python test_btree.py
# bytehound server memory-profiling_*.dat
import os

os.environ['LANCE_MEM_POOL_SIZE'] = str(20 * 1024 * 1024)  # 20 MB

import pyarrow as pa
import lance
from datetime import datetime

def gen_data():
    # 1GB of strings
    for i in range(100):
        # 10MB of strings
        array_len = 10 * 1024 * 1024 // 8
        arr = pa.array([f'{j:>8}' for j in range(i * array_len, (i + 1) * array_len)])
        yield pa.RecordBatch.from_arrays(
            [arr],
            ['strings']
        )

reader = pa.RecordBatchReader.from_batches(
    schema=pa.schema([pa.field('strings', pa.string())]),
    batches=gen_data()
)

ds = lance.write_dataset(
    reader,
    './tmp',
    mode='overwrite'
)

print(datetime.now(), "Creating index")

ds.create_scalar_index('strings', index_type="BTREE")

Before: 2.7GB peak

After: 240MB peak

I believe the memory use is from the scanner buffering IO and reading batches ahead. After the initial scan, it does a pretty good job of keeping with the memory pool amount.