[Python] DataSet uses too much memory when filtering

[lnicola]

I'm running this query over a 14 GB Arrow IPC file:

>>> ds = dataset.dataset("foo.ipc", format="ipc")
>>> t = ds.to_table(filter=dataset.field('ID') <= 1000).to_pandas()
>>> t
[snip]
[914 rows x 617 columns]

If I'm reading the documentation correctly, it should scan the file collecting the results, but not load it in memory. However, the RSS grows up to about 14 GB while running it, then goes back down.

[wesm]

Filtering does indeed load the filtered data into memory at present time. Can you point out the documentation that indicates otherwise?

[lnicola]

I guess it doesn't say that filtering doesn't load the whole file, but:

The pyarrow.dataset module provides functionality to efficiently work with tabular, potentially larger than memory and multi-file datasets:

But filter takes an expression and it would seem possible to implement without loading the full table (e.g. using dataset.scan), so it's quite surprising that it works like this.

Anyway, we can close this if it's by design, or we can leave it open if it's simply something not implemented yet.

[wesm]

We're actively working toward having a complete streaming query engine and a low-level data frame API that will permit analytics on filtered, memory-mapped data without having to materialize everything in memory. It has been discussed at length on the mailing list. In the meantime to read datasets into pandas requires materializing them. So I'll close this for now and refer you to further discussions over e-mail