[Python] Memory leak in pyarrow functions .ipc.serialize_pandas/deserialize_pandas

[asfimport]

Used python version is 3.7.4 (conda distribution)

OS: Ubunty 18.04

pandas version is 0.24.2

numpy version is 1.16.4

 

To reproduce the issue run the attached script pyarrow_mem_leak_test.py. Also put the attached file loans.parquet to the folder of working directory.

 

Also the reading and writing to parquet in memory do has memory leaks. To reproduce this run function test_parquet_leak() from the attached file pyarrow_mem_leak_test.py

The memory leak is 100% reproducible.

Reporter: Denis
Assignee: Wes McKinney / @wesm

Related issues:

• [Python] Define process for testing procedures that check for no macro-level memory leaks (relates to)

Original Issue Attachments:

• loans.parquet
• pyarrow_mem_leak_test.py

_{Note: This issue was originally created as ARROW-7956. Please see the migration documentation for further details.}

[asfimport]

Denis:
Not reproduced with pyarrow==0.16.0.

Closing this ticket

[asfimport]

Denis:
Not reproduced with latest version

[asfimport]

Wes McKinney / @wesm:
I reopened this as I want to make sure there is an appropriate unit test (or equivalent) for this

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
It seems the object dtype is the trigger. I can reproduce this on 0.15 with the following simplified snippet (without involving a parquet file):

import pyarrow as pa
import pandas as pd

def test_pyarrow_leak():
    df = pd.DataFrame({'a': np.arange(10000), 'b': [pd.util.testing.rands(5) for _ in range(10000)]})
    for i in range(4000):
        print(f'Iteration {i}')
        df_bytes = pa.ipc.serialize_pandas(df).to_pybytes()
        buf = pa.py_buffer(df_bytes)
        df = pa.ipc.deserialize_pandas(buf)

    print('End of script')

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@wesm I think this was closed by #6551 ?

[asfimport]

Wes McKinney / @wesm:
Yes, resolved as part of the patch for ARROW-4120