[Python] Define process for testing procedures that check for no macro-level memory leaks

[asfimport]

Some kinds of memory leaks may be difficult to unit test for, and they may not cause valgrind errors necessarily

I had written some ad hoc leak tests in https://github.com/apache/arrow/blob/master/python/scripts/test_leak.py. We have some more of this in ARROW-3324.

It would be useful to be able to create a sort of "test suite" of memory leak checks. They are a bit too intensive to run in CI (since you may have to run something many iterations to see whether it leaks), but we could run them in a nightly build

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

• [Python] Memory leak in pyarrow functions .ipc.serialize_pandas/deserialize_pandas (is related to)
• [Python] Run memory leak tests nightly as follow up to ARROW-4120 (is related to)

PRs and other links:

• GitHub Pull Request #6551

_{Note: This issue was originally created as ARROW-4120. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
I'm implementing something very simple for ARROW-3324, but we should think about this more holistically

[asfimport]

Antoine Pitrou / @pitrou:
Is it specifically about Python reference leaks? I have written this set of leak checkers for pytest:
https://github.com/dask/distributed/blob/master/distributed/pytest_resourceleaks.py

[asfimport]

Wes McKinney / @wesm:
In the context of ARROW-6060, we had a case with runaway peak memory use. Otherwise there was no memory leak nor issue with Python references etc.

I think we should implement tests that check that peak memory use (at least according to what a MemoryPool is able to account for) does not exceed a given level. Then we could write better tests

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 6551
#6551