Segfault when plasma client goes out of scope before buffer.

[asfimport]

The following causes a segfault.

 

First start a plasma store with

plasma_store -s /tmp/store -m 10000000000

Then run the following in Python.

import pyarrow.plasma as plasma
import numpy as np

client = plasma.connect('/tmp/store', '', 0)

object_id = client.put(np.zeros(3))

buf = client.get(object_id)

del client

del buf  # This segfaults.

The backtrace is 

(lldb) bt

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xfffffffffffffffc)

  * frame #0: 0x00000001056deaee libplasma.0.dylib`plasma::PlasmaClient::Release(plasma::UniqueID const&) + 142

    frame #1: 0x00000001056de9e9 libplasma.0.dylib`plasma::PlasmaBuffer::~PlasmaBuffer() + 41

    frame #2: 0x00000001056dec9f libplasma.0.dylib`arrow::Buffer::~Buffer() + 63

    frame #3: 0x0000000106206661 lib.cpython-36m-darwin.so`std::__1::shared_ptr<arrow::Buffer>::~shared_ptr() [inlined] std::__1::__shared_count::__release_shared(this=0x00000001019b7d20) at memory:3444

    frame #4: 0x0000000106206617 lib.cpython-36m-darwin.so`std::__1::shared_ptr<arrow::Buffer>::~shared_ptr() [inlined] std::__1::__shared_weak_count::__release_shared(this=0x00000001019b7d20) at memory:3486

    frame #5: 0x0000000106206617 lib.cpython-36m-darwin.so`std::__1::shared_ptr<arrow::Buffer>::~shared_ptr(this=0x0000000100791780) at memory:4412

    frame #6: 0x0000000106002b35 lib.cpython-36m-darwin.so`std::__1::shared_ptr<arrow::Buffer>::~shared_ptr(this=0x0000000100791780) at memory:4410

    frame #7: 0x00000001061052c5 lib.cpython-36m-darwin.so`void __Pyx_call_destructor<std::__1::shared_ptr<arrow::Buffer> >(x=std::__1::shared_ptr<arrow::Buffer>::element_type @ 0x00000001019b7d38 strong=0 weak=1) at lib.cxx:486

    frame #8: 0x0000000106104f93 lib.cpython-36m-darwin.so`__pyx_tp_dealloc_7pyarrow_3lib_Buffer(o=0x0000000100791768) at lib.cxx:107704

    frame #9: 0x00000001069fcd54 multiarray.cpython-36m-darwin.so`array_dealloc + 292

    frame #10: 0x00000001000e8daf libpython3.6m.dylib`_PyDict_DelItem_KnownHash + 463

    frame #11: 0x0000000100171899 libpython3.6m.dylib`_PyEval_EvalFrameDefault + 13321

    frame #12: 0x00000001001791ef libpython3.6m.dylib`_PyEval_EvalCodeWithName + 2447

    frame #13: 0x000000010016e3d4 libpython3.6m.dylib`PyEval_EvalCode + 100

    frame #14: 0x00000001001a3bd6 libpython3.6m.dylib`PyRun_InteractiveOneObject + 582

    frame #15: 0x00000001001a350e libpython3.6m.dylib`PyRun_InteractiveLoopFlags + 222

    frame #16: 0x00000001001a33fc libpython3.6m.dylib`PyRun_AnyFileExFlags + 60

    frame #17: 0x00000001001bc835 libpython3.6m.dylib`Py_Main + 3829

    frame #18: 0x0000000100000df8 python`main + 232

    frame #19: 0x00007fff6cd80015 libdyld.dylib`start + 1

    frame #20: 0x00007fff6cd80015 libdyld.dylib`start + 1

Basically, the issue is that when the buffer goes out of scope, it calls Release on the plasma client, but the client has already been deallocated.

Reporter: Robert Nishihara / @robertnishihara
Assignee: Philipp Moritz / @pcmoritz

Related issues:

• [Plasma] Build error on macOS (contains)

PRs and other links:

• GitHub Pull Request #1939

_{Note: This issue was originally created as ARROW-2448. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Yes. The issue is that we don't control PlasmaClient lifetime (the user allocates it however they want, as the constructor is public). And there's no notion of an invalidated buffer.

Also, I don't understand why the buffer still has valid contents after the client is destroyed.

[asfimport]

Robert Nishihara / @robertnishihara:
We may need to wrap the PlasmaClient in a wrapper class, which the user allocates, and which has a shared pointer to the actual PlasmaClient. Similarly, each PlasmaBuffer needs a shared pointer to the PlasmaClient. What do you think about something like that?

I think something like this could be made to work.

Not sure if I understand your question correctly, but the buffer can still point to a valid region of memory after the client is destroyed (since the store is still running).

[asfimport]

Antoine Pitrou / @pitrou:

Not sure if I understand your question correctly, but the buffer can still point to a valid region of memory after the client is destroyed (since the store is still running).

In that case, it's impossible to release that buffer's memory, right? Even when the client process dies, as long as the store is still running, the memory would still be reserved...

Similarly, each PlasmaBuffer needs a shared pointer to the PlasmaClient. What do you think about something like that?

That would probably work to fix the crash, indeed. Something like a "pimpl" pattern?

[asfimport]

Philipp Moritz / @pcmoritz:
Issue resolved by pull request 1939
#1939