[C++] Enable Arrow to share an application's jemalloc instance

[asfimport]

Is there any good way for an application that uses Arrow to tell Arrow to share its own jemalloc instance?

Normally when Arrow uses jemalloc, it creates its own instance of it. The only method that I think would work to get Arrow to use the same jemalloc instance as the application would be to do something hacky like this:

• Patch cpp/src/arrow/memory_pool.cc to use the application's jemalloc header

• Configure the application's jemalloc instance to use the same configuration that Arrow uses, as shown in je_arrow_malloc_conf

• Patch ThirdpartyToolchain.cmake to use the application's jemalloc instance

  Is there any non-hacky way to achieve this? If not, would it be possible to add a feature to Arrow to enable the user to tell it to use a specific jemalloc instance at runtime?

  The benefit of this would be that a unified jemalloc instance could (at least hypothetically) allocate memory more efficiently than two separate ones running on the same machine.

Reporter: Ian Cook / @ianmcook
Assignee: Kouhei Sutou / @kou

Related issues:

• [Packaging] Add support for Coan C/C++ package manager (relates to)

PRs and other links:

• GitHub Pull Request #13373

_{Note: This issue was originally created as ARROW-16717. Please see the migration documentation for further details.}

[asfimport]

Kouhei Sutou / @kou:
Does "an application's jemalloc instance" mean that system jemalloc?

[asfimport]

Ian Cook / @ianmcook:
In my case, yes it is the system jemalloc. I apologize, I am not very knowledgeable about memory allocators

[asfimport]

Kouhei Sutou / @kou:
No problem. :-)
I was willing to implement the feature for Conan package.
I'll work on this.

[asfimport]

Antoine Pitrou / @pitrou:
Can it happen at compile-time or does it have to be a runtime setting?

[asfimport]

Kouhei Sutou / @kou:
Compile-time.
I'll add support for choosing system jemalloc in ThirdpartyToolchain.cmake.

[asfimport]

Ian Cook / @ianmcook:
Compile time is fine. Thanks Kou!

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 13373
#13373