ARROW-6979: [R] Enable jemalloc in autobrew formula

[nealrichardson]

No description provided.

[nealrichardson]

@ursabot crossbow submit homebrew-cpp-autobrew

[ursabot]

AMD64 Conda Crossbow Submit (#84105) builder has been succeeded.

Revision: f569cd64ed83fab1c54aa71efaef4e8a62dc4abe

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-411

Task	Status
homebrew-cpp-autobrew

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6979

[nealrichardson]

@ursabot crossbow submit macos-r-autobrew

[ursabot]

AMD64 Conda Crossbow Submit (#84106) builder has been succeeded.

Revision: f569cd64ed83fab1c54aa71efaef4e8a62dc4abe

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-412

Task	Status
macos-r-autobrew

[nealrichardson]

This fails IIUC because while the vendored jemalloc is built, it's not included in the resulting "bottle" package:

6.15s$ brew test $ARROW_FORMULA
Testing apache-arrow
==> /usr/bin/clang++ test.cpp -std=c++11 -I/usr/local/Cellar/apache-arrow/HEAD-42a2e43/include -L/usr/local/Cellar/apache-arrow/HEAD-42a2e43/lib -larrow -lparquet -lthrift -llz4 -lboost_system -lboost_regex -lsnappy -o test
Last 15 lines from /Users/travis/Library/Logs/Homebrew/apache-arrow/test.01.clang++:
      arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::JemallocAllocator>::Reallocate(long long, long long, unsigned char**) in libarrow.a(memory_pool.cc.o)
      arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::JemallocAllocator>::Free(unsigned char*, long long) in libarrow.a(memory_pool.cc.o)
  "_je_arrow_free", referenced from:
      arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::SystemAllocator>::Reallocate(long long, long long, unsigned char**) in libarrow.a(memory_pool.cc.o)
      arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::SystemAllocator>::Free(unsigned char*, long long) in libarrow.a(memory_pool.cc.o)
  "_je_arrow_mallctl", referenced from:
      arrow::jemalloc_set_decay_ms(int) in libarrow.a(memory_pool.cc.o)
  "_je_arrow_mallocx", referenced from:
      arrow::(anonymous namespace)::JemallocAllocator::AllocateAligned(long long, unsigned char**) in libarrow.a(memory_pool.cc.o)
  "_je_arrow_posix_memalign", referenced from:
      arrow::(anonymous namespace)::SystemAllocator::AllocateAligned(long long, unsigned char**) in libarrow.a(memory_pool.cc.o)
  "_je_arrow_rallocx", referenced from:
      arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::JemallocAllocator>::Reallocate(long long, long long, unsigned char**) in libarrow.a(memory_pool.cc.o)
ld: symbol(s) not found for architecture x86_64

What's the right way to get jemalloc.a to be included--somehow copy it into the install directory alongside libarrow.a et al.?

[wesm]

This is tricky. It seems we need to INSTALL (with CMake) our vendored private libjemalloc.a and arrange somehow for it to be included when linking

[wesm]

Another possibility is to somehow glue the jemalloc symbols into libarrow.a. There seems to be a way to do this using ar though might require some fiddling to get it to work both with macOS and Linux

https://stackoverflow.com/questions/37924383/combining-several-static-libraries-into-one-using-cmake

[xhochy]

Another possibility is to somehow glue the jemalloc symbols into libarrow.a. There seems to be a way to do this using ar though might require some fiddling to get it to work both with macOS and Linux

https://stackoverflow.com/questions/37924383/combining-several-static-libraries-into-one-using-cmake

I would prefer this approach as the arrow-built libjemalloc is an implementation detail and we don't want other users to depend on it.

[wesm]

I would prefer this approach as the arrow-built libjemalloc is an implementation detail and we don't want other users to depend on it.

I agree, it's probably also worth the investment of time to have a general-purpose CMake function(s) to stitch together static libraries.

[nealrichardson]

FYI #6068 (comment) sounds similar to this discussion.

In the meantime, I'll see if I can use the Homebrew jemalloc instead of the vendored.

[wesm]

In the meantime, I'll see if I can use the Homebrew jemalloc instead of the vendored.

I don't think it's possible. We have things carefully configured to use our private built-by-us jemalloc

[wesm]

I am going to have a tinker with this and see where I can get with the static library merging

[wesm]

Looks like this problem is solved in https://cristianadam.eu/20190501/bundling-together-static-libraries-with-cmake/. The blog content is CC-BY 4.0 which is not compatible with ASF projects for source inclusion. I e-mailed the author to see if they will relicense

[wesm]

I spent a little time looking at this. If we can reuse the code from that blog post then I don't think this is too bad. We have to have the following happen:

• arrow_static target yields static library libarrow_unbundled.a. Currently this is called libarrow.a
• New arrow_static_bundled target yields libarrow.a which includes jemalloc or mimalloc

Internal targets linking to arrow_static will get libarrow_unbundled.a along with its transitive dependencies, that's fine. Downstream projects can link to libarrow.a and they'll get the bundled library version

It's a little bit hacky (this will have to happen inside add_arrow_lib) but otherwise seems tractable

[wesm]

Here's a prototype that bundled jemalloc or mimalloc on Linux (I modified from the blog post linked above, so if we can't get that appropriately licensed we'll have to write a sufficiently fresh replacement for what I have here)

master...wesm:glue-jemalloc-into-libarrow

Some different commands are likely required on macOS but this is at least a POC

[wesm]

Here's the code under MIT license https://gist.github.com/cristianadam/ef920342939a89fae3e8a85ca9459b49. I'll try to get this working on macOS a bit later

[wesm]

I'll put up a separate PR today with this working on the 3 major platforms. Not sure how it will impact packaging so we will see

[nealrichardson]

Re packaging, it shouldn't affect windows because jemalloc doesn't work on windows (right?). On Linux IIUC you'll need to remove jemalloc from https://github.com/apache/arrow/blob/master/r/configure#L103

[wesm]

Since we build mimalloc on Windows I think this issue will impact static linking on Windows with -DARROW_MIMALLOC=ON