Skip to content

ARROW-3539: [CI/Packaging] Update scripts to build against vendored jemalloc #2779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kszucs wants to merge 2 commits into from
Closed

ARROW-3539: [CI/Packaging] Update scripts to build against vendored jemalloc #2779

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
92899bf
remove jemalloc installations
kszucs Oct 17, 2018
016575c
make build args
kszucs Oct 18, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion c_glib/Brewfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ brew "cmake"
brew "git"
brew "gobject-introspection"
brew "gtk-doc"
brew "jemalloc"
brew "libtool"
brew "lua"
brew "luarocks"
Expand Down
1 change: 0 additions & 1 deletion ci/conda_env_cpp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ flatbuffers
gflags
glog
gtest
jemalloc
libprotobuf
lz4-c
python
Expand Down
2 changes: 1 addition & 1 deletion ci/travis_install_linux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

sudo apt-get install -y -q \
gdb binutils ccache libboost-dev libboost-filesystem-dev \
libboost-system-dev libboost-regex-dev libjemalloc-dev
libboost-system-dev libboost-regex-dev

if [ "$CXX" == "g++-4.9" ]; then
sudo apt-get install -y -q g++-4.9
Expand Down
13 changes: 11 additions & 2 deletions cpp/cmake_modules/ThirdpartyToolchain.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,15 @@ if ("${MAKE}" STREQUAL "")
endif()
endif()

# Using make -j in sub-make is fragile
# see discussion https://github.com/apache/arrow/pull/2779
if (${CMAKE_GENERATOR} MATCHES "Makefiles")
set(MAKE_BUILD_ARGS "")
else()
# limit the maximum number of jobs for ninja
Copy link
Member

@kou kou Oct 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may misunderstand. This argument is used by make not ninja.

Copy link
Member Author

@kszucs kszucs Oct 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see, so the problem was using make -j instead make -jN, I'll update it.

Copy link
Member Author

@kszucs kszucs Oct 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kou No, I don't fully understand :)
My conclusion from your reasoning and patch was to not parallelize make at all, and parallelize but limit the number of jobs for ninja -j4.

BTW The build is passing with the current changeset.

Copy link
Member

@kou kou Oct 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right.

set(MAKE_BUILD_ARGS "-j4")
endif()

# ----------------------------------------------------------------------
# Find pthreads

Expand Down Expand Up @@ -736,9 +745,9 @@ if (ARROW_JEMALLOC)
CONFIGURE_COMMAND ./autogen.sh "--prefix=${JEMALLOC_PREFIX}" "--with-jemalloc-prefix=je_arrow_" "--with-private-namespace=je_arrow_private_" "--disable-tls"
${EP_LOG_OPTIONS}
BUILD_IN_SOURCE 1
BUILD_COMMAND ${MAKE} -j
BUILD_COMMAND ${MAKE} ${MAKE_BUILD_ARGS}
BUILD_BYPRODUCTS "${JEMALLOC_STATIC_LIB}" "${JEMALLOC_SHARED_LIB}"
INSTALL_COMMAND ${MAKE} -j1 install)
INSTALL_COMMAND ${MAKE} install)

# Don't use the include directory directly so that we can point to a path
# that is unique to our codebase.
Expand Down
3 changes: 1 addition & 2 deletions dev/container/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,8 @@ ENV PATH="/miniconda/bin:$PATH"
RUN conda create -y -q -n pyarrow-dev \
python=3.6 numpy six setuptools cython pandas pytest \
cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
gflags brotli jemalloc lz4-c zstd -c conda-forge \
gflags brotli lz4-c zstd -c conda-forge \
&& conda clean --all

ADD script ./script
RUN chmod u=rwx ./script/*.sh

1 change: 0 additions & 1 deletion dev/docker_common/Dockerfile.xenial.base
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ RUN wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -
gtest \
re2 \
brotli \
jemalloc \
lz4-c \
zstd \
setuptools \
Expand Down
1 change: 0 additions & 1 deletion dev/spark_integration/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ RUN conda create -y -q -c conda-forge -n pyarrow-dev \
zlib \
gflags \
brotli \
jemalloc \
lz4-c \
zstd \
setuptools \
Expand Down
1 change: 0 additions & 1 deletion dev/tasks/linux-packages/apt/debian-stretch/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ RUN \
libboost-system-dev \
libgirepository1.0-dev \
libglib2.0-doc \
libjemalloc-dev \
libprotobuf-dev \
libprotoc-dev \
lsb-release \
Expand Down
1 change: 0 additions & 1 deletion dev/tasks/linux-packages/apt/ubuntu-bionic/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ RUN \
libboost-system-dev \
libgirepository1.0-dev \
libglib2.0-doc \
libjemalloc-dev \
libprotobuf-dev \
libprotoc-dev \
lsb-release \
Expand Down
1 change: 0 additions & 1 deletion dev/tasks/linux-packages/apt/ubuntu-trusty/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ RUN \
libboost-system-dev \
libgirepository1.0-dev \
libglib2.0-doc \
libjemalloc-dev \
lsb-release \
pkg-config && \
apt-get clean && \
Expand Down
1 change: 0 additions & 1 deletion dev/tasks/linux-packages/apt/ubuntu-xenial/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ RUN \
libboost-system-dev \
libgirepository1.0-dev \
libglib2.0-doc \
libjemalloc-dev \
libprotobuf-dev \
libprotoc-dev \
lsb-release \
Expand Down
1 change: 0 additions & 1 deletion dev/tasks/linux-packages/debian.ubuntu-trusty/control
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ Build-Depends:
libboost-regex-dev,
libboost-system-dev,
libgirepository1.0-dev,
libjemalloc-dev,
pkg-config
Build-Depends-Indep: libglib2.0-doc
Standards-Version: 3.9.6
Expand Down
1 change: 0 additions & 1 deletion dev/tasks/linux-packages/debian/control
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ Build-Depends:
libboost-regex-dev,
libboost-system-dev,
libgirepository1.0-dev,
libjemalloc-dev,
libprotobuf-dev,
libprotoc-dev,
nvidia-cuda-toolkit,
Expand Down
2 changes: 0 additions & 2 deletions dev/tasks/linux-packages/yum/arrow.spec.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@ BuildRequires: gcc-c++
BuildRequires: cmake3
BuildRequires: boost-devel
BuildRequires: git
BuildRequires: jemalloc-devel
%if %{use_python}
BuildRequires: python34-devel
BuildRequires: python34-numpy
Expand Down Expand Up @@ -112,7 +111,6 @@ License: Apache-2.0
Requires: boost-system
Requires: boost-filesystem
Requires: boost-regex
Requires: jemalloc

%description libs
This package contains the libraries for Apache Arrow C++.
Expand Down
1 change: 0 additions & 1 deletion dev/tasks/linux-packages/yum/centos-6/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ RUN \
git \
gobject-introspection-devel \
gtk-doc \
jemalloc-devel \
pkg-config \
python34 \
python34-devel \
Expand Down
1 change: 0 additions & 1 deletion dev/tasks/linux-packages/yum/centos-7/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ RUN \
git \
gobject-introspection-devel \
gtk-doc \
jemalloc-devel \
pkg-config \
python34 \
python34-devel \
Expand Down
1 change: 0 additions & 1 deletion python/Brewfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,5 @@
# under the License.

brew "ccache"
brew "jemalloc"
brew "boost"
brew "thrift"