From 46292458a81c5a77a1e05b8039c51a7d6dffbcc4 Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Tue, 30 Mar 2021 18:07:38 -0400 Subject: [PATCH 01/11] Updates for vcpkg and Visual Studio 2019 --- docs/source/developers/cpp/windows.rst | 69 ++++++++++++++++++++++---- 1 file changed, 60 insertions(+), 9 deletions(-) diff --git a/docs/source/developers/cpp/windows.rst b/docs/source/developers/cpp/windows.rst index fece978cacb..86b0b382860 100644 --- a/docs/source/developers/cpp/windows.rst +++ b/docs/source/developers/cpp/windows.rst @@ -28,8 +28,8 @@ System Setup ============ Microsoft provides the free Visual Studio Community edition. When doing -development in the shell, you must initialize the development -environment. +development in the shell, you must initialize the development environment +each time you open the shell. For Visual Studio 2015, execute the following batch script: @@ -43,6 +43,12 @@ For Visual Studio 2017, the script is: "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\Tools\VsDevCmd.bat" -arch=amd64 +For Visual Studio 2019, the script is: + +.. code-block:: shell + + "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\VsDevCmd.bat" -arch=amd64 + One can configure a console emulator like `cmder `_ to automatically launch this when starting a new development console. @@ -99,6 +105,34 @@ source. from ``conda-forge``. You can examine the installed packages in your environment (and their origin) with ``conda list`` +Using vcpkg for build dependencies +======================================== + +`vcpkg `_ is an open source package manager +manager from Microsoft. It hosts community-contributed ports of C and C++ +packages and their dependencies. Arrow includes a manifest file `cpp/vcpkg.json +`_ that specifies +which ``vcpkg`` packages are required to build the C++ library. + +To use ``vcpkg`` for your C++ build dependencies on Windows, first +`install `_ and +`integrate `_ +``vcpkg``. Then change working directory in ``cmd.exe`` to the root directory +of Arrow and run the command: + +.. code-block:: shell + + vcpkg install ^ + --triplet x64-windows ^ + --x-manifest-root cpp ^ + --clean-after-build + +On Windows, ``vcpkg`` builds dynamic link libraries by default. +Use the triplet ``x64-windows-static`` to build static libraries. + +``vcpkg`` downloads source packages and compiles them locally, so installing +packages with ``vcpkg`` is more time-consuming than with ``conda``. + Building using Visual Studio (MSVC) Solution Files ================================================== @@ -110,20 +144,31 @@ an out of source build by generating a MSVC solution: cd cpp mkdir build cd build - cmake .. -G "Visual Studio 14 2015 Win64" ^ + cmake .. -G "Visual Studio 14 2015" -A x64 ^ + -DARROW_DEPENDENCY_SOURCE=VCPKG ^ + -DARROW_DEPENDENCY_USE_SHARED=ON ^ -DARROW_BUILD_TESTS=ON cmake --build . --config Release +For newer versions of Visual Studio, specify the generator +``Visual Studio 15 2017`` or ``Visual Studio 16 2019``. If you built static +libraries with ``vcpkg``, specify ``-DARROW_DEPENDENCY_USE_SHARED=OFF``. + Building with Ninja and clcache =============================== -The `Ninja `_ build system offsets better build +The `Ninja `_ build system offers better build parallelization, and the optional `clcache -`_ compiler cache which keeps track of +`_ compiler cache keeps track of past compilations to avoid running them over and over again (in a way similar to the Unix-specific ``ccache``). -Activate your conda build environment to first install those utilities: +Newer versions of Visual Studio include Ninja. To see if your Visual Studio +includes Ninja, run the initialization command shown above (``vcvarsall.bat`` +or ``VsDevCmd.bat``), then run ``ninja --version``. + +If Ninja is not included in your version of Visual Studio, and you are using +conda, activate your conda environment and install Ninja and clcache: .. code-block:: shell @@ -131,7 +176,13 @@ Activate your conda build environment to first install those utilities: conda install -c conda-forge ninja pip install git+https://github.com/frerich/clcache.git -Change working directory in ``cmd.exe`` to the root directory of Arrow and +If you are not using conda, +`install Ninja from another source `_ +and optionally +`install clcache from another source `_ +. + +After installation is complete, change working directory in ``cmd.exe`` to the root directory of Arrow and do an out of source build by generating Ninja files: .. code-block:: shell @@ -214,7 +265,7 @@ Debug builds ============ To build a Debug version of Arrow, you should have pre-installed a Debug -version of Boost. It's recommended to configure cmake build with the +version of Boost. It's recommended to configure ``cmake build`` with the following variables for Debug build: * ``-DARROW_BOOST_USE_SHARED=OFF``: enables static linking with boost debug @@ -229,7 +280,7 @@ The command line to build Arrow in Debug mode will look something like this: cd cpp mkdir build cd build - cmake .. -G "Visual Studio 14 2015 Win64" ^ + cmake .. -G "Visual Studio 14 2015" -A x64 ^ -DARROW_BOOST_USE_SHARED=OFF ^ -DCMAKE_BUILD_TYPE=Debug ^ -DBOOST_ROOT=C:/local/boost_1_63_0 ^ From 308c0ded3a8450052339080123538a9cd2b6a3dc Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Wed, 31 Mar 2021 11:32:06 -0400 Subject: [PATCH 02/11] Reorganize and add more content --- docs/source/developers/cpp/windows.rst | 43 +++++++++++++++++--------- 1 file changed, 29 insertions(+), 14 deletions(-) diff --git a/docs/source/developers/cpp/windows.rst b/docs/source/developers/cpp/windows.rst index 86b0b382860..a1783f7ecd9 100644 --- a/docs/source/developers/cpp/windows.rst +++ b/docs/source/developers/cpp/windows.rst @@ -112,26 +112,44 @@ Using vcpkg for build dependencies manager from Microsoft. It hosts community-contributed ports of C and C++ packages and their dependencies. Arrow includes a manifest file `cpp/vcpkg.json `_ that specifies -which ``vcpkg`` packages are required to build the C++ library. +which vcpkg packages are required to build the C++ library. -To use ``vcpkg`` for your C++ build dependencies on Windows, first +To use vcpkg for C++ build dependencies on Windows, first `install `_ and `integrate `_ -``vcpkg``. Then change working directory in ``cmd.exe`` to the root directory +vcpkg. Then change working directory in ``cmd.exe`` to the root directory of Arrow and run the command: .. code-block:: shell vcpkg install ^ - --triplet x64-windows ^ - --x-manifest-root cpp ^ - --clean-after-build + --triplet x64-windows ^ + --x-manifest-root cpp ^ + --clean-after-build -On Windows, ``vcpkg`` builds dynamic link libraries by default. -Use the triplet ``x64-windows-static`` to build static libraries. +On Windows, vcpkg builds dynamic link libraries by default. Use the triplet +``x64-windows-static`` to build static libraries. vcpkg downloads source +packages and compiles them locally, so installing dependencies with vcpkg is +more time-consuming than with conda. + +Then in your ``cmake`` command, to use dependencies installed by vcpkg, set: + +.. code-block:: shell + + -DARROW_DEPENDENCY_SOURCE=VCPKG + +You can optionally set other variables to override the default CMake +configurations for vcpkg, including: + +* ``-DCMAKE_TOOLCHAIN_FILE``: sets the path to ``vcpkg.cmake`` +* ``-DVCPKG_TARGET_TRIPLET``: sets the vcpkg + `triplet `_ +* ``-DARROW_DEPENDENCY_USE_SHARED``: default is ``ON``; set to ``OFF`` for + static libraries +* ``-DVCPKG_MANIFEST_MODE``: default is ``ON``; set to ``OFF`` to ignore the + ``vcpkg.json`` manifest file and only look for vcpkg packages that are + already installed under the directory where vcpkg is installed -``vcpkg`` downloads source packages and compiles them locally, so installing -packages with ``vcpkg`` is more time-consuming than with ``conda``. Building using Visual Studio (MSVC) Solution Files ================================================== @@ -145,14 +163,11 @@ an out of source build by generating a MSVC solution: mkdir build cd build cmake .. -G "Visual Studio 14 2015" -A x64 ^ - -DARROW_DEPENDENCY_SOURCE=VCPKG ^ - -DARROW_DEPENDENCY_USE_SHARED=ON ^ -DARROW_BUILD_TESTS=ON cmake --build . --config Release For newer versions of Visual Studio, specify the generator -``Visual Studio 15 2017`` or ``Visual Studio 16 2019``. If you built static -libraries with ``vcpkg``, specify ``-DARROW_DEPENDENCY_USE_SHARED=OFF``. +``Visual Studio 15 2017`` or ``Visual Studio 16 2019``. Building with Ninja and clcache =============================== From d3e0f25b771a24df0b2e7583fc604aa90007832d Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Wed, 31 Mar 2021 11:33:57 -0400 Subject: [PATCH 03/11] Minor formatting fixes in building.rst --- docs/source/developers/cpp/building.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/developers/cpp/building.rst b/docs/source/developers/cpp/building.rst index 00a97042963..48369e5e3aa 100644 --- a/docs/source/developers/cpp/building.rst +++ b/docs/source/developers/cpp/building.rst @@ -61,7 +61,7 @@ On Alpine Linux: gcc \ make -On macOS, you can use `Homebrew `_. +On macOS, you can use `Homebrew `_: .. code-block:: shell @@ -276,7 +276,7 @@ The build system supports a number of third-party dependencies The CMake option ``ARROW_DEPENDENCY_SOURCE`` is a global option that instructs the build system how to resolve each dependency. There are a few options: -* ``AUTO``: try to find package in the system default locations and build from +* ``AUTO``: Try to find package in the system default locations and build from source if not found * ``BUNDLED``: Building the dependency automatically from source * ``SYSTEM``: Finding the dependency in system paths using CMake's built-in From 1d05ce31fd616f798db4449986f394bfba893015 Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Wed, 31 Mar 2021 11:34:35 -0400 Subject: [PATCH 04/11] Mention vcpkg in other places in building.rst --- docs/source/developers/cpp/building.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/source/developers/cpp/building.rst b/docs/source/developers/cpp/building.rst index 48369e5e3aa..3592798cb56 100644 --- a/docs/source/developers/cpp/building.rst +++ b/docs/source/developers/cpp/building.rst @@ -69,6 +69,16 @@ On macOS, you can use `Homebrew `_: cd arrow brew update && brew bundle --file=cpp/Brewfile +With `vcpkg `_: + +.. code-block:: shell + + git clone https://github.com/apache/arrow.git + cd arrow + vcpkg install \ + --x-manifest-root cpp \ + --clean-after-build + On MSYS2: .. code-block:: shell @@ -241,7 +251,7 @@ LLVM and Clang Tools We are currently using LLVM 8 for library builds and for other developer tools such as code formatting with ``clang-format``. LLVM can be installed via most -modern package managers (apt, yum, conda, Homebrew, chocolatey). +modern package managers (apt, yum, conda, Homebrew, vcpkg, chocolatey). .. _cpp-build-dependency-management: From 822549f0a9f8d6406f50b83ae1146c1e258be140 Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Wed, 31 Mar 2021 11:34:59 -0400 Subject: [PATCH 05/11] Improve description of ARROW_DEPENDENCY_SOURCE=VCPKG --- docs/source/developers/cpp/building.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/developers/cpp/building.rst b/docs/source/developers/cpp/building.rst index 3592798cb56..dfaee0ff7c0 100644 --- a/docs/source/developers/cpp/building.rst +++ b/docs/source/developers/cpp/building.rst @@ -293,7 +293,8 @@ the build system how to resolve each dependency. There are a few options: ``find_package`` function, or using ``pkg-config`` for packages that do not have this feature * ``CONDA``: Use ``$CONDA_PREFIX`` as alternative ``SYSTEM`` PATH -* ``VCPKG``: Find dependencies installed by ``vcpkg`` +* ``VCPKG``: Find dependencies installed by vcpkg, and if not found, run + ``vpckg install`` to install them * ``BREW``: Use Homebrew default paths as an alternative ``SYSTEM`` path The default method is ``AUTO`` unless you are developing within an active conda From 3851ad943590dab49e2b0681605c731e402838f7 Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Wed, 31 Mar 2021 11:56:24 -0400 Subject: [PATCH 06/11] Describe how to open VS IDE with conda environment activated --- docs/source/developers/cpp/windows.rst | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/source/developers/cpp/windows.rst b/docs/source/developers/cpp/windows.rst index a1783f7ecd9..4e83fe1b61e 100644 --- a/docs/source/developers/cpp/windows.rst +++ b/docs/source/developers/cpp/windows.rst @@ -66,7 +66,8 @@ download and install a 64-bit distribution from the `Miniconda homepage `_ To configure ``conda`` to use the ``conda-forge`` channel by default, launch a -command prompt (``cmd.exe``) and run the command: +command prompt (``cmd.exe``), run the initialization command shown above +(``vcvarsall.bat`` or ``VsDevCmd.bat``), and run the command: .. code-block:: shell @@ -94,9 +95,16 @@ resolving the build dependencies. This is equivalent to setting -DARROW_DEPENDENCY_SOURCE=SYSTEM ^ -DARROW_PACKAGE_PREFIX=%CONDA_PREFIX%\Library -Note that these packages are only supported for release builds. If you intend -to use ``-DCMAKE_BUILD_TYPE=debug`` then you must build the packages from -source. +To use the Visual Studio IDE with this conda environment activated, launch it by +running the command: + +.. code-block:: shell + + devenv + +Note that these conda packages are only supported for release builds. If you +intend to use ``-DCMAKE_BUILD_TYPE=debug`` then you must build the packages +from source. .. note:: From b25f5b85525b62066b2086e94353c857855a58fd Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Wed, 31 Mar 2021 12:00:13 -0400 Subject: [PATCH 07/11] Trigger CI From 9ad6bdeb1ef1e7ea1159ab3bb038daeeb78872c5 Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Thu, 1 Apr 2021 15:04:44 -0400 Subject: [PATCH 08/11] Link to initialization commands --- docs/source/developers/cpp/windows.rst | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/source/developers/cpp/windows.rst b/docs/source/developers/cpp/windows.rst index 4e83fe1b61e..1ed16ad278d 100644 --- a/docs/source/developers/cpp/windows.rst +++ b/docs/source/developers/cpp/windows.rst @@ -24,6 +24,8 @@ Developing on Windows Like Linux and macOS, we have worked to enable builds to work "out of the box" with CMake for a reasonably large subset of the project. +.. _windows-system-setup: + System Setup ============ @@ -66,8 +68,9 @@ download and install a 64-bit distribution from the `Miniconda homepage `_ To configure ``conda`` to use the ``conda-forge`` channel by default, launch a -command prompt (``cmd.exe``), run the initialization command shown above -(``vcvarsall.bat`` or ``VsDevCmd.bat``), and run the command: +command prompt (``cmd.exe``), run the initialization command shown +:ref:`above` (``vcvarsall.bat`` or ``VsDevCmd.bat``), then +run the command: .. code-block:: shell @@ -187,8 +190,9 @@ past compilations to avoid running them over and over again (in a way similar to the Unix-specific ``ccache``). Newer versions of Visual Studio include Ninja. To see if your Visual Studio -includes Ninja, run the initialization command shown above (``vcvarsall.bat`` -or ``VsDevCmd.bat``), then run ``ninja --version``. +includes Ninja, run the initialization command shown +:ref:`above` (``vcvarsall.bat`` or ``VsDevCmd.bat``), then + run ``ninja --version``. If Ninja is not included in your version of Visual Studio, and you are using conda, activate your conda environment and install Ninja and clcache: From 945642a8b6bc59c7b384ef17fefe2af76229de3e Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Thu, 1 Apr 2021 15:04:59 -0400 Subject: [PATCH 09/11] Fix errors --- docs/source/developers/cpp/windows.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/developers/cpp/windows.rst b/docs/source/developers/cpp/windows.rst index 1ed16ad278d..e07866f9d59 100644 --- a/docs/source/developers/cpp/windows.rst +++ b/docs/source/developers/cpp/windows.rst @@ -120,8 +120,8 @@ Using vcpkg for build dependencies ======================================== `vcpkg `_ is an open source package manager -manager from Microsoft. It hosts community-contributed ports of C and C++ -packages and their dependencies. Arrow includes a manifest file `cpp/vcpkg.json +from Microsoft. It hosts community-contributed ports of C and C++ packages and +their dependencies. Arrow includes a manifest file `cpp/vcpkg.json `_ that specifies which vcpkg packages are required to build the C++ library. @@ -292,8 +292,8 @@ Debug builds ============ To build a Debug version of Arrow, you should have pre-installed a Debug -version of Boost. It's recommended to configure ``cmake build`` with the -following variables for Debug build: +version of Boost. It's recommended to configure ``cmake`` with the following +variables for Debug build: * ``-DARROW_BOOST_USE_SHARED=OFF``: enables static linking with boost debug libs and simplifies run-time loading of 3rd parties From 67324c11849e0aa02e0bfb22c630705acf67a562 Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Tue, 6 Apr 2021 15:43:09 -0400 Subject: [PATCH 10/11] Improve windows.rst --- docs/source/developers/cpp/windows.rst | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/docs/source/developers/cpp/windows.rst b/docs/source/developers/cpp/windows.rst index e07866f9d59..297348d4f51 100644 --- a/docs/source/developers/cpp/windows.rst +++ b/docs/source/developers/cpp/windows.rst @@ -99,15 +99,13 @@ resolving the build dependencies. This is equivalent to setting -DARROW_PACKAGE_PREFIX=%CONDA_PREFIX%\Library To use the Visual Studio IDE with this conda environment activated, launch it by -running the command: +running the command ``devenv`` from the same command prompt. -.. code-block:: shell - - devenv - -Note that these conda packages are only supported for release builds. If you -intend to use ``-DCMAKE_BUILD_TYPE=debug`` then you must build the packages -from source. +Note that dependencies installed as conda packages are built in release mode and +cannot link with debug builds. If you intend to use ``-DCMAKE_BUILD_TYPE=debug`` +then you must build the packages from source. +``-DCMAKE_BUILD_TYPE=relwithdebinfo`` is also available, which produces a build +that can both be linked with release libraries and be debugged. .. note:: From 92e6f8ae05d82e1553944e25e0d0a6f24bce1859 Mon Sep 17 00:00:00 2001 From: Ian Cook Date: Wed, 7 Apr 2021 15:51:24 -0400 Subject: [PATCH 11/11] Improve descriptions of CMake configs for vcpkg --- docs/source/developers/cpp/windows.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/source/developers/cpp/windows.rst b/docs/source/developers/cpp/windows.rst index 297348d4f51..8df443d097a 100644 --- a/docs/source/developers/cpp/windows.rst +++ b/docs/source/developers/cpp/windows.rst @@ -150,9 +150,13 @@ Then in your ``cmake`` command, to use dependencies installed by vcpkg, set: You can optionally set other variables to override the default CMake configurations for vcpkg, including: -* ``-DCMAKE_TOOLCHAIN_FILE``: sets the path to ``vcpkg.cmake`` -* ``-DVCPKG_TARGET_TRIPLET``: sets the vcpkg - `triplet `_ +* ``-DCMAKE_TOOLCHAIN_FILE``: by default, the CMake scripts automatically find + the location of the vcpkg CMake toolchain file ``vcpkg.cmake``; use this to + instead specify its location +* ``-DVCPKG_TARGET_TRIPLET``: by default, the CMake scripts attempt to infer the + vcpkg + `triplet `_; + use this to instead specify the triplet * ``-DARROW_DEPENDENCY_USE_SHARED``: default is ``ON``; set to ``OFF`` for static libraries * ``-DVCPKG_MANIFEST_MODE``: default is ``ON``; set to ``OFF`` to ignore the