ARROW-5631: [C++] Fix FindBoost targets with cmake3.2 #4605

fsaintjacques · 2019-06-18T16:29:52Z

cmake's FindBoost pre 3.5 did not export IMPORTED targets. This also fixes the newly introduced trusty docker image.

fsaintjacques · 2019-06-18T16:30:55Z

cpp/cmake_modules/ThirdpartyToolchain.cmake

I'm not sure why we need to stash BUILD_SHARED_LIBS, if someones wants to clarify, I'll gladly add a comment to the PR (@xhochy, @wesm).

This is needed to work with BoostConfig.cmake provided by Boost (not CMake).
The Boost package provided by MSYS2 installed BoostConfig.cmake at the time.
But the current Boost package doesn't install BoostConfig.cmake: https://github.com/msys2/MINGW-packages/pull/5326/files

See also: msys2/MINGW-packages#5233

There is no platform that needs this. So we can remove this for now but we may need this again in the future.

fsaintjacques · 2019-06-18T17:23:37Z

A note that this patch does not fully fix the cmake3.2 issue, see https://jira.apache.org/jira/browse/ARROW-5631?filter=-1

But at least it gets a cmake build directory populated.

wesm · 2019-06-18T17:26:05Z

Looks like @kou introduced that logic here

f014d76

I'm not sure what it's for as we don't use this flag other places in our build system. Some projects use the flag to globally toggle between building SHARED/STATIC when using add_library but we are explicit about this everywhere.

fsaintjacques · 2019-06-18T17:54:40Z

@kou afaik, find_package shouldn't be affected by BUILD_SHARED_LIBS?

wesm · 2019-06-18T19:30:05Z

I think it's fine to keep the stashing; it seems like it was more likely future proofing

fsaintjacques · 2019-06-18T19:31:48Z

Don't merge for now, it doesn't fix the docker cmake32 image.

fsaintjacques · 2019-06-20T19:39:03Z

rebased with master, ready to land (once it passes tests). This passes locally, minus the linking error fixed by #4635.

kou · 2019-06-20T21:54:58Z

It seems that the current changes don't fix anything.
Should we really fix this problem or update title and description of this pull request?

fsaintjacques · 2019-06-20T21:57:58Z

@kou previously cmake didn't even succeed at generating a build folder. The ctest failures are fixed in the PR linked in previous comment (#4635).

codecov-io · 2019-06-21T01:03:52Z

Codecov Report

Merging #4605 into master will increase coverage by 0.41%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4605      +/-   ##
==========================================
+ Coverage   88.49%   88.91%   +0.41%     
==========================================
  Files         890      710     -180     
  Lines      110406    96682   -13724     
  Branches     1418        0    -1418     
==========================================
- Hits        97703    85960   -11743     
+ Misses      12422    10722    -1700     
+ Partials      281        0     -281

Impacted Files	Coverage Δ
go/arrow/ipc/writer.go
go/arrow/math/uint64_amd64.go
go/arrow/memory/memory_avx2_amd64.go
go/arrow/ipc/file_reader.go
js/src/builder/index.ts
js/src/enum.ts
go/arrow/array/builder.go
js/src/util/fn.ts
js/src/Arrow.node.ts
js/src/schema.ts
... and 170 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0e1fbb...891407e. Read the comment docs.

fsaintjacques · 2019-06-21T13:50:20Z

@ursabot crossbox --help

ursabot · 2019-06-21T13:50:21Z

No such command "crossbox".

fsaintjacques · 2019-06-21T13:50:30Z

@ursabot crossbow --help

ursabot · 2019-06-21T13:50:31Z

Usage: @ursabot crossbow [OPTIONS] COMMAND [ARGS]...

  Trigger crossbow builds for this pull request

Options:
  --help  Show this message and exit.

Commands:
  package  Submit crossbow packaging tasks.
  test     Submit crossbow testing tasks.

fsaintjacques · 2019-06-21T13:54:53Z

@ursabot crossbow test

ursabot · 2019-06-21T13:55:05Z

AMD64 Conda Crossbow (#23233) builder failed.

Revision: f097a4b4ea5ae527a6cf8bfc7a927e5dbca5616b

Crossbow: 'python crossbow.py ...' step's stderr:

Traceback (most recent call last):
  File "crossbow.py", line 924, in <module>
    crossbow(obj={}, auto_envvar_prefix='CROSSBOW')
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "crossbow.py", line 751, in submit
    job = Job(target=target, tasks=tasks)
  File "crossbow.py", line 574, in __init__
    raise ValueError('no tasks were provided for the job')
ValueError: no tasks were provided for the job

fsaintjacques · 2019-06-21T13:56:15Z

@ursabot crossbow test -g docker

ursabot · 2019-06-21T13:56:32Z

AMD64 Conda Crossbow (#23234) builder has been succeeded.

Revision: f097a4b4ea5ae527a6cf8bfc7a927e5dbca5616b

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-41

Task	Status
docker-java
docker-js
docker-python-3.6-alpine
docker-python-3.6-nopandas
docker-rust
docker-python-3.7
docker-cpp-alpine
docker-go
docker-docs
docker-python-2.7-alpine
docker-cpp-cmake32
docker-python-2.7-nopandas
docker-clang-format
docker-r
docker-lint
docker-dask-integration
docker-python-2.7
docker-hdfs-integration
docker-turbodbc-integration
docker-python-3.6
docker-pandas-master
docker-iwyu
docker-cpp
docker-spark-integration
docker-c_glib

fsaintjacques · 2019-06-21T14:59:26Z

@ursabot crossbow test docker docker-cpp docker-cpp-cmake32 docker-rust

ursabot · 2019-06-21T14:59:38Z

AMD64 Conda Crossbow (#23279) builder failed.

Revision: 0e11383c1589ac73b2a8abb2c158038a1477de40

Crossbow: 'python crossbow.py ...' step's stderr:

Error: Invalid task(s) {'docker'}. Must be one of {'docker-clang-format', 'docker-rust', 'docker-dask-integration', 'docker-python-3.6-alpine', 'docker-lint', 'docker-python-2.7-alpine', 'docker-python-3.6-nopandas', 'docker-python-2.7-nopandas', 'docker-c_glib', 'docker-python-2.7', 'docker-python-3.6', 'docker-pandas-master', 'docker-js', 'docker-java', 'docker-turbodbc-integration', 'docker-hdfs-integration', 'docker-cpp-alpine', 'docker-iwyu', 'docker-python-3.7', 'docker-cpp', 'docker-cpp-cmake32', 'docker-go', 'docker-spark-integration', 'docker-r', 'docker-docs'}

fsaintjacques · 2019-06-21T15:00:27Z

@ursabot crossbow test docker-cpp docker-cpp-cmake32 docker-rust

ursabot · 2019-06-21T15:00:42Z

AMD64 Conda Crossbow (#23280) builder has been succeeded.

Revision: 0e11383c1589ac73b2a8abb2c158038a1477de40

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-42

Task	Status
docker-cpp
docker-cpp-cmake32
docker-rust

fsaintjacques · 2019-06-21T15:02:40Z

@ursabot crossbow test docker-c_glib

ursabot · 2019-06-21T15:02:59Z

AMD64 Conda Crossbow (#23290) builder has been succeeded.

Revision: 0e11383c1589ac73b2a8abb2c158038a1477de40

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-43

Task	Status
docker-c_glib

fsaintjacques · 2019-06-21T15:12:38Z

@kszucs do you know why the badge is at failed? The last one are waiting to be run.

fsaintjacques · 2019-06-21T16:12:32Z

@kou could you check locally with this patch why c_glib docker image fails? I've exausted my debugging of meson. I think that you need to namespace gandiva's enums.c.

gandiva-glib/meson.build:50:0: ERROR:  Tried to create target "enums.c", but a target of that name already exists.

I'll let you decide if renaming breaks anything.

fsaintjacques · 2019-06-21T16:13:50Z

@ursabot crossbow test docker-c_glib

ursabot · 2019-06-21T16:17:18Z

AMD64 Conda Crossbow (#23312) builder has been succeeded.

Revision: dbf6fe7d9d813e1179d697c7eeab0b31900257b1

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-44

Task	Status
docker-c_glib

fsaintjacques · 2019-06-21T18:28:45Z

@ursabot crossbow test docker-r

ursabot · 2019-06-21T18:29:03Z

AMD64 Conda Crossbow (#23410) builder has been succeeded.

Revision: 46d5da0b3b08be423333e17e7206a824f437162e

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-45

Task	Status
docker-r

fsaintjacques · 2019-06-21T19:09:08Z

@ursabot crossbow test -g docker

ursabot · 2019-06-21T19:09:26Z

AMD64 Conda Crossbow (#23414) builder has been succeeded.

Revision: 46d5da0b3b08be423333e17e7206a824f437162e

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-46

Task	Status
docker-iwyu
docker-dask-integration
docker-hdfs-integration
docker-python-3.6
docker-cpp-cmake32
docker-spark-integration
docker-c_glib
docker-r
docker-rust
docker-cpp
docker-python-3.6-alpine
docker-python-2.7-nopandas
docker-clang-format
docker-go
docker-pandas-master
docker-python-3.7
docker-lint
docker-python-2.7
docker-js
docker-python-2.7-alpine
docker-java
docker-turbodbc-integration
docker-python-3.6-nopandas
docker-docs
docker-cpp-alpine

conda's cmake3.2 does not support https for third party download.

- Ensure that LD_LIBRARY_PATH is set before installing R packages because conda's pkg-config would pickup libicu from conda at compile but LD_LIBRARY_PATH was not exported yet, thus package would fail tests. - Fix JSON payload on systems where locales are missing.

fsaintjacques · 2019-06-21T20:06:09Z

This fixes the most pressing issues, notably

docker-cpp
docker-cmake32

I fixed also fixed docker-rust, and partly advanced on docker-r and docker-c_glib.

fsaintjacques · 2019-06-21T20:06:26Z

@ursabot crossbow test docker-cpp docker-cpp-cmake32

ursabot · 2019-06-21T20:06:41Z

AMD64 Conda Crossbow (#23443) builder has been succeeded.

Revision: 01f67dd

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-47

Task	Status
docker-cpp
docker-cpp-cmake32

kou

gandiva-glib/meson.build:50:0: ERROR:  Tried to create target "enums.c", but a target of that name already exists.

This is a Meson bug. It has been fixed in the latest Meson. But the latest Meson (0.51.0) has a environment variable related bug: mesonbuild/meson#5503 mesonbuild/meson#5502

So we need to use 0.50.1 for now.

kou · 2019-06-21T20:57:16Z

r/Dockerfile


 # r-base includes tzdata. Get around interactive stop in that package
 ENV DEBIAN_FRONTEND=noninteractive
+# workaround for install_github Github API rate limit


Github -> GitHub

kou · 2019-06-21T21:01:32Z

c_glib/Dockerfile

    ARROW_INSTALL_NAME_RPATH=OFF \
    LD_LIBRARY_PATH="${CONDA_PREFIX}/lib" \
-    PKG_CONFIG_PATH="${CONDA_PREFIX}/lib/pkgconfig" \
+    PKG_CONFIG_PATH="${CONDA_PREFIX}/lib/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig" \


Is this needed?
The added path is included in system pkg-config path. So it'll be searched without this.

I'm going to revert this for now

kou · 2019-06-21T21:07:40Z

BTW, I couldn't find any build log on CircleCI.

https://circleci.com/gh/ursa-labs/crossbow/tree/ursabot-46-circle-docker-c_glib shows

And "build" link shows

wesm

+1. Made a couple of tweaks. Merging -- please open follow up JIRAs about other docker failures

wesm · 2019-06-21T22:02:44Z

c_glib/Dockerfile

    ARROW_INSTALL_NAME_RPATH=OFF \
    LD_LIBRARY_PATH="${CONDA_PREFIX}/lib" \
-    PKG_CONFIG_PATH="${CONDA_PREFIX}/lib/pkgconfig" \
+    PKG_CONFIG_PATH="${CONDA_PREFIX}/lib/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig" \


I'm going to revert this for now

wesm · 2019-06-21T22:03:31Z

r/Dockerfile


 # r-base includes tzdata. Get around interactive stop in that package
 ENV DEBIAN_FRONTEND=noninteractive
+# workaround for install_github Github API rate limit


fsaintjacques commented Jun 18, 2019

View reviewed changes

fsaintjacques changed the title ~~ARROW-5631: [C++] Fix FindBoost targets with cmake3.2~~ [WIP] ARROW-5631: [C++] Fix FindBoost targets with cmake3.2 Jun 18, 2019

fsaintjacques force-pushed the ARROW-5631-cmake32-boost branch from c4d7d34 to 033a944 Compare June 19, 2019 15:12

fsaintjacques changed the title ~~[WIP] ARROW-5631: [C++] Fix FindBoost targets with cmake3.2~~ ARROW-5631: [C++] Fix FindBoost targets with cmake3.2 Jun 19, 2019

fsaintjacques force-pushed the ARROW-5631-cmake32-boost branch from 033a944 to 8a89596 Compare June 20, 2019 19:37

fsaintjacques force-pushed the ARROW-5631-cmake32-boost branch from f097a4b to 0e11383 Compare June 21, 2019 14:55

fsaintjacques mentioned this pull request Jun 21, 2019

ARROW-5653: [C++] Ensure testing libraries are not installed #4635

Closed

fsaintjacques force-pushed the ARROW-5631-cmake32-boost branch from 0b1a568 to dbf6fe7 Compare June 21, 2019 16:13

fsaintjacques added 7 commits June 21, 2019 15:13

Fix trusty docker build

78e41cc

Add custom docker and vendor boost

bd85741

conda's cmake3.2 does not support https for third party download.

Remove debug docker command

7544850

ARROW-5653: [CI] Fix conda libraries

4b1c29f

Fix rust docker image

3b70ff3

Fix c_glib PKG_CONFIG_PATH in docker

6c63207

Fix R docker image

01f67dd

- Ensure that LD_LIBRARY_PATH is set before installing R packages because conda's pkg-config would pickup libicu from conda at compile but LD_LIBRARY_PATH was not exported yet, thus package would fail tests. - Fix JSON payload on systems where locales are missing.

fsaintjacques force-pushed the ARROW-5631-cmake32-boost branch from 46d5da0 to 01f67dd Compare June 21, 2019 19:14

kou reviewed Jun 21, 2019

View reviewed changes

Code review feedback

9b946da

wesm approved these changes Jun 21, 2019

View reviewed changes

wesm closed this in 00db153 Jun 21, 2019

asfimport mentioned this pull request Jun 25, 2019

[C++] CMake 3.2 build is broken #22069

Closed

ARROW-5631: [C++] Fix FindBoost targets with cmake3.2 #4605

ARROW-5631: [C++] Fix FindBoost targets with cmake3.2 #4605

Uh oh!

Conversation

fsaintjacques commented Jun 18, 2019

Uh oh!

fsaintjacques Jun 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou Jun 18, 2019

Choose a reason for hiding this comment

Uh oh!

fsaintjacques commented Jun 18, 2019

Uh oh!

wesm commented Jun 18, 2019

Uh oh!

fsaintjacques commented Jun 18, 2019

Uh oh!

wesm commented Jun 18, 2019

Uh oh!

fsaintjacques commented Jun 18, 2019

Uh oh!

fsaintjacques commented Jun 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Jun 20, 2019

Uh oh!

fsaintjacques commented Jun 20, 2019

Uh oh!

codecov-io commented Jun 21, 2019

Codecov Report

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

ursabot commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

ursabot commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

ursabot commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

ursabot commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

ursabot commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

ursabot commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

ursabot commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

ursabot commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

ursabot commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

ursabot commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

Uh oh!

fsaintjacques commented Jun 21, 2019

fsaintjacques Jun 18, 2019 •

edited

Loading

fsaintjacques commented Jun 20, 2019 •

edited

Loading