CI: Strange issue building with clang-5.0 and clang-7 - some shared objects not created

[jimklimov]

As seen in an instrumented build branch, at https://travis-ci.org/github/jimklimov/nut/builds/743841433#L2911 the clients/.libs/ does not contain the actual versioned SO-name for libnutclient.so.1.0.0 (C++), although e.g. a libupsclient.so.5.0.0 (plain C) is there:

  2309925      4 drwxrwxr-x   2 travis   travis       4096 Nov 16 01:54 ./clients/.libs
  2309718      0 lrwxrwxrwx   1 travis   travis         21 Nov 16 01:54 ./clients/.libs/libnutclient.so -> libnutclient.so.1.0.0
  2309725     60 -rwxrwxr-x   1 travis   travis      58864 Nov 16 01:54 ./clients/.libs/upsrw
  2309720      4 -rw-rw-r--   1 travis   travis        370 Nov 16 01:54 ./clients/.libs/libupsclient.ver
  2309133     76 -rwxrwxr-x   1 travis   travis      74152 Nov 16 01:54 ./clients/.libs/libupsclient.so.5.0.0
  2309938    500 -rw-rw-r--   1 travis   travis     511840 Nov 16 01:54 ./clients/.libs/nutclient.o
  2309727     60 -rwxrwxr-x   1 travis   travis      57552 Nov 16 01:54 ./clients/.libs/upsset.cgi
  2309941     88 -rw-rw-r--   1 travis   travis      88790 Nov 16 01:54 ./clients/.libs/libupsclient.a
  2309311     56 -rwxrwxr-x   1 travis   travis      55296 Nov 16 01:54 ./clients/.libs/upsimage.cgi
  2309927     32 -rw-rw-r--   1 travis   travis      29888 Nov 16 01:54 ./clients/.libs/upsclient.o
  2308899     72 -rwxrwxr-x   1 travis   travis      72920 Nov 16 01:54 ./clients/.libs/upsmon
  2309947      0 lrwxrwxrwx   1 travis   travis         18 Nov 16 01:54 ./clients/.libs/libnutclient.la -> ../libnutclient.la
  2309635     48 -rwxrwxr-x   1 travis   travis      49080 Nov 16 01:54 ./clients/.libs/upscmd
  2309944    592 -rw-rw-r--   1 travis   travis     603634 Nov 16 01:54 ./clients/.libs/libnutclient.a
  2309946      4 -rw-rw-r--   1 travis   travis       1047 Nov 16 01:54 ./clients/.libs/libnutclient.lai
  2309932      0 lrwxrwxrwx   1 travis   travis         21 Nov 16 01:54 ./clients/.libs/libupsclient.so.5 -> libupsclient.so.5.0.0
  2309936      4 -rw-rw-r--   1 travis   travis       1062 Nov 16 01:54 ./clients/.libs/libupsclient.lai
  2309934      0 lrwxrwxrwx   1 travis   travis         21 Nov 16 01:54 ./clients/.libs/libupsclient.so -> libupsclient.so.5.0.0
  2309928      4 -rw-rw-r--   1 travis   travis        327 Nov 16 01:54 ./clients/.libs/libupsclient.exp
  2309937      0 lrwxrwxrwx   1 travis   travis         18 Nov 16 01:54 ./clients/.libs/libupsclient.la -> ../libupsclient.la
  2309719     36 -rwxrwxr-x   1 travis   travis      34632 Nov 16 01:54 ./clients/.libs/upsc
  2309637     40 -rwxrwxr-x   1 travis   travis      40088 Nov 16 01:54 ./clients/.libs/upslog
  2309196      0 lrwxrwxrwx   1 travis   travis         21 Nov 16 01:54 ./clients/.libs/libnutclient.so.1 -> libnutclient.so.1.0.0
  2309717     60 -rwxrwxr-x   1 travis   travis      59136 Nov 16 01:54 ./clients/.libs/upsstats.cgi

As a consequence, recently updated unit-tests can not be built:

Making all in tests
make[1]: Entering directory '/home/travis/build/jimklimov/nut/tests'

/bin/bash ../libtool  --tag=CXX   --mode=link ccache clang++-5.0 -I../include    -I/home/travis/build/jimklimov/nut/tmp/include -std=gnu++99 -std=c++11 -I/home/travis/build/jimklimov/nut/tmp/include -std=gnu++99 -std=c++11 -lcppunit -L/home/travis/build/jimklimov/nut/tmp/lib -o cppunittest cppunittest-example.o cppunittest-nutclienttest.o cppunittest-cpputest.o ../clients/libnutclient.la -lpthread 

libtool: link: LD_RUN_PATH="/home/travis/build/jimklimov/nut/tmp/lib:" ccache clang++-5.0 -I../include -I/home/travis/build/jimklimov/nut/tmp/include -std=gnu++99 -std=c++11 -I/home/travis/build/jimklimov/nut/tmp/include -std=gnu++99 -std=c++11 -o .libs/cppunittest cppunittest-example.o cppunittest-nutclienttest.o cppunittest-cpputest.o  -lcppunit -L/home/travis/build/jimklimov/nut/tmp/lib ../clients/.libs/libnutclient.so -lpthread

clang: error: no such file or directory: '../clients/.libs/libnutclient.so'

Makefile:703: recipe for target 'cppunittest' failed
make[1]: *** [cppunittest] Error 1
make[1]: Target 'all' not remade because of errors.
make[1]: Leaving directory '/home/travis/build/jimklimov/nut/tests'
make[1]: Entering directory '/home/travis/build/jimklimov/nut'
make[1]: Nothing to be done for 'all-am'.
make[1]: Leaving directory '/home/travis/build/jimklimov/nut'
Makefile:546: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
make: Target 'all' not remade because of errors.
Command exited with non-zero status 2

Probably there is some mess-up in the Makefiles, but curiously this is the only set-up where this issue manifests ( NUT_MATRIX_TAG="gnu99-clang-5.0-nowarn-force" BUILD_TYPE=default-all-errors CFLAGS="-std=gnu99" CXXFLAGS="-std=gnu++99" CC=clang-5.0 CXX=clang++-5.0). With newer compiler and/or C standard versions, e.g. clang-8 in NUT_MATRIX_TAG="gnu17-clang-8-nowarn" BUILD_TYPE=default-all-errors CFLAGS="-std=gnu17" CXXFLAGS="-std=gnu++17" CC=clang-8 CXX=clang++-8 tests work as expected.

For now, this test case was demoted to allow_failures so it does not block testing other improvements for the effort started in #823 / #844.

[jimklimov]

For a bit more detail (and annotated to help spot the differences), in the same log https://travis-ci.org/github/jimklimov/nut/builds/743841433#L1164 the operations for libnutclient look like this:

depbase=`echo upsclient.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\

# INITIAL CMD
/bin/bash ../libtool  --tag=CC   --mode=compile \
  ccache clang-5.0 \
  -DHAVE_CONFIG_H -I. -I../include   \
  -I/home/travis/build/jimklimov/nut/tmp/include   \
  -I../include   \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu99 -Wall -Wsign-compare -MT upsclient.lo -MD -MP -MF \
  $depbase.Tpo -c -o upsclient.lo upsclient.c &&\

mv -f $depbase.Tpo $depbase.Plo

# COMPILE FOR .libs/
libtool: compile:  ccache clang-5.0 \
  -DHAVE_CONFIG_H -I. -I../include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -I../include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu99 -Wall -Wsign-compare -MT upsclient.lo -MD -MP -MF \
  .deps/upsclient.Tpo -c upsclient.c  -fPIC -DPIC -o .libs/upsclient.o

# COMPILE FOR main dir
libtool: compile:  ccache clang-5.0 \
  -DHAVE_CONFIG_H -I. -I../include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -I../include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu99 -Wall -Wsign-compare -MT upsclient.lo -MD -MP -MF \
  .deps/upsclient.Tpo -c upsclient.c -o upsclient.o >/dev/null 2>&1

# REQUEST TO LINK .la FILE
/bin/bash ../libtool  --tag=CC   --mode=link ccache clang-5.0 -I../include   \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu99 -Wall -Wsign-compare -version-info 5:0:0 -export-symbols-regex ^upscli_ \
  -L/home/travis/build/jimklimov/nut/tmp/lib \
  -o libupsclient.la \
  -rpath /home/travis/build/jimklimov/nut/tmp/lib \
  upsclient.lo ../common/libcommonclient.la -lssl -lcrypto -lpthread 

libtool: link: /usr/bin/nm -B  .libs/upsclient.o   ../common/.libs/libcommonclient.a \
  | sed -n -e 's/^.*[	 ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[	 ][	 ]*\([_A-Za-z][_A-Za-z0-9]*\)$/\1 \2 \2/p' \
  | sed '/ __gnu_lto/d' | /bin/sed 's/.* //' | sort | uniq > .libs/libupsclient.exp

libtool: link: /bin/grep -E -e "^upscli_" ".libs/libupsclient.exp" > ".libs/libupsclient.expT"
libtool: link: mv -f ".libs/libupsclient.expT" ".libs/libupsclient.exp"
libtool: link: echo "{ global:" > .libs/libupsclient.ver
libtool: link:  cat .libs/libupsclient.exp | sed -e "s/\(.*\)/\1;/" >> .libs/libupsclient.ver
libtool: link:  echo "local: *; };" >> .libs/libupsclient.ver

# ACTUAL LINK
libtool: link:  ccache clang-5.0 -shared  -fPIC -DPIC  .libs/upsclient.o  \
  -Wl,--whole-archive ../common/.libs/libcommonclient.a \
  -Wl,--no-whole-archive  -L/home/travis/build/jimklimov/nut/tmp/lib \
  -lssl -lcrypto -lpthread    \
  -Wl,-soname -Wl,libupsclient.so.5 \
  -Wl,-version-script -Wl,.libs/libupsclient.ver \
  -o .libs/libupsclient.so.5.0.0

### SYMLINKS FOR .so
libtool: link: (cd ".libs" && rm -f "libupsclient.so.5" && ln -s "libupsclient.so.5.0.0" "libupsclient.so.5")

libtool: link: (cd ".libs" && rm -f "libupsclient.so" && ln -s "libupsclient.so.5.0.0" "libupsclient.so")


### PROCESS LIBRARY .a
libtool: link: (cd .libs/libupsclient.lax/libcommonclient.a && /usr/bin/ar x "/home/travis/build/jimklimov/nut/clients/../common/.libs/libcommonclient.a")

libtool: link: /usr/bin/ar cru .libs/libupsclient.a  upsclient.o  \
  .libs/libupsclient.lax/libcommonclient.a/common.o \
  .libs/libupsclient.lax/libcommonclient.a/parseconf.o \
  .libs/libupsclient.lax/libcommonclient.a/state.o \
  .libs/libupsclient.lax/libcommonclient.a/str.o 

/usr/bin/ar: `u' modifier ignored since `D' is the default (see `U')

libtool: link: ranlib .libs/libupsclient.a

### CLEANUP
libtool: link: rm -fr .libs/libupsclient.lax
libtool: link: ( cd ".libs" && rm -f "libupsclient.la" && ln -s "../libupsclient.la" "libupsclient.la" )

...and for libupsclient just above:

depbase=`echo nutclient.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\

# INITIAL CMD
/bin/bash ../libtool  --tag=CXX   --mode=compile \
  ccache clang++-5.0 \
  -DHAVE_CONFIG_H -I. -I../include   \
  -I/home/travis/build/jimklimov/nut/tmp/include   \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu++99 -std=c++11 -MT nutclient.lo -MD -MP -MF \
  $depbase.Tpo -c -o nutclient.lo nutclient.cpp &&\

mv -f $depbase.Tpo $depbase.Plo

# COMPILE FOR .libs/
libtool: compile:  ccache clang++-5.0 \
  -DHAVE_CONFIG_H -I. -I../include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu++99 -std=c++11 -MT nutclient.lo -MD -MP -MF \
  .deps/nutclient.Tpo -c nutclient.cpp 

libtool: compile: mv -f "nutclient.o" ".libs/nutclient.o"

# COMPILE FOR main dir
libtool: compile:  ccache clang++-5.0 \
  -DHAVE_CONFIG_H -I. -I../include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu++99 -std=c++11 -MT nutclient.lo -MD -MP -MF \
  .deps/nutclient.Tpo -c nutclient.cpp >/dev/null 2>&1

# REQUEST TO LINK .la FILE
/bin/bash ../libtool  --tag=CXX   --mode=link ccache clang++-5.0  \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu++99 -std=c++11 -version-info 1:0:0 \
  -L/home/travis/build/jimklimov/nut/tmp/lib \
  -o libnutclient.la \
  -rpath /home/travis/build/jimklimov/nut/tmp/lib \
  nutclient.lo  -lpthread 

### SYMLINKS FOR .so
libtool: link: (cd ".libs" && rm -f "libnutclient.so.1" && ln -s "libnutclient.so.1.0.0" "libnutclient.so.1")
libtool: link: (cd ".libs" && rm -f "libnutclient.so" && ln -s "libnutclient.so.1.0.0" "libnutclient.so")

### PROCESS LIBRARY .a
libtool: link: /usr/bin/ar cru .libs/libnutclient.a  nutclient.o
/usr/bin/ar: `u' modifier ignored since `D' is the default (see `U')
libtool: link: ranlib .libs/libnutclient.a

### CLEANUP
libtool: link: ( cd ".libs" && rm -f "libnutclient.la" && ln -s "../libnutclient.la" "libnutclient.la" )

One thing to pop out is that C++ build dies not make the -fPIC object file separately, but just moves the one made for ordinary builds into .libs/ dir, and a lot less happens in "# REQUEST TO LINK .la FILE" and "### PROCESS LIBRARY .a" phases (in latter case, maybe just less dependency files listed for the lib?), and as the cornerstone there is no "# ACTUAL LINK" phase in C++ case.

[jimklimov]

For further comparison, here is a similar log part from builds with newer C++ standards which do end well:

depbase=`echo nutclient.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\

# INITIAL CMD
/bin/bash ../libtool  --tag=CXX   --mode=compile \
  ccache clang++-5.0 \
  -DHAVE_CONFIG_H -I. -I../include   \
  -I/home/travis/build/jimklimov/nut/tmp/include   \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu++11 -MT nutclient.lo -MD -MP -MF \
  $depbase.Tpo -c -o nutclient.lo nutclient.cpp &&\

mv -f $depbase.Tpo $depbase.Plo

# COMPILE FOR .libs/
libtool: compile:  ccache clang++-5.0 \
  -DHAVE_CONFIG_H -I. -I../include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu++11 -MT nutclient.lo -MD -MP -MF \
  .deps/nutclient.Tpo -c nutclient.cpp \
  -fPIC -DPIC -o .libs/nutclient.o

# COMPILE FOR main dir
libtool: compile:  ccache clang++-5.0 \
  -DHAVE_CONFIG_H -I. -I../include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu++11 -MT nutclient.lo -MD -MP -MF \
  .deps/nutclient.Tpo -c nutclient.cpp -o nutclient.o >/dev/null 2>&1

# REQUEST TO LINK .la FILE
/bin/bash ../libtool  --tag=CXX   --mode=link ccache clang++-5.0  \
  -I/home/travis/build/jimklimov/nut/tmp/include \
  -std=gnu++11 -version-info 1:0:0 \
  -L/home/travis/build/jimklimov/nut/tmp/lib \
  -o libnutclient.la \
  -rpath /home/travis/build/jimklimov/nut/tmp/lib \
  nutclient.lo  -lpthread 

libtool: link: ccache clang++-5.0  -fPIC -DPIC -shared -nostdlib /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/crtbeginS.o  .libs/nutclient.o   -L/home/travis/build/jimklimov/nut/tmp/lib -lpthread -L/usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../.. -L/usr/lib/llvm-5.0/bin/../lib -L/lib -L/usr/lib -lstdc++ -lm -lc -lgcc_s /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/crtendS.o /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../x86_64-linux-gnu/crtn.o    -Wl,-soname -Wl,libnutclient.so.1 -o .libs/libnutclient.so.1.0.0

### SYMLINKS FOR .so
libtool: link: (cd ".libs" && rm -f "libnutclient.so.1" && ln -s "libnutclient.so.1.0.0" "libnutclient.so.1")
libtool: link: (cd ".libs" && rm -f "libnutclient.so" && ln -s "libnutclient.so.1.0.0" "libnutclient.so")

### PROCESS LIBRARY .a
libtool: link: /usr/bin/ar cru .libs/libnutclient.a  nutclient.o
/usr/bin/ar: `u' modifier ignored since `D' is the default (see `U')
libtool: link: ranlib .libs/libnutclient.a

### CLEANUP
libtool: link: ( cd ".libs" && rm -f "libnutclient.la" && ln -s "../libnutclient.la" "libnutclient.la" )

[jimklimov]

And the winner is... probably... the -std=gnu++99 -std=c++11 in command-lines of the failing builds.

Fuller matrix can be reviewed at https://travis-ci.org/github/jimklimov/nut/builds/743850786 (don't mind the failed builds for C17 which clang-3.5 and clang-5.0 do not support, and C11 which is not for clang-3.5 either).

[jimklimov]

So, checking for C++11 support in current compiler... yes, GCC-style (as C++11) should not happen if the caller has some other C++ standard set - or mix-ups would happen.