gitian: GLIBC_2.29 not found error at bitcoind startup on bionic

[willyko]

bitcoind & bitcoin-qt built from gitian on master branch have not been working on Bionic for the last month or so.
The bitcoind works on Focal though.
Following @jonasschnelli 's nightly gitian build, the last working bitcoind binary was dated January 28 on commit d1ddead

Expected behavior
bitcoind to run

Actual behavior

$ ./bitcoind
bitcoind: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by bitcoind)
bitcoind: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by bitcoind)

To reproduce
On Ubuntu Bionic, download any of the bitcoin-*-x86_64-linux-gnu.tar.gz after January 28th and run the bitcoind

System information
The binaries are downloaded from https://bitcoin.jonasschnelli.ch/gitian/?show=nightly#nighly
I ran the gitian build myself and hit the same error too.
The binaries work on Focal but not on Bionic. Did not test on other Ubuntu releases

[maflcko]

6a726cb is from the 28th

[maflcko]

guix is also affected

[hebasto]

FWIW,

$ lsb_release -d
Description:	Ubuntu 18.04.5 LTS
$ ldd --version | head -1
ldd (Ubuntu GLIBC 2.27-3ubuntu1.4) 2.27

[hebasto]

6a726cb is from the 28th

Just tested the gitian build of 6a726cb on bionic.
It works flawlessly.

[sipa]

@jonasschnelli How are your nightlies built? Do they use gitian?

[willyko]

There were no nightly build from January 29 - February 10, and it looks like some builds were failing due to No space left on device error according to the log. I was able to gitian build and run on bionic all the way up to c969ab4 and that's the commit right before the switch to focal for gitian vm

[hebasto]

I can confirm that issue is caused by ca85449 (#21036 merging) -- gitian builds work fine before this commit.

[fanquake]

The offending symbols are:

0000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  log
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  pow
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.29  exp
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.28  fcntl64

[hebasto]

The offending symbols are:

Should they be substituted in glibc_compat.cpp?

[fanquake]

I was confused as to how this could have happened, because our gitian/guix builds (symbol checks) all should have failed in #21036 if there were newer-than-allowed symbols being introduced. The issue seems to be that Pixie is incorrectly returning None, rather than a symbol version quite often. i.e with:

index 436f179d6..831346c5b 100755
--- a/contrib/devtools/symbol-check.py
+++ b/contrib/devtools/symbol-check.py
@@ -174,6 +174,7 @@ def check_imported_symbols(filename) -> bool:
             continue
         sym = symbol.name.decode()
         version = symbol.version.decode() if symbol.version is not None else None
+        print(sym, version)
         if version and not check_version(MAX_VERSIONS, version, elf.hdr.e_machine):
             print('{}: symbol {} from unsupported version {}'.format(filename, cppfilt(sym), version))
             ok = False

I'm seeing this:

<trim>
fcntl64 None
memmove None
pthread_rwlock_unlock None
realloc None
__fprintf_chk None
fseeko64 None
pthread_once None
_Unwind_GetRegionStart None
__pthread_key_create None
getgrgid None
fchown None
__tls_get_addr GLIBC_2.3
__memcpy_chk None
_Exit None
open64 None
freeifaddrs None
pow None
_ITM_deregisterTMCloneTable None
strpbrk None
pthread_getschedparam None
_Unwind_Resume None
timerfd_create None
getauxval None
getprotobynumber None
strcat None
mmap None
umask None
setsockopt None
getpgid None
exp None
_Unwind_SetIP None
<trim>

when running python3 bitcoin/contrib/devtools/symbol-check.py bitcoin-ee0dc02c6f93/bin/bitcoind. Which is against a recent nightly build downloaded from bitcoin builds.