Unable to build `workerd` on Ubuntu 22.04.1

[jonasclaes]

Expected outcome

workerd should build fine using bazel.

Current outcome

When I try to compile workerd, it fails with the following error message:

jonas@jonas-asus:~/repos/clones/cloudflare/workerd$ bazel build -c opt //src/workerd/server:workerd
INFO: Analyzed target //src/workerd/server:workerd (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/jonas/.cache/bazel/_bazel_jonas/c86f39fb6ae3f61b12a0ae374033648e/external/capnp-cpp/src/kj/BUILD.bazel:5:11: Compiling src/kj/source-location.c++ failed: undeclared inclusion(s) in rule '@capnp-cpp//src/kj:kj':
this rule is missing dependency declarations for the following files included by 'src/kj/source-location.c++':
  '/usr/lib/llvm-14/include/c++/v1/initializer_list'
  '/usr/lib/llvm-14/include/c++/v1/__config'
  '/usr/lib/llvm-14/include/c++/v1/__config_site'
  '/usr/lib/llvm-14/include/c++/v1/cstddef'
  '/usr/lib/llvm-14/include/c++/v1/version'
  '/usr/lib/llvm-14/include/c++/v1/__nullptr'
  '/usr/lib/llvm-14/include/c++/v1/stddef.h'
  '/usr/lib/llvm-14/include/c++/v1/cstring'
  '/usr/lib/llvm-14/include/c++/v1/string.h'
Target //src/workerd/server:workerd failed to build
INFO: Elapsed time: 1.298s, Critical Path: 0.52s
INFO: 18 processes: 17 internal, 1 linux-sandbox.
FAILED: Build did NOT complete successfully

Tries

I have tried the following:

• Install clang-14 using apt
• Install libc++-dev using apt
• Install libc++abi-dev using apt

OS info

jonas@jonas-asus:~/repos/clones/cloudflare/workerd$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

[kentonv]

This is some sort of bazel bug where it gets confused about built-in headers. Can you try adding this flag to bazel build:

--action_env=CXX=$(which clang++)

This tells Bazel the full path to clang, which seems to help it understand better where to expect built-in includes to live.

[kentonv]

Hmm I'm also a little confused as the GitHub Action I defined in #8 uses Ubuntu 22.04 and the build managed to succeed there?

[Erisa]

This is some sort of bazel bug where it gets confused about built-in headers. Can you try adding this flag to bazel build:

--action_env=CXX=$(which clang++)

This tells Bazel the full path to clang, which seems to help it understand better where to expect built-in includes to live.

Wasn't me that opened this issue but I was having the same (or similar) issue on Ubuntu 22.04 and this doesn't help:

erisa@Tuturu:~/workerd$ bazel build -c opt //src/workerd/server:workerd --action_env=CXX=$(which clang++)
Starting local Bazel server and connecting to it...
INFO: Analyzed target //src/workerd/server:workerd (199 packages loaded, 16161 targets configured).
INFO: Found 1 target...
ERROR: /home/erisa/.cache/bazel/_bazel_erisa/75b9d3e535bdb9de178afa4e37c61de2/external/capnp-cpp/src/kj/BUILD.bazel:5:11: Compiling src/kj/list.c++ failed: undeclared inclusion(s) in rule '@capnp-cpp//src/kj:kj':
this rule is missing dependency declarations for the following files included by 'src/kj/list.c++':
  '/usr/lib/llvm-14/include/c++/v1/initializer_list'
  '/usr/lib/llvm-14/include/c++/v1/stddef.h'
  '/usr/lib/llvm-14/include/c++/v1/__config'
  '/usr/lib/llvm-14/include/c++/v1/__config_site'
  '/usr/lib/llvm-14/include/c++/v1/__nullptr'
  '/usr/lib/llvm-14/include/c++/v1/cstring'
  '/usr/lib/llvm-14/include/c++/v1/string.h'
  '/usr/lib/llvm-14/include/c++/v1/cstddef'
  '/usr/lib/llvm-14/include/c++/v1/version'
ERROR: /home/erisa/.cache/bazel/_bazel_erisa/75b9d3e535bdb9de178afa4e37c61de2/external/capnp-cpp/src/kj/BUILD.bazel:5:11: Compiling src/kj/string-tree.c++ failed: undeclared inclusion(s) in rule '@capnp-cpp//src/kj:kj':
this rule is missing dependency declarations for the following files included by 'src/kj/string-tree.c++':
  '/usr/lib/llvm-14/include/c++/v1/initializer_list'
  '/usr/lib/llvm-14/include/c++/v1/__config'
  '/usr/lib/llvm-14/include/c++/v1/__config_site'
  '/usr/lib/llvm-14/include/c++/v1/cstddef'
  '/usr/lib/llvm-14/include/c++/v1/version'
  '/usr/lib/llvm-14/include/c++/v1/__nullptr'
  '/usr/lib/llvm-14/include/c++/v1/stddef.h'
  '/usr/lib/llvm-14/include/c++/v1/cstring'
  '/usr/lib/llvm-14/include/c++/v1/string.h'
Target //src/workerd/server:workerd failed to build
INFO: Elapsed time: 4.912s, Critical Path: 0.82s
INFO: 35 processes: 32 internal, 2 linux-sandbox, 1 local.
FAILED: Build did NOT complete successfully

Though clang++ is a binary that exists, and the files it's referring to in the error also appear to exist, which is the confusing part.

I also tried installing libc++abi-dev (Which wasn't installed) alongside the other deps (Which were installed) as noted in #8 (comment), but that doesn't seem to have made a difference.

The C++ file(s) that it cites in the error seem to change across runs, likely just whichever ones it gets to first.

[jonasclaes]

This is some sort of bazel bug where it gets confused about built-in headers. Can you try adding this flag to bazel build:

--action_env=CXX=$(which clang++)

This tells Bazel the full path to clang, which seems to help it understand better where to expect built-in includes to live.

I have just tried this, but to no avail. I have added the output below. Could this be an LLVM 14 version issue?

jonas@jonas-asus:~/repos/clones/cloudflare/workerd$ bazel build --action_env=CXX=$(which clang++) -c opt //src/workerd/server:workerd
INFO: Analyzed target //src/workerd/server:workerd (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/jonas/.cache/bazel/_bazel_jonas/c86f39fb6ae3f61b12a0ae374033648e/external/capnp-cpp/src/kj/BUILD.bazel:5:11: Compiling src/kj/memory.c++ failed: undeclared inclusion(s) in rule '@capnp-cpp//src/kj:kj':
this rule is missing dependency declarations for the following files included by 'src/kj/memory.c++':
  '/usr/lib/llvm-14/include/c++/v1/initializer_list'
  '/usr/lib/llvm-14/include/c++/v1/stddef.h'
  '/usr/lib/llvm-14/include/c++/v1/__config'
  '/usr/lib/llvm-14/include/c++/v1/__config_site'
  '/usr/lib/llvm-14/include/c++/v1/__nullptr'
  '/usr/lib/llvm-14/include/c++/v1/cstring'
  '/usr/lib/llvm-14/include/c++/v1/string.h'
  '/usr/lib/llvm-14/include/c++/v1/cstddef'
  '/usr/lib/llvm-14/include/c++/v1/version'
Target //src/workerd/server:workerd failed to build
INFO: Elapsed time: 0.873s, Critical Path: 0.33s
INFO: 18 processes: 18 internal.
FAILED: Build did NOT complete successfully

Showing the versions of the libraries:

jonas@jonas-asus:~/repos/clones/cloudflare/workerd$ apt-cache show libllvm14
Package: libllvm14
Architecture: amd64
Version: 1:14.0.0-1ubuntu1
Multi-Arch: same
Priority: optional
Section: libs
Source: llvm-toolchain-14
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 106918
Depends: libc6 (>= 2.34), libedit2 (>= 2.11-20080614-0), libffi8 (>= 3.4), libgcc-s1 (>= 3.3), libstdc++6 (>= 12), libtinfo6 (>= 6), libxml2 (>= 2.7.4), zlib1g (>= 1:1.2.0)
Filename: pool/main/l/llvm-toolchain-14/libllvm14_14.0.0-1ubuntu1_amd64.deb
Size: 24009152
MD5sum: e4b3510371f248e57c1c6e44fcefcb36
SHA1: cc9fffefcc76c4af648a9634868926e8052388a4
SHA256: f22411c833909b3e4fdfd2f182974302d70c114a799499def6f005c7a2d07854
SHA512: 00b890240a896200e098dbb22d1329e7a54b0552729e0ed38afedfcfbc7f541925798f5891c8125abc393fcee038b01968884c80455d293e3010b2fce13b869e
Homepage: https://www.llvm.org/
Description-en: Modular compiler and toolchain technologies, runtime library
 LLVM is a collection of libraries and tools that make it easy to build
 compilers, optimizers, just-in-time code generators, and many other
 compiler-related programs.
 .
 This package contains the LLVM runtime library.
Description-md5: a8854f4edf76a8c14ce34dc209bd1d11
Task: postgresql-server

jonas@jonas-asus:~/repos/clones/cloudflare/workerd$ apt-cache show clang
Package: clang
Architecture: amd64
Version: 1:14.0-55~exp2
Priority: optional
Section: universe/devel
Source: llvm-defaults (0.55~exp2)
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 25
Depends: clang-14 (>= 14~)
Breaks: clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1)
Replaces: clang (<< 3.2-1~exp2), clang-3.2, clang-3.3, clang-3.4 (<< 1:3.4.2-7~exp1), clang-3.5 (<< 1:3.5~+rc1-3~exp1)
Filename: pool/universe/l/llvm-defaults/clang_14.0-55~exp2_amd64.deb
Size: 3558
MD5sum: 9012efb7d550940265a49a8b3b986b1d
SHA1: a27e72f7f3e0d3ba8a84cf9e5612849ef11a46d9
SHA256: 4b4c7dee18823b5a1348d5496806810cb8fd40289377003adee6511c44339466
SHA512: 0d7b949d9bf5ebf7c3b3eb5eb5a542e1e88fc65106297ee34ebce055e950f71f06c3470e494fbf3b5c2128771b3c1cc33dead514bde19bb9aaf35b51162bcb13
Description-en: C, C++ and Objective-C compiler (LLVM based), clang binary
 Clang project is a C, C++, Objective C and Objective C++ front-end
 for the LLVM compiler. Its goal is to offer a replacement to the GNU Compiler
 Collection (GCC).
 .
 Clang implements all of the ISO C++ 1998, 11 and 14 standards and also
 provides most of the support of C++17.
 .
 This is a dependency package providing the default clang compiler.
Description-md5: 2916bca3b5e11836b745c64f4b346b34

jonas@jonas-asus:~/repos/clones/cloudflare/workerd$ apt-cache show libc++-dev
Package: libc++-dev
Architecture: amd64
Version: 1:14.0-55~exp2
Multi-Arch: same
Priority: extra
Section: universe/libdevel
Source: llvm-defaults (0.55~exp2)
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 16
Depends: libc++-14-dev (>= 14~)
Filename: pool/universe/l/llvm-defaults/libc++-dev_14.0-55~exp2_amd64.deb
Size: 3184
MD5sum: 9c88d7342fd54ae327505a12791fee2f
SHA1: caf79f60ec7fc5d0bff56c9bb25372a0eb07e90b
SHA256: 2de8b8c689056325b4691b6ead1aa112666edf71b65f0ded69f471bf2f7c2d34
SHA512: 48c310b9dd9009e8772c44944a2d23211094c9b87c051428d5cd318e236a9edf440b00b6904de05be556b12de683e726e2c96905ff7744a681e677339f180784
Description-en: LLVM C++ Standard library (development files)
 libc++ is another implementation of the C++ standard library
 .
 Features and Goals
 .
   * Correctness as defined by the C++ standards.
   * Fast execution.
   * Minimal memory use.
   * Fast compile times.
   * ABI compatibility with gcc's libstdc++ for some low-level features such
     as exception objects, rtti and memory allocation.
   * Extensive unit tests.
 This is a dependency package providing the default LLVM C++ Standard library
 development files.
Description-md5: d6ddd828f191a901b236b63f6c33ebd7

jonas@jonas-asus:~/repos/clones/cloudflare/workerd$ apt-cache show libc++abi-dev
Package: libc++abi-dev
Architecture: amd64
Version: 1:14.0-55~exp2
Multi-Arch: same
Priority: extra
Section: universe/libdevel
Source: llvm-defaults (0.55~exp2)
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: LLVM Packaging Team <pkg-llvm-team@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 19
Depends: libc++abi-14-dev (>= 14~)
Filename: pool/universe/l/llvm-defaults/libc++abi-dev_14.0-55~exp2_amd64.deb
Size: 3320
MD5sum: 311dc971342d2015e555428b4887d125
SHA1: d47b21f3dbc8c4064919209b2810412d3297dc62
SHA256: 8a327d997c55ac16b8ae44a33de26319f7f8810bf1cc2dae28b6304e61865506
SHA512: e9409c516d4f2c7533e49f141d0fe8119cbbeb2a32b32b9d7cc714aaf5319f052faf7ca39581d0637d2dd044a1e4d3ccee5c14c3208f27f74fd59bbff1633e22
Description-en: LLVM low level support for a standard C++ library (development files)
 libc++abi is another implementation of low level support for a standard C++
 library.
 .
 Features and Goals
 .
   * Correctness as defined by the C++ standards.
   * Provide a portable sublayer to ease the porting of libc++
 This is the development package providing low level support to
 LLVM C++ Standard library.
Description-md5: 170a0b79ec2e8808672bcd34549e6625

[dom96]

I recall running into similar things, IIRC the answer in https://stackoverflow.com/questions/53527134/bazel-missing-dependency-declarations-for-stddef-h-and-others-ubuntu-18-04 helped, i.e. try running bazel clean --expunge and see if bazel build -c opt //src/workerd/server:workerd then works.

[Erisa]

I recall running into similar things, IIRC the answer in https://stackoverflow.com/questions/53527134/bazel-missing-dependency-declarations-for-stddef-h-and-others-ubuntu-18-04 helped, i.e. try running bazel clean --expunge and see if bazel build -c opt //src/workerd/server:workerd then works.

Thanks for this! Running the clean fixed it for me. It's now building.

[kentonv]

OK, so what's the best way for us to help people avoid this going forward? Some README note or is there more we can do?

[jonasclaes]

@kentonv I have created PR #59 which adds this little step to the documentation. It should resolve most of the cases of this specific issue.

[jonasclaes]

This has been merged with PR #59. Thank all!