Add nix derivation for static builds by saschagrunert · Pull Request #366 · containers/crun

saschagrunert · 2020-05-19T16:21:28Z

Switching from the current static build to a nix based derivation.

Fixes #364

saschagrunert · 2020-05-20T08:48:31Z

The latest version at least builds the static binary:

> nix-build nix
…
/nix/store/5y45a98d8lc00inhhp960jy96kariw5j-crun-static

> ldd result/bin/crun
        not a dynamic executable

> ./result/bin/crun --version
crun version 0.13
commit: 0.13
spec: 1.0.0
+SELINUX +APPARMOR +CAP +EBPF +YAJL

hswong3i · 2020-05-20T09:10:20Z

@saschagrunert when I download the original https://github.com/containers/crun/releases/download/0.13/crun-0.13-static-x86_64 it should look like:

hswong3i@hswong3i-XPS-13-7390:/tmp$ ./crun-0.13-static-x86_64 --version
crun version 0.13
commit: e79e4de4ac16da0ce48777afb72c6241de870525
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

giuseppe · 2020-05-20T09:19:14Z

I think libcap should also be static. I've tried with this version:

{ system ? builtins.currentSystem }:
let
  pkgs = (import ./nixpkgs.nix {
    config = {
      packageOverrides = pkg: {
        systemd = pkg.systemd.overrideAttrs(x: {
          postInstall = ''
            ${x.postInstall}
            ninja libsystemd.a
            mv libsystemd.a $lib/lib/libsystemd.a
          '';
          postFixup = ''
            ${x.postFixup}
            sed -ri "s;$out/(.*);$nukedRef/\1;g" $lib/lib/libsystemd.a
            rm -f $lib/lib/libsystemd.so*
	    '';
        });
      };
    };
  });

  self = with pkgs; {
    crun-static = (crun.overrideAttrs(x: {
      name = "crun-static";
      src = ./..;
      configureFlags = [ "--without-shared" "--disable-shared" ];
      nativeBuildInputs = [ autoreconfHook pkgconfig python3 ];
      buildInputs = x.buildInputs ++ [ glibc glibc.static];
      preConfigure = ''
        export LDFLAGS="-static-libgcc -static"
        export CRUN_LDFLAGS="-all-static"
        export LIBS="${glibc.static}/lib/librt.a ${glibc.static}/lib/libpthread.a ${yajl}/lib/libyajl_s.a"
      '';
    })).override {
      yajl = yajl.overrideAttrs(x: {
        buildInputs = [ glibc glibc.static ];
        preConfigure = ''
          export CMAKE_STATIC_LINKER_FLAGS="-static"
        '';
      });
      libcap = libcap.overrideAttrs(x: {
        configureFlags = [ "--without-shared" "--disable-shared" ];
        enableStatic = true;
        postInstall = ''
          mkdir -p "$doc/share/doc/${x.pname}-${x.version}"
          cp License "$doc/share/doc/${x.pname}-${x.version}/"
          mkdir -p "$pam/lib/security"
          mv "$lib"/lib/security "$pam/lib"
        '';
      });
      libseccomp = libseccomp.overrideAttrs(x: {
        configureFlags = [ "--without-shared" "--disable-shared" ];
        dontDisableStatic = true;
        enableSharedExecutables = false;
        enableStatic = true;
      });
    };
  };
in self

and I get:

$ file /nix/store/izr0kyx30jlqg5dswvn916yj5fsmjfa4-crun-static/bin/crun
/nix/store/izr0kyx30jlqg5dswvn916yj5fsmjfa4-crun-static/bin/crun: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, not stripped

It is still missing libseccomp.

Also, could we move it under contrib/ or is there any requirement from nix build to find the files under nix/?

hswong3i · 2020-05-20T09:40:21Z

Also, could we move it under contrib/ or is there any requirement from nix build to find the files under nix/?

I thinks if it locate under contrib/nix then above line src = ./..; for crun-static need to update accordingly?

People from CRI-O and Podman seems putting /nix and the build procedures into the CI plan, in order to generate the static build binary for release page (please correct me). If we officially support static build looks like /nix should be good enough?

giuseppe · 2020-05-20T09:50:19Z

People from CRI-O and Podman seems putting /nix and the build procedures into the CI plan, in order to generate the static build binary for release page (please correct me). If we officially support static build looks like /nix should be good enough?

Good point, If CRI-O and Podman are already doing it, let's leave nix/

saschagrunert · 2020-05-20T10:24:37Z

We're getting closer to an actual solution. The current version is able to build, but I'm wondering why it does not pickup the static libseccomp.a:

> ./result/bin/crun --version
crun version 0.13
commit: 0.13
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +EBPF +YAJL

I also had to skip the tests (via doCheck = false;) due to some segmentation faults:

make  check-TESTS
make[4]: Entering directory '/build/crun/libocispec'
make[5]: Entering directory '/build/crun/libocispec'
./build-aux/test-driver: line 107:  7708 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
./build-aux/test-driver: line 107:  7701 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/test-3
./build-aux/test-driver: line 107:  7714 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/test-1
FAIL: tests/test-4
./build-aux/test-driver: line 107:  7711 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/test-2
./build-aux/test-driver: line 107:  7716 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
./build-aux/test-driver: line 107:  7717 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/test-5
FAIL: tests/test-6
./build-aux/test-driver: line 107:  7719 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/test-7
./build-aux/test-driver: line 107:  7720 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/test-8

I think that's not good ☝️

Since we're using glibc, it will also complain about some runtime deps which are still needed when linking statically:

/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: /nix/store/20q6zlvf7dabbiyn5jl2qn77qqby7yvn-systemd-245.5-lib/lib/libsystemd.a(src_basic_user-util.c.o): in function `get_group_creds':
(.text.get_group_creds+0x116): warning: Using 'getgrgid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: (.text.get_group_creds+0x17c): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: /nix/store/20q6zlvf7dabbiyn5jl2qn77qqby7yvn-systemd-245.5-lib/lib/libsystemd.a(src_basic_user-util.c.o): in function `gid_to_name':
(.text.gid_to_name+0xac): warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: /nix/store/20q6zlvf7dabbiyn5jl2qn77qqby7yvn-systemd-245.5-lib/lib/libsystemd.a(src_basic_user-util.c.o): in function `get_user_creds':
(.text.get_user_creds+0x244): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: (.text.get_user_creds+0x96): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: ./.libs/libcrun.a(libcrun_la-utils.o): in function `getsubidrange':
/build/crun/src/libcrun/utils.c:1187: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: /nix/store/20q6zlvf7dabbiyn5jl2qn77qqby7yvn-systemd-245.5-lib/lib/libsystemd.a(src_basic_user-util.c.o): in function `get_group_creds':
(.text.get_group_creds+0x116): warning: Using 'getgrgid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: (.text.get_group_creds+0x17c): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: /nix/store/20q6zlvf7dabbiyn5jl2qn77qqby7yvn-systemd-245.5-lib/lib/libsystemd.a(src_basic_user-util.c.o): in function `gid_to_name':
(.text.gid_to_name+0xac): warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: /nix/store/20q6zlvf7dabbiyn5jl2qn77qqby7yvn-systemd-245.5-lib/lib/libsystemd.a(src_basic_user-util.c.o): in function `get_user_creds':
(.text.get_user_creds+0x244): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: (.text.get_user_creds+0x96): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: ./.libs/libcrun.a(libcrun_la-utils.o): in function `getsubidrange':
/build/crun/src/libcrun/utils.c:1187: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/a57856fs4m8ir6vlv14h3gq3sv9aq2lb-binutils-2.31.1/bin/ld: /nix/store/20q6zlvf7dabbiyn5jl2qn77qqby7yvn-systemd-245.5-lib/lib/libsystemd.a(src_libsystemd_sd-bus_sd-bus.c.o): in function `bus_start_address':
(.text.bus_start_address+0xa17): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

Beside that, the libocispec is complaining that it still tries to be linked dynamically:

*** Warning: Linking the shared library libocispec.la against the
*** static library /nix/store/f1bk23abhim7fsqwv2g1n2z7gsga8rd5-glibc-2.30-static/lib/libc.a is not portable!

*** Warning: Linking the shared library libocispec.la against the
*** static library /nix/store/f1bk23abhim7fsqwv2g1n2z7gsga8rd5-glibc-2.30-static/lib/libpthread.a is not portable!

*** Warning: Linking the shared library libocispec.la against the
*** static library /nix/store/f1bk23abhim7fsqwv2g1n2z7gsga8rd5-glibc-2.30-static/lib/librt.a is not portable!

*** Warning: Linking the shared library libocispec.la against the
*** static library /nix/store/vl1va1ny6g2l5q7d7l4aa9gdq5g4r9a3-libcap-2.27-lib/lib/libcap.a is not portable!

*** Warning: Linking the shared library libocispec.la against the
*** static library /nix/store/iflpf2pwrmn8ar1fdgbqsczlakf6gizc-libseccomp-2.4.3-lib/lib/libseccomp.a is not portable!

*** Warning: Linking the shared library libocispec.la against the
*** static library /nix/store/qimsaw2s5bxn1q5km8xxkignhgnc0zmq-yajl-2.1.0/lib/libyajl_s.a is not portable!

saschagrunert · 2020-05-20T10:27:08Z

One solution could be to switch to musl libc. This would result in a much higher build time for sure.

hswong3i · 2020-05-20T10:31:49Z

Since we're using glibc, it will also complain about some runtime deps which are still needed when linking statically:

Maybe with --enable-static-nss (see https://stackoverflow.com/questions/2725255/create-statically-linked-binary-that-uses-getaddrinfo#comment40629454_3087067, also see https://sourceware.org/glibc/wiki/FAQ#Even_statically_linked_programs_need_some_shared_libraries_which_is_not_acceptable_for_me.__What_can_I_do.3F)

saschagrunert · 2020-05-20T10:39:11Z

Since we're using glibc, it will also complain about some runtime deps which are still needed when linking statically:

Maybe with --enable-static-nss (see https://stackoverflow.com/questions/2725255/create-statically-linked-binary-that-uses-getaddrinfo#comment40629454_3087067)

It should not hurt to add that. 👍

giuseppe · 2020-05-20T10:43:41Z

We're getting closer to an actual solution. The current version is able to build, but I'm wondering why it does not pickup the static libseccomp.a:

I think the issue is with using pkg-config for seccomp.

If I revert 3310cd2 then it works fine.

@hswong3i could we rewrite 3310cd2 to not use pkg-config?

Is it still needed now that we have a static build? :-)

hswong3i · 2020-05-20T10:55:04Z

We're getting closer to an actual solution. The current version is able to build, but I'm wondering why it does not pickup the static libseccomp.a:

I think the issue is with using pkg-config for seccomp.

If I revert 3310cd2 then it works fine.

@hswong3i could we rewrite 3310cd2 to not use pkg-config?

Is it still needed now that we have a static build? :-)

Iet’s revert it, if nix build works and crun master won’t works for openSUSE 15.1/15.2 due to systemd 234 < 237, we should have no active reason to keep it ;-P

saschagrunert · 2020-05-20T11:33:25Z

Yes the PKG config seems to fail in the configure script, not sure what the exact reason is because usually it should find the right libs. In any case, we now have support for criu as well:

> ./result/bin/crun --version
crun version 0.13
commit: 0.13
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +EBPF +CRIU +YAJL

giuseppe · 2020-05-20T11:51:08Z

Yes the PKG config seems to fail in the configure script, not sure what the exact reason is because usually it should find the right libs. In any case, we now have support for criu as well:

I think it is better if we revert the patch and not use pkg-config. It is not needed anymore now that we have a proper static build :-)

saschagrunert · 2020-05-20T11:52:13Z

Yes the PKG config seems to fail in the configure script, not sure what the exact reason is because usually it should find the right libs. In any case, we now have support for criu as well:

I think it is better if we revert the patch and not use pkg-config. It is not needed anymore now that we have a proper static build :-)

I opened the PR for the revert in #368

hswong3i · 2020-05-20T11:55:59Z

P.S. I will create an Ansible Role for install nix with root, therefore update role for crun building static binary with nix; will also PR for .travis.yml so we could include nix build by ourselves independently.

giuseppe · 2020-05-20T12:12:45Z

P.S. I will create an Ansible Role for install nix with root, therefore update role for crun building static binary with nix; will also PR for .travis.yml so we could include nix build by ourselves independently.

we are currently creating a static binary with .github/workflows/release.yaml so that we have an artifact for each PR. Does it work fine for you? We could adapt the workflow there to use the new builder

saschagrunert · 2020-05-20T12:38:21Z

P.S. I will create an Ansible Role for install nix with root, therefore update role for crun building static binary with nix; will also PR for .travis.yml so we could include nix build by ourselves independently.

we are currently creating a static binary with .github/workflows/release.yaml so that we have an artifact for each PR. Does it work fine for you? We could adapt the workflow there to use the new builder

We probably should push the static builder somewhere since it contains dependencies which take some time to compile. I'll change the Dockerfile and Makefile accordingly to the nix build.

saschagrunert · 2020-05-20T12:50:42Z

Alright, ready for review. Something like make -C contrib/static-builder-x86_64 should now work. @giuseppe do we want to push the image somewhere (quay.io) that we can re-use it for the GitHub actions?

giuseppe · 2020-05-20T14:36:49Z

thanks, it LGTM.

Could you just tweak the build-aux/release.sh script as part of the commit to use ./crun instead of static-build/crun? That should fix the artifact failure.

@giuseppe do we want to push the image somewhere (quay.io) that we can re-use it for the GitHub actions?

yes, I was planning on doing it (https://quay.io/organization/crun) but I never started working on it :-)

rhatdan · 2020-05-20T14:38:58Z

We have quay.io/containers
If you want to place it there.

Signed-off-by: Sascha Grunert <sgrunert@suse.com>

saschagrunert · 2020-05-20T14:44:14Z

 make -C contrib/static-builder-x86_64 build-image RUNTIME=$RUNTIME
 make -C contrib/static-builder-x86_64 build-crun CRUN_SOURCE=$(pwd) RUNTIME=$RUNTIME

-strip static-build/crun


the nix build process already strips

saschagrunert · 2020-05-20T14:56:30Z

I thought the build will take forever but now I'm kinda impressed that we managed to build the binary in roughly 11 minutes.

giuseppe · 2020-05-20T14:58:29Z

thanks, all green now!

:)

giuseppe · 2020-05-20T15:02:54Z

I thought the build will take forever but now I'm kinda impressed that we managed to build the binary in roughly 11 minutes.

yes, 11 minutes is not too bad. I think it is always faster than Travis

saschagrunert changed the title ~~Add nix derivation for static builds~~ WIP: Add nix derivation for static builds May 19, 2020

saschagrunert commented May 19, 2020

View reviewed changes

Comment thread nix/default.nix Outdated

saschagrunert force-pushed the nix branch from a0e524b to fe6feb3 Compare May 20, 2020 08:45

saschagrunert force-pushed the nix branch from fe6feb3 to 3bcd71f Compare May 20, 2020 10:20

saschagrunert force-pushed the nix branch from 3bcd71f to 38e4347 Compare May 20, 2020 10:39

saschagrunert force-pushed the nix branch from 38e4347 to 3a33ac0 Compare May 20, 2020 11:33

saschagrunert mentioned this pull request May 20, 2020

Revert " [openSUSE 15.1] Missing libseccomp headers" #368

Merged

saschagrunert force-pushed the nix branch 3 times, most recently from 81a6f9b to 3ed8c19 Compare May 20, 2020 12:36

saschagrunert force-pushed the nix branch 2 times, most recently from affa6d3 to 71051b9 Compare May 20, 2020 12:47

saschagrunert changed the title ~~WIP: Add nix derivation for static builds~~ Add nix derivation for static builds May 20, 2020

Add nix derivation for static builds

587d5d4

Signed-off-by: Sascha Grunert <sgrunert@suse.com>

saschagrunert force-pushed the nix branch from 71051b9 to 587d5d4 Compare May 20, 2020 14:42

saschagrunert commented May 20, 2020

View reviewed changes

giuseppe merged commit bd74c78 into containers:master May 20, 2020

saschagrunert deleted the nix branch May 20, 2020 14:59

hswong3i mentioned this pull request May 21, 2020

Static Build with nix Package Manager containers/conmon#156

Closed

hswong3i added a commit to alvistack/ansible-role-crun that referenced this pull request May 21, 2020

https://github.com/containers/crun/pull/366

d3f0bc3

Conversation

saschagrunert commented May 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

saschagrunert commented May 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hswong3i commented May 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

giuseppe commented May 20, 2020

Uh oh!

hswong3i commented May 20, 2020

Uh oh!

giuseppe commented May 20, 2020

Uh oh!

saschagrunert commented May 20, 2020

Uh oh!

saschagrunert commented May 20, 2020

Uh oh!

hswong3i commented May 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saschagrunert commented May 20, 2020

Uh oh!

giuseppe commented May 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hswong3i commented May 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saschagrunert commented May 20, 2020

Uh oh!

giuseppe commented May 20, 2020

Uh oh!

saschagrunert commented May 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hswong3i commented May 20, 2020

Uh oh!

giuseppe commented May 20, 2020

Uh oh!

saschagrunert commented May 20, 2020

Uh oh!

saschagrunert commented May 20, 2020

Uh oh!

giuseppe commented May 20, 2020

Uh oh!

rhatdan commented May 20, 2020

Uh oh!

saschagrunert May 20, 2020

Choose a reason for hiding this comment

Uh oh!

saschagrunert commented May 20, 2020

Uh oh!

giuseppe commented May 20, 2020

Uh oh!

giuseppe commented May 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

saschagrunert commented May 19, 2020 •

edited

Loading

saschagrunert commented May 20, 2020 •

edited

Loading

hswong3i commented May 20, 2020 •

edited

Loading

hswong3i commented May 20, 2020 •

edited

Loading

giuseppe commented May 20, 2020 •

edited

Loading

hswong3i commented May 20, 2020 •

edited

Loading

saschagrunert commented May 20, 2020 •

edited

Loading