Static Build with nix Package Manager

[hswong3i]

From CRI-O toolchain point of view following packages already coming with "nix Package Manager" (https://nixos.org/nix/) for static build support:

• crun (Add nix derivation for static builds crun#366)
• Podman (Add podman static build podman#5566)
• CRI-O (Add static crio binary build for x86-64 (glibc and musl) cri-o/cri-o#2270)

As long as conmon already coming with static build support in Podman style (https://github.com/containers/conmon/blob/master/Makefile#L57-L65), we should also able to simplify it with nix-based solution, in order to get rid of the "chicken or the egg" issue.

This could also unify the static build process with nix-based, at least from crun >> conmon >> crio; afterward we may also promote it for skopeo >> buildah >> podman, which is also partly ready.

Shall we also go for a similar solution?

[hswong3i]

@saschagrunert @giuseppe any idea?

[saschagrunert]

I played around with some derivations and this would work:

{ system ? builtins.currentSystem }:
let
  pkgs = (import ./nixpkgs.nix {
    overlays = [ (final: prev: {
      pcre = prev.pcre.overrideAttrs (x: {
        configureFlags = x.configureFlags ++ [ "--enable-static" ];
      });
      p11-kit = prev.p11-kit.overrideAttrs(x: { doCheck = false; });
      e2fsprogs = prev.e2fsprogs.overrideAttrs(x: { doCheck = false; });
      gnutls = prev.gnutls.overrideAttrs(x: { doCheck = false; });
    })];
    config = {
      packageOverrides = pkg: {
        systemd = pkg.systemd.overrideAttrs(x: {
          mesonFlags = x.mesonFlags ++ [ "-Dstatic-libsystemd=true" ];
          postFixup = ''
            ${x.postFixup}
            sed -ri "s;$out/(.*);$nukedRef/\1;g" $lib/lib/libsystemd.a
          '';
        });
        glib = pkg.glib.overrideAttrs(x: {
          outputs = [ "bin" "out" "dev" ];
          mesonFlags = [
            "-Ddefault_library=static"
            "-Ddevbindir=${placeholder ''dev''}/bin"
            "-Dgtk_doc=false"
            "-Dnls=disabled"
          ];
          doCheck = false;
        });
      };
    };
  });

  self = with pkgs; {
    conmon-static = conmon.overrideAttrs(x: {
      name = "conmon-static";
      src = ./..;
      buildInputs = x.buildInputs ++ [ pcre ];
      LDFLAGS="-static-libgcc -static";
    });
  };
in self

The build will take some time because we had to overwrite pcre which is part of stdenv. Anyhow, the result is a static binary:

> file result/bin/conmon
result/bin/conmon: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, not stripped

[hswong3i]

@saschagrunert with a clean new nix install above derivations take very very very long time... Any chance if we could simplify it as like as https://github.com/containers/conmon/blob/master/install.md#debian-ubuntu-and-related-distributions?

Or, why could https://github.com/containers/conmon/blob/master/Makefile#L64-L65 be such simple and tiny?

hswong3i@hswong3i-XPS-13-7390:/tmp$ wget https://github.com/containers/conmon/releases/download/v2.0.16/conmon
hswong3i@hswong3i-XPS-13-7390:/tmp$ ldd conmon 
	statically linked

[saschagrunert]

@saschagrunert with a clean new nix install above derivations take very very very long time... Any chance if we could simplify it as like as https://github.com/containers/conmon/blob/master/install.md#debian-ubuntu-and-related-distributions?

Or, why could https://github.com/containers/conmon/blob/master/Makefile#L64-L65 be such simple and tiny?

hswong3i@hswong3i-XPS-13-7390:/tmp$ wget https://github.com/containers/conmon/releases/download/v2.0.16/conmon
hswong3i@hswong3i-XPS-13-7390:/tmp$ ldd conmon 
	statically linked

Hm, we could either:

• pre package the dependencies in a container image
• use something like cachix, but this requires local configuration changes before installing
• remove glib from conmon (I would prefer that)

[hswong3i]

pre package the dependencies in a container image

Hmm... This looks like chicken or the egg again for me...

use something like cachix, but this requires local configuration changes before installing

So how much size of result files need to be cached or installed? Since https://github.com/containers/conmon/releases/download/v2.0.16/conmon just 897KB, IMHO cache and install something, e.g. more than 50MB, should be overkilled?

remove glib from conmon (I would prefer that)

I vote for this too. BTW when compare with the build dependency with CRI-O (see https://github.com/cri-o/cri-o/blob/master/tutorials/setup.md#fedora---rhel-7---centos):

yum install -y \
  containers-common \
  device-mapper-devel \
  git \
  glib2-devel \
  glibc-devel \
  glibc-static \
  go \
  gpgme-devel \
  libassuan-devel \
  libgpg-error-devel \
  libseccomp-devel \
  libselinux-devel \
  pkgconfig \
  make \
  runc

With build dependency for conmon (see https://github.com/containers/conmon/blob/master/install.md#fedora-centos-rhel-and-related-distributions):

sudo yum install -y \
  gcc \
  git \
  glib2-devel \
  glibc-devel \
  make \
  pkgconfig \
  runc

Most likely they are similar but now conmon is missed with glibc-static? Could this be a possible shortcut?

[hswong3i]

@saschagrunert Ok i got the trick that now conmon static are using (see https://github.com/containers/conmon/blob/master/Makefile#L48-L65):

define DOCKERFILE
	FROM alpine:latest
	RUN apk add --update --no-cache bash make git gcc pkgconf libc-dev glib-dev glib-static
	COPY . /go/src/$(PROJECT)
	WORKDIR /go/src/$(PROJECT)
	RUN make static
endef
export DOCKERFILE

That's because conmon static build now based on Alpine's glib-static (https://pkgs.alpinelinux.org/package/edge/main/x86/glib-static) which should be musl-based?

In case of nix-based could we also compile with musl but not glibc? e.g.

• https://input-output-hk.github.io/haskell.nix/user-guide/cross-compilation/#static-executables-with-musl-libc
• https://github.com/nh2/static-haskell-nix/blob/master/default.nix

[hswong3i]

@saschagrunert I am now referencing https://github.com/cri-o/cri-o/blob/b99dd3cb2317a71a34d745520cf74fc1a507408b/nix/default.nix with pkgsMusl for #157:

{ system ? builtins.currentSystem }:
let
  pkgs = (import ./nixpkgs.nix {}).pkgsMusl;

  self = with pkgs; {
    conmon-static = conmon.overrideAttrs(x: {
      name = "conmon-static";
      src = ./..;
    });
  };
in self

But coming with following error message:

hswong3i@hswong3i-XPS-13-7390:~/Documents/github/conmon$ sudo nix build -f nix/ --show-trace
error: while evaluating the attribute 'buildInputs' of the derivation 'conmon-static' at /home/hswong3i/Documents/github/conmon/nix/default.nix:7:7:
while evaluating the attribute 'buildInputs' of the derivation 'glib-2.64.1' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'buildInputs' of the derivation 'libselinux-2.9' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'buildInputs' of the derivation 'fts-netbsd-8.0' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/os-specific/bsd/netbsd/default.nix:28:5:
while evaluating the attribute 'extraPaths' of the derivation 'compat-netbsd-8.0' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/os-specific/bsd/netbsd/default.nix:28:5:
while evaluating the attribute 'nativeBuildInputs' of the derivation 'cvs-export' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/build-support/fetchcvs/default.nix:11:3:
while evaluating the attribute 'buildInputs' of the derivation 'openssh-8.2p1' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'buildInputs' of the derivation 'libfido2-1.4.0' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'buildInputs' of the derivation 'hidapi-0.9.0' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'preFixup' of the derivation 'libusb-1.0.23' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating 'optionalString' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/lib/strings.nix:180:5, called from /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/development/libraries/libusb1/default.nix:36:14:
while evaluating the attribute 'buildInputs' of the derivation 'systemd-245.5' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
infinite recursion encountered, at undefined position

May you share some hints for compiling with musl, as what we had for CRI-O before?

[saschagrunert]

@saschagrunert I am now referencing https://github.com/cri-o/cri-o/blob/b99dd3cb2317a71a34d745520cf74fc1a507408b/nix/default.nix with pkgsMusl for #157:

{ system ? builtins.currentSystem }:
let
  pkgs = (import ./nixpkgs.nix {}).pkgsMusl;

  self = with pkgs; {
    conmon-static = conmon.overrideAttrs(x: {
      name = "conmon-static";
      src = ./..;
    });
  };
in self

But coming with following error message:

hswong3i@hswong3i-XPS-13-7390:~/Documents/github/conmon$ sudo nix build -f nix/ --show-trace
error: while evaluating the attribute 'buildInputs' of the derivation 'conmon-static' at /home/hswong3i/Documents/github/conmon/nix/default.nix:7:7:
while evaluating the attribute 'buildInputs' of the derivation 'glib-2.64.1' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'buildInputs' of the derivation 'libselinux-2.9' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'buildInputs' of the derivation 'fts-netbsd-8.0' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/os-specific/bsd/netbsd/default.nix:28:5:
while evaluating the attribute 'extraPaths' of the derivation 'compat-netbsd-8.0' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/os-specific/bsd/netbsd/default.nix:28:5:
while evaluating the attribute 'nativeBuildInputs' of the derivation 'cvs-export' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/build-support/fetchcvs/default.nix:11:3:
while evaluating the attribute 'buildInputs' of the derivation 'openssh-8.2p1' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'buildInputs' of the derivation 'libfido2-1.4.0' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'buildInputs' of the derivation 'hidapi-0.9.0' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'preFixup' of the derivation 'libusb-1.0.23' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating 'optionalString' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/lib/strings.nix:180:5, called from /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/development/libraries/libusb1/default.nix:36:14:
while evaluating the attribute 'buildInputs' of the derivation 'systemd-245.5' at /nix/store/10knp2l4kbxbna9qbgg0qbdz7jia2ai9-nixos-unstable/pkgs/stdenv/generic/make-derivation.nix:192:11:
infinite recursion encountered, at undefined position

May you share some hints for compiling with musl, as what we had for CRI-O before?

Looks like that the muslPkgs got broken on the current master branch of nixpkgs. When testing with an earlier commit like this then it will work:

let
  nixpkgs = import (builtins.fetchTarball {
    name = "nixos-unstable";
    url = "https://github.com/nixos/nixpkgs/archive/" +
      "221274e1552fdb84e8caf0831dbd9140b111131e.tar.gz";
    sha256 = "1ic9d9c27qpcrx5n5dn1mak3hrbbdq1is24p11rqrj9xas79lf89";
  });
in nixpkgs

In any case, building with musl will result in an even longer build time than my proposed way because we will rebuild everything by hand.

[hswong3i]

Looks like that the muslPkgs got broken on the current master branch of nixpkgs. When testing with an earlier commit like this then it will work:

let
  nixpkgs = import (builtins.fetchTarball {
    name = "nixos-unstable";
    url = "https://github.com/nixos/nixpkgs/archive/" +
      "221274e1552fdb84e8caf0831dbd9140b111131e.tar.gz";
    sha256 = "1ic9d9c27qpcrx5n5dn1mak3hrbbdq1is24p11rqrj9xas79lf89";
  });
in nixpkgs

In any case, building with musl will result in an even longer build time than my proposed way because we will rebuild everything by hand.

Yes building with musl looks much simple stupid for maintenance, but take much much longer time for compile :-(

@saschagrunert I am now rolling back #157 as your #156 (comment) ;-)