Support for rootless builds

[KurtSchluss]

Hi, IMHO one of the most exciting features of podman is the rootless execution of containers.

I have tried to share /dev/kvm using --mount=type=bind,source=/dev/kvm,destination=/dev/kvm.
This works for init and fetch, but build fails:

user@host ~/coreos % podman --log-level info run -ti --rm --net=host --userns=host --privileged -v $(pwd):/srv --workdir /srv --mount=type=bind,source=/dev/kvm,destination=/dev/kvm quay.io/coreos-assembler/coreos-assembler build
INFO[0000] running as rootless                          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
INFO[0000] running as rootless                          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
WARN[0037] Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied 
Using manifest: /srv/src/config/manifest.yaml
libostree:
 Version: '2018.10'
 Git: c5aaabe9d754d2ada17d5cc006e13105ab5cecc2
 DevelBuild: yes
 Features:
  - libcurl
  - no-http2
  - libsoup
  - gpgme
  - libarchive
  - selinux
  - openssl
  - libmount
  - devel
  - p2p
rpm-ostree:
 Version: '2018.9'
 Git: 0e24944c4ec0a45eb88169a60c96c8b76e86cd61
 Features:
  - compose
  - rust
Previous commit: none
Kickstart checksum: 93de3934c4488ab094a1b8ffce1b00ac1cf8d1c4d7091519202ceb484f2d25fc
Running: rpm-ostree compose tree --repo=/srv/repo --cachedir=/srv/cache --touch-if-changed /srv/tmp/treecompose.changed --unified-core /srv/src/config/manifest.yaml --cache-only --add-metadata-from-json /srv/tmp/build/tmp/commit-metadata-input.json --write-composejson-to /srv/tmp/build/tmp/compose.json
RPM-OSTree Version: 2018.9
No previous commit for fedora/29/x86_64/coreos
Enabled rpm-md repositories: dustymabe-ignition fedora fedora-updates fedora-updates-testing
rpm-md repo 'dustymabe-ignition' (cached); generated: 2018-11-21T22:40:42Z
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'fedora-updates' (cached); generated: 2018-11-23T02:15:41Z
rpm-md repo 'fedora-updates-testing' (cached); generated: 2018-11-23T02:31:21Z
Importing rpm-md... done
Resolving dependencies... done
Installing 388 packages:
  GeoIP-1.6.12-4.fc29.x86_64 (fedora)
...
  zlib-1.2.11-14.fc29.x86_64 (fedora)
Input state hash: 22c7dab4135b85f9d8bea7c3d4e75989a1460b566e1c60f2f5a6972d6f700cdc
Checking out packages... done
⠁ Running pre scripts... libini_config
fuse: device not found, try 'modprobe fuse' first
⠉ Running pre scripts... dbus-common
dbus-common.prein: bwrap: execvp /bin/sh: No such file or directory
fusermount: failed to unmount /tmp/rpmostree-rofiles-fuse.XODVvm: Invalid argument
Running pre scripts... done
error: Running %prein for dbus-common: Executing bwrap(/bin/sh): Child process killed by signal 1
podman --log-level info run -ti --rm --net=host --userns=host --privileged -v  11,15s user 21,99s system 54% cpu 1:00,25 total

Do you see any chance to enable rootless builds?

podman version 0.11.1.1
Arch Linux

[KurtSchluss]

Adding --mount=type=bind,src=/dev/fuse,dst=/dev/fuse did help.

Build was successful with one error message:

Completed install to disk image: /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
++ pwd
+ /usr/lib/coreos-assembler/gf-anaconda-cleanup /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
+ coreos_gf_run_mount /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
+ coreos_gf_launch /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
+ local src=/srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
+ shift
+ local guestfish
+ guestfish[0]=guestfish
+ guestfish[1]=--listen
+ guestfish[3]=-a
+ guestfish[4]=/srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
++ guestfish --listen -a /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2
+ eval 'GUESTFISH_PID=13649; export GUESTFISH_PID'
++ GUESTFISH_PID=13649
++ export GUESTFISH_PID
+ '[' -z 13649 ']'
+ coreos_gf run
+ guestfish --remote -- run
+ local root
++ coreos_gf findfs-label root
++ guestfish --remote -- findfs-label root
+ root=/dev/sda2
+ coreos_gf mount /dev/sda2 /
+ guestfish --remote -- mount /dev/sda2 /
+ local boot
++ coreos_gf findfs-label boot
++ guestfish --remote -- findfs-label boot
+ boot=/dev/sda1
+ coreos_gf mount /dev/sda1 /boot
+ guestfish --remote -- mount /dev/sda1 /boot
++ coreos_gf -findfs-label var
libguestfs: error: findfs_label: findfs exited with status 1: findfs: unable to resolve 'LABEL=var'
+ var=
++ coreos_gf ls /ostree/deploy
++ guestfish --remote -- ls /ostree/deploy
+ stateroot=/ostree/deploy/fedora-coreos
++ coreos_gf ls /ostree/deploy/fedora-coreos/deploy
++ grep -v .origin
++ guestfish --remote -- ls /ostree/deploy/fedora-coreos/deploy
+ deploydir=/ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0
+ '[' -n '' ']'
+ export stateroot deploydir
+ coreos_gf rm-rf /ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0/etc/sysconfig/anaconda
+ guestfish --remote -- rm-rf /ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0/etc/sysconfig/anaconda
+ coreos_gf rm-rf /ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0/etc/systemd/system/default.target
+ guestfish --remote -- rm-rf /ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0/etc/systemd/system/default.target
+ coreos_gf rm-rf '/ostree/deploy/fedora-coreos/var/*'
+ guestfish --remote -- rm-rf '/ostree/deploy/fedora-coreos/var/*'
+ coreos_gf_shutdown
+ coreos_gf umount-all
+ guestfish --remote -- umount-all
+ coreos_gf exit
+ guestfish --remote -- exit
+ _coreos_gf_cleanup
+ guestfish --remote -- exit
+ :
++ pwd
++ pwd
+ /usr/lib/coreos-assembler/gf-oemid /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2 /srv/tmp/build/fedora-coreos-29-qemu.qcow2 qemu
++ mktemp -td gf-oemid.XXXXXX
+ tmpd=/tmp/gf-oemid.l7JGf7
+ tmp_dest=/tmp/gf-oemid.l7JGf7/box.img
+ qemu_wrapper=/tmp/gf-oemid.l7JGf7/qemu-wrapper.sh
+ cat
+ chmod +x /tmp/gf-oemid.l7JGf7/qemu-wrapper.sh
+ export LIBGUESTFS_HV=/tmp/gf-oemid.l7JGf7/qemu-wrapper.sh
+ LIBGUESTFS_HV=/tmp/gf-oemid.l7JGf7/qemu-wrapper.sh
+ cp --reflink=auto /srv/tmp/build/tmp/fedora-coreos-29-base.qcow2 /tmp/gf-oemid.l7JGf7/box.img
+ chmod u+w /tmp/gf-oemid.l7JGf7/box.img
+ coreos_gf_run_mount /tmp/gf-oemid.l7JGf7/box.img
+ coreos_gf_launch /tmp/gf-oemid.l7JGf7/box.img
+ local src=/tmp/gf-oemid.l7JGf7/box.img
+ shift
+ local guestfish
+ guestfish[0]=guestfish
+ guestfish[1]=--listen
+ guestfish[3]=-a
+ guestfish[4]=/tmp/gf-oemid.l7JGf7/box.img
++ guestfish --listen -a /tmp/gf-oemid.l7JGf7/box.img
+ eval 'GUESTFISH_PID=13865; export GUESTFISH_PID'
++ GUESTFISH_PID=13865
++ export GUESTFISH_PID
+ '[' -z 13865 ']'
+ coreos_gf run
+ guestfish --remote -- run
+ local root
++ coreos_gf findfs-label root
++ guestfish --remote -- findfs-label root
+ root=/dev/sda2
+ coreos_gf mount /dev/sda2 /
+ guestfish --remote -- mount /dev/sda2 /
+ local boot
++ coreos_gf findfs-label boot
++ guestfish --remote -- findfs-label boot
+ boot=/dev/sda1
+ coreos_gf mount /dev/sda1 /boot
+ guestfish --remote -- mount /dev/sda1 /boot
++ coreos_gf -findfs-label var
libguestfs: error: findfs_label: findfs exited with status 1: findfs: unable to resolve 'LABEL=var'
+ var=
++ coreos_gf ls /ostree/deploy
++ guestfish --remote -- ls /ostree/deploy
+ stateroot=/ostree/deploy/fedora-coreos
++ coreos_gf ls /ostree/deploy/fedora-coreos/deploy
++ guestfish --remote -- ls /ostree/deploy/fedora-coreos/deploy
++ grep -v .origin
+ deploydir=/ostree/deploy/fedora-coreos/deploy/fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881.0
+ '[' -n '' ']'
+ export stateroot deploydir
+ grubcfg_path=/boot/loader/grub.cfg
+ coreos_gf download /boot/loader/grub.cfg /tmp/gf-oemid.l7JGf7/grub.cfg
+ guestfish --remote -- download /boot/loader/grub.cfg /tmp/gf-oemid.l7JGf7/grub.cfg
+ sed -i -e 's, coreos.oem.id=[a-zA-Z0-9]*,,g' /tmp/gf-oemid.l7JGf7/grub.cfg
+ sed -i -e 's,^\(linux16 .*\),\1 coreos.oem.id=qemu,' /tmp/gf-oemid.l7JGf7/grub.cfg
+ coreos_gf upload /tmp/gf-oemid.l7JGf7/grub.cfg /boot/loader/grub.cfg
+ guestfish --remote -- upload /tmp/gf-oemid.l7JGf7/grub.cfg /boot/loader/grub.cfg
++ coreos_gf glob-expand '/boot/loader/entries/ostree-*.conf'
++ guestfish --remote -- glob-expand '/boot/loader/entries/ostree-*.conf'
+ blscfg_path=/boot/loader/entries/ostree-1-fedora-coreos.conf
+ coreos_gf download /boot/loader/entries/ostree-1-fedora-coreos.conf /tmp/gf-oemid.l7JGf7/bls.conf
+ guestfish --remote -- download /boot/loader/entries/ostree-1-fedora-coreos.conf /tmp/gf-oemid.l7JGf7/bls.conf
+ sed -i -e 's, coreos.oem.id=[a-zA-Z0-9]*,,g' /tmp/gf-oemid.l7JGf7/bls.conf
+ sed -i -e 's,^\(options .*\),\1 coreos.oem.id=qemu,' /tmp/gf-oemid.l7JGf7/bls.conf
+ coreos_gf upload /tmp/gf-oemid.l7JGf7/bls.conf /boot/loader/entries/ostree-1-fedora-coreos.conf
+ guestfish --remote -- upload /tmp/gf-oemid.l7JGf7/bls.conf /boot/loader/entries/ostree-1-fedora-coreos.conf
+ coreos_gf_shutdown
+ coreos_gf umount-all
+ guestfish --remote -- umount-all
+ coreos_gf exit
+ guestfish --remote -- exit
+ mv /tmp/gf-oemid.l7JGf7/box.img /srv/tmp/build/fedora-coreos-29-qemu.qcow2
+ rm /tmp/gf-oemid.l7JGf7 -rf
+ _coreos_gf_cleanup
+ guestfish --remote -- exit
+ :
+ set +x
Ignoring non-directory .build-commit
Pruning repo
Total objects: 30887
No unreachable objects
podman --log-level info run -ti --rm --net=host --userns=host --privileged -v  11,14s user 22,20s system 5% cpu 9:55,82 total

Executing run works as well. :)

[jlebon]

If you referring to:

++ coreos_gf -findfs-label var
libguestfs: error: findfs_label: findfs exited with status 1: findfs: unable to resolve 'LABEL=var'

You can ignore that. Hmm, it already has 2>/dev/null, so it's printing this on stdout I guess?

Anyway, glad to hear this works!

[KurtSchluss]

Yes, I did.

Unfortunately there is one exception and I don't know if this is related to the rootless execution, but
re-running build fails:

user@host ~/coreos % podman --log-level info run -ti --rm --net=host --userns=host --privileged -v $(pwd):/srv --workdir /srv --mount=type=bind,source=/dev/kvm,destination=/dev/kvm --mount=type=bind,src=/dev/fuse,dst=/dev/fuse quay.io/coreos-assembler/coreos-assembler build
INFO[0000] running as rootless                          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
INFO[0000] running as rootless                          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
WARN[0032] Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied 
Using manifest: /srv/src/config/manifest.yaml
libostree:
 Version: '2018.10'
 Git: c5aaabe9d754d2ada17d5cc006e13105ab5cecc2
 DevelBuild: yes
 Features:
  - libcurl
  - no-http2
  - libsoup
  - gpgme
  - libarchive
  - selinux
  - openssl
  - libmount
  - devel
  - p2p
rpm-ostree:
 Version: '2018.9'
 Git: 0e24944c4ec0a45eb88169a60c96c8b76e86cd61
 Features:
  - compose
  - rust
Previous build: 29
Previous commit: fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881
Kickstart checksum: 93de3934c4488ab094a1b8ffce1b00ac1cf8d1c4d7091519202ceb484f2d25fc
Running: rpm-ostree compose tree --repo=/srv/repo --cachedir=/srv/cache --touch-if-changed /srv/tmp/treecompose.changed --unified-core /srv/src/config/manifest.yaml --cache-only --add-metadata-from-json /srv/tmp/build/tmp/commit-metadata-input.json --write-composejson-to /srv/tmp/build/tmp/compose.json
RPM-OSTree Version: 2018.9
Previous commit: fa5ec34e3062ae69703c22908693fd1ab38e1c90870a3a4e7c58b2782a317881
error: fsetxattr: Operation not permitted
podman --log-level info run -ti --rm --net=host --userns=host --privileged -v  11,22s user 21,23s system 84% cpu 38,480 total

After a clean, build succeeds again.

[jlebon]

Hmm, not sure. Though you should be able to use the supermin backend instead (#190). Should be able to just drop the --privileged flag and it'll automatically fallback to that, or otherwise use FORCE_UNPRIVILEGED. But note this requires an up to date rpm-ostree: #194 (comment)

[KurtSchluss]

Thank you very much for that hint, dropping --privileged works like a charm!
Also, re-running build causes no issue anymore.
Only thing I had to do was wiping the directory and starting from scratch.

For the record, here is the alias I have been using:
alias coreos-assembler='podman run -ti --rm --net=host --userns=host -v $(pwd):/srv --workdir /srv --mount=type=bind,source=/dev/kvm,destination=/dev/kvm --mount=type=bind,src=/dev/fuse,dst=/dev/fuse quay.io/coreos-assembler/coreos-assembler'