checkpointing a container requires root

[nerijus]

Although https://podman.io/blogs/2018/10/10/checkpoint-restore.html has examples of checkpoint run as user, on podman-0.12.1.2-1.git9551f6b.fc29.x86_64 it does not work:
$ podman container checkpoint teamviewer10
checkpointing a container requires root

[rhatdan]

I am not sure if we have rootless support for check pointing.

Could you try this with running podman as root.

[mheon]

I'm pretty sure we should just change the examples here - @adrianreber can confirm, but I think we have a requirement for root with checkpoint/restore

[nerijus]

checkpoint works as root, restore no:

# podman ps
CONTAINER ID  IMAGE                                COMMAND  CREATED         STATUS             PORTS  NAMES
8e802732b330  quay.io/nerijus/teamviewer10:latest  bash     12 seconds ago  Up 12 seconds ago         amazing_yalow
# podman container checkpoint 8e802732b330
8e802732b3309582e8d36af03f639decfb88a1ffcce2b6b8ee9a09e4ee041880
# podman container restore 8e802732b330
unable to find container 8e802732b330: no container with name or ID 8e802732b330 found: no such container

[adrianreber]

Checkpointing can only work as root as CRIU has no support for non-root checkpointing. Some years ago there was a push to get it working as non root, but right now development is stuck upstream.

There are a few kernel interfaces missing to get it working as non-root. It is doable, but right now I am not aware of anybody in the CRIU community actively working on it.

Google is using CRIU for container live migration internally as non-root: https://linuxplumbersconf.org/event/2/contributions/209/attachments/27/24/Task_Migration_at_Google_Using_CRIU.pdf

One of the main problems is that restoring a process with the same PID requires CAP_SYS_ADMIN.

[adrianreber]

@nerijus Where do you see that it works as non-root?

[nerijus]

https://podman.io/blogs/2018/10/10/checkpoint-restore.html :

"I can now checkpoint the container:
$ podman container checkpoint podman-criu-test"

[adrianreber]

@nerijus hm, okay, sorry if it confusing, but this should not mean as non-root. Why does this mean as non-root? What would be better to make it clear it works only as root?

[nerijus]

Because of prompt - $. It ususally means user, # means root.

[adrianreber]

Ah, right. Did not think about that. Good point for the future. Thanks.

[adrianreber]

checkpoint works as root, restore no:

# podman ps
CONTAINER ID  IMAGE                                COMMAND  CREATED         STATUS             PORTS  NAMES
8e802732b330  quay.io/nerijus/teamviewer10:latest  bash     12 seconds ago  Up 12 seconds ago         amazing_yalow
# podman container checkpoint 8e802732b330
8e802732b3309582e8d36af03f639decfb88a1ffcce2b6b8ee9a09e4ee041880
# podman container restore 8e802732b330
unable to find container 8e802732b330: no container with name or ID 8e802732b330 found: no such container

Can you run podman ps -a to share the state of the checkpointed container.

Can you run podman container checkpoint -k 8e802732b330. With -k there should be a file dump.log and a checkpoint directory. Can you share the log file? If you run podman with --log-level debug there should be a line where dump.log is mentioned. Something like:

Writing checkpoint logs to /var/lib/containers/storage/overlay-containers/e6463dc939fdcdf895fc549f907d4f825c9a3d1e9197240844b892a940a0361f/userdata 

Maybe also a listing of the content of the checkpoint directory.

[nerijus]

# podman ps -a
CONTAINER ID  IMAGE                                COMMAND  CREATED        STATUS            PORTS  NAMES
6767c3669a50  quay.io/nerijus/teamviewer10:latest  bash     2 seconds ago  Up 2 seconds ago         tv10
# podman container checkpoint -k tv10
6767c3669a50630dc3cc8df95acb10a7db8ae41214b3ae655c30d18c7a4d163e
# ls -l dump.log  checkpoint
ls: cannot access 'dump.log': No such file or directory
ls: cannot access 'checkpoint': No such file or directory
# podman ps -a
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES

[nerijus]

There is no "Writing checkpoint logs" with --log-level debug. After checkpoint I see this:

DEBU[0033] Checking container d379c38c493ed7ec681cba659baa26f9be9fea9ab52e7999159d14633cba0320 status... 
DEBU[0033] Cleaning up container d379c38c493ed7ec681cba659baa26f9be9fea9ab52e7999159d14633cba0320 
DEBU[0033] Network is already cleaned up, skipping...   
DEBU[0033] Storage is already unmounted, skipping...    
DEBU[0033] Storage is already unmounted, skipping...    
DEBU[0034] [graphdriver] trying provided driver "overlay" 
DEBU[0034] overlay test mount with multiple lowers succeeded 
DEBU[0034] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true 

[adrianreber]

Interesting. Not sure what is happening, yet.

Which distribution do you have, which version of Podman and which version of CRIU?

[nerijus]

F29, podman-0.12.1.2-1.git9551f6b.fc29.x86_64, criu-3.11-1.fc29.x86_64.

[adrianreber]

On F29 I see the following:

[root@fedora29 ~]# podman run -it -d alpine top
6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30
[root@fedora29 ~]# podman --log-level debug container checkpoint -l
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay test mount with multiple lowers succeeded 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] Trying to checkpoint container 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 
DEBU[0000] Writing checkpoint to /var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/checkpoint 
DEBU[0000] Writing checkpoint logs to /var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata 
DEBU[0000] Checkpointed container 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 
DEBU[0000] Cleaning up container 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 
DEBU[0000] Tearing down network namespace at /var/run/netns/cni-0fcc1cd4-ae21-0bb5-dbbc-9a993631fa98 for container 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 
INFO[0000] Got pod network &{Name:focused_carson Namespace:focused_carson ID:6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 NetNS:/var/run/netns/cni-0fcc1cd4-ae21-0bb5-dbbc-9a993631fa98 PortMappings:[] Networks:[] NetworkConfig:map[]} 
INFO[0000] About to del CNI network podman (type=bridge) 
DEBU[0000] unmounted container "6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30" 
DEBU[0000] Successfully cleaned up container 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 
6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30
[root@fedora29 ~]# podman --log-level debug container restore^Cl
[root@fedora29 ~]# podman ps
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
[root@fedora29 ~]# podman --log-level debug container restore -l
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay test mount with multiple lowers succeeded 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] Trying to restore container 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 
DEBU[0000] Restoring container with IP=10.88.0.58       
DEBU[0000] Made network namespace at /var/run/netns/cni-cc50d539-fa39-b2eb-52a7-39dc40580e10 for container 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 
INFO[0000] Got pod network &{Name:focused_carson Namespace:focused_carson ID:6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 NetNS:/var/run/netns/cni-cc50d539-fa39-b2eb-52a7-39dc40580e10 PortMappings:[] Networks:[] NetworkConfig:map[]} 
INFO[0000] About to add CNI network cni-loopback (type=loopback) 
INFO[0000] Got pod network &{Name:focused_carson Namespace:focused_carson ID:6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 NetNS:/var/run/netns/cni-cc50d539-fa39-b2eb-52a7-39dc40580e10 PortMappings:[] Networks:[] NetworkConfig:map[]} 
INFO[0000] About to add CNI network podman (type=bridge) 
DEBU[0000] mounted container "6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30" at "/var/lib/containers/storage/overlay/f138492bec7c9eac13080667a74a0445dae5aed2df0f473ddb8f9939910f964c/merged" 
DEBU[0000] Created root filesystem for container 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 at /var/lib/containers/storage/overlay/f138492bec7c9eac13080667a74a0445dae5aed2df0f473ddb8f9939910f964c/merged 
DEBU[0000] [0] CNI result: Interfaces:[{Name:cni0 Mac:46:5a:2f:5b:3b:5c Sandbox:} {Name:veth84e85e70 Mac:56:37:b9:6c:84:95 Sandbox:} {Name:eth0 Mac:0e:5d:f8:74:67:f2 Sandbox:/var/run/netns/cni-cc50d539-fa39-b2eb-52a7-39dc40580e10}], IP:[{Version:4 Interface:0xc0004d49b8 Address:{IP:10.88.0.58 Mask:ffff0000} Gateway:10.88.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}], DNS:{Nameservers:[] Domain: Search:[] Options:[]} 
DEBU[0000] generate.NewFromFile at /var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/config.json 
DEBU[0000] Created OCI spec for container 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 at /var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/config.json 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args=[-s -c 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 -u 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata -p /var/run/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/pidfile -l /var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/ctr.log --exit-dir /var/run/libpod/exits --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 --socket-dir-path /var/run/libpod/socket -t --log-level debug --syslog --restore /var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/checkpoint]
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30.scope 
DEBU[0000] Received container pid: 15501                
DEBU[0000] Restored container 6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30 
DEBU[0000] Non-fatal: removal of checkpoint file (/var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/dump.log) failed: remove /var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/dump.log: no such file or directory 
DEBU[0000] Non-fatal: removal of checkpoint file (/var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/stats-dump) failed: remove /var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/stats-dump: no such file or directory 
6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30
[root@fedora29 ~]# podman ps
CONTAINER ID  IMAGE                            COMMAND  CREATED         STATUS             PORTS  NAMES
6f724578a797  docker.io/library/alpine:latest  top      31 seconds ago  Up 31 seconds ago         focused_carson
[root@fedora29 ~]#

If you try those commands, does that work? In this case the log file would be at: /var/lib/containers/storage/overlay-containers/6f724578a7972c75f5cfe848ec0ef6c71cec13e4b6190f5d7d63565bcf318e30/userdata/dump.log