Podman with image volume fails to create directory after restart

[mheon]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

Description
Restarting a container using an image volumes makes podman unable to create directories inside the volume mount.

Steps to reproduce the issue:

1. podman run -ti --name zmdb -e MYSQL_USER=zmuser -e MYSQL_PASSWORD=zmpass -e MYSQL_DATABASE=zm -e MYSQL_ROOT_PASSWORD=mysqlpassword centos/mariadb-101-centos7

2. podman stop --latest

3. podman start --latest

Describe the results you received:

Fatal error Can't create database directory '/var/lib/mysql/data/mysql'

Describe the results you expected:

Mariadb starts successfully in the container

Additional information you deem important (e.g. issue happens only occasionally):

The image mariadb-101-centos7 defines the following volume:
VOLUME ["/var/lib/mysql/data"]

Output of podman version:

Observed on upstream master and Fedora released 0.5.3

[vrothberg]

@mheon I tried to reproduce on current master but it works on my machine. Can you try to reproduce on current master?

[mheon]

@vrothberg On latest master, I've actually regressed further, to the point that I get the permission denied without a restart, just using the first podman run command. setenforce 0 makes no difference, so it's not SELinux. I execed into the container, and permissions on the directory look fine.

[mheon]

After some testing, @vrothberg and I determined this occurs with the overlay storage driver but not btrfs.

[mheon]

--cap-add=all will allow it to succeed on overlay, so we're looking at a missing capability.

[vrothberg]

After some testing, @vrothberg and I determined this occurs with the overlay storage driver but not btrfs.

For whatever reason, I am now also able to reproduce on btrfs.

[marcov]

I can also replicate the bug using a locally compiled podman (65033b5).
For me the bug is already happening on the very first run of the container.

The problem seems related to wrong write permissions for the folder /var/lib/mysql/data. Running ls -ls /var/lib/mysql from inside the container:
0 drwxr-xr-x 1 root root 0 Jun 14 08:51 data

Given that the container is run as user mysql, write access is denied. If I force running the container as root, then write access is granted.
Another way to allow write access is to add the capability dac_override that disables checks on file write, as reported by @mheon.

I tried to dump the filesystem of the container image and the data folder has the right permissions:
0 drwxrwxr-x 2 27 root 6 Jun 5 17:05 data
(UID 27 = mysql, and mysql user is in the root group)

So the problem may be with podman setting up the container with wrong permissions for the data folder.

[rhatdan]

So podman is creating a volume on disk and mounting it on /var/lib/mysql/data owned by root:root rather then mysql:root

[mheon]

In Docker, the directory is 775 mysql:root. In Podman, it is 755 root:root.
I suspect that Docker may be using the owner and permissions of the parent directory of the image volume, as the permissions on their volume are identical to the ones on the directory containing the volume.

[rhatdan]

Right, I would say that is the correct behavior we should take the security attributes of the directory we are mounting over and set it to the new volume.

[marcov]

I hacked together a patch to assign the correct owner and permissions to the volume folder (@rhatdan approach is maybe better).

With this patch write access works for the mysql user but now I can actually fall in the bug as originally reported, that is the start command following a stop does not remount the volume correctly.

After the start, an ls -ls /var/lib/mysql/data from inside the container gives the following:
0 drwx------ 1 root root 92 Jun 14 15:38 data

And by checking inside the data folder (as root user), I see that something is definitely wrong:

artifacts  attach  config.json  ctl  ctr.log  shm  volumes

Also there seems to be an extra mount point that should not be there:

# mount
...
shm on /var/lib/mysql/data/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=64000k)
...

Finally here's the patch I mentioned before:

diff --git a/libpod/container_internal.go b/libpod/container_internal.go
index f6d8fc3..f15cb6a 100644
--- a/libpod/container_internal.go
+++ b/libpod/container_internal.go
@@ -1324,6 +1324,21 @@ func (c *Container) addImageVolumes(ctx context.Context, g *generate.Generator)
 			if err = os.MkdirAll(volumePath, 0755); err != nil {
 				return errors.Wrapf(err, "error creating directory %q for volume %q in container %q", volumePath, k, c.ID)
 			}
+
+			if c.config.User != "" {
+				if !c.state.Mounted {
+					return errors.Wrapf(ErrCtrStateInvalid, "container %s must be mounted in order to translate User field", c.ID())
+				}
+				uid, gid, err := chrootuser.GetUser(c.state.Mountpoint, c.config.User)
+				if err != nil {
+					return err
+				}
+
+				if err = os.Chown(volumePath, int(uid), int(gid)); err != nil {
+					return errors.Wrapf(err, "error chowning directory %q for volume %q in container %q", volumePath, k, c.ID)
+				}
+			}
+
 			if err = label.Relabel(volumePath, c.config.MountLabel, false); err != nil {
 				return errors.Wrapf(err, "error relabeling directory %q for volume %q in container %q", volumePath, k, c.ID)
 			}

[marcov]

I just tested with other images configured to use a volume.
The problem with volume remount when doing stop + start is there as well, so it's definitely something to investigate.

[mheon]

@marcov Thanks for the patch. I'll look into getting it merged this afternoon (might need to change it a bit for userns support).

Agree that the original bug still seems to be present with it applied.

[marcov]

I found the original bug reason: the Source field of the spec.Mount is left uninitialized when
restarting a container, leading to mounting an incorrect path instead of the volume path on the host.

I am submitting a pull request with the fix to this problem and other improvements.