Erratic behaviour with toolbox /etc/resolv.conf when changing its symlink target in host

[nanonyme]

Describe the bug
toolbox seems to resolve the final target of /etc/resolv.conf when creating the image. This is causing a set of weird side-effects

• If /etc/resolv.conf is a real file when creating toolbox and is changed to point to /run/systemd/resolve/stub-resolv.conf after creating toolbox, there will be a broken symlink chain /etc/resolv.conf -> /run/host/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf (this should be fixed through Also bind mount systemd-resolved /run directories #707)
• If /etc/resolv.conf points to /run/systemd/resolve/stub-resolv.conf and it's change to /run/systemd/resolve/resolv.conf, the change is not reflected inside toolbox at all

Steps how to reproduce the behaviour

1. sudo rm /etc/resolv.conf && sudo touch /etc/resolv.conf
2. toolbox enter
3. readlink -f /etc/resolv.conf
4. exit
5. sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
6. toolbox enter
7. readlink -f /etc/resolv.conf

Expected behaviour
Step 3 outputs /run/host/etc/resolv.conf, step 7 outputs /run/systemd/resolv/stub-resolv.conf

Actual behaviour
Step 3 outputs /run/host/etc/resolv.conf, step 7 outputs nothing because /etc/resolv.conf is now a symlink to a broken symlink

Output of toolbox --version (v0.0.90+)
toolbox version 0.0.99.1

Toolbox package info (rpm -q toolbox)
toolbox-0.0.99.1-1.fc34.x86_64

Output of podman version

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.16
Built:        Mon Feb 22 16:08:57 2021
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)
podman-3.0.1-1.fc34.x86_64

Info about your OS
Fedora SIlverblue 34

[nanonyme]

@debarshiray any thoughts? What I would suggest after the bind mount is that we stop resolving /etc/resolv.conf further. Just allow it to always be a symlink to /run/host/resolv.conf and things should work consistently.

[debarshiray]

Is this still a problem? Isn't this a duplicate of #187 ?

This is what we have in the containers by default:

⬢[rishi@toolbox ~]$ ls -l /etc/resolv.conf 
lrwxrwxrwx. 1 root root 25 Nov 13 03:35 /etc/resolv.conf -> /run/host/etc/resolv.conf
⬢[rishi@toolbox ~]$ ls -l /run/host/etc/resolv.conf 
lrwxrwxrwx. 1 nobody nobody 39 Oct 20  2020 /run/host/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

Generally speaking, things work best when the symbolic links are relative, not absolute.

See the redirectPath function in src/cmd/initContainer.go for the details.

[debarshiray]

* If /etc/resolv.conf is a real file when creating toolbox and

is changed to point to /run/systemd/resolve/stub-resolv.conf
after creating toolbox, there will be a broken symlink chain
/etc/resolv.conf -> /run/host/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
(this should be fixed through Also bind mount systemd-resolved /run directories #707)

This one sounds like a duplicate of #187 and should definitely be fixed by #707 now.

Even before, it would work just fine, if /etc/resolv.conf on the host was a relative symbolic link to ../run/systemd/resolve/stub-resolv.conf. #460 added some heuristics to make it better in the most obvious case.

* If /etc/resolv.conf points to /run/systemd/resolve/stub-resolv.conf

and it's change to /run/systemd/resolve/resolv.conf, the change is not
reflected inside toolbox at all

Umm... I don't understand this one.

From a high-level, we want the container's /etc/resolv.conf to point to the host's version at /run/host/etc/resolv.conf, and if possible avoid overwriting any changes the user might have made to the container on their own.

The problem is that /etc/resolv.conf is a regular file in containers created like this:

podman create --dns none --network host --no-hosts ...

... and we do want to overwrite that. So, we take the lazy way out and always stomp over /etc/resolv.conf if it's a regular file.

If it's a symbolic link that doesn't resolve, then we also stomp over it, because it's broken anyway.

That means, we only skip touching valid symbolic links. Hence, users can tweak their containers by changing the target of the container's /etc/resolv.conf symlink.

[nanonyme]

Iirc the original problem was toolbox was trying to be too smart resolving symlinks before creating image. And the path didn't update on subsequent runs resulting in mess if admin or distro reconfigures host DNS setup afterwards. I haven't tried to repro afterwards. The original bug happened because Fedora had bug where systemd-resolved wasn't used in first-time install and them rolled out package update that took it into use that broke any existing toolboxes.

[debarshiray]

Iirc the original problem was toolbox was trying to be too
smart resolving symlinks before creating image.

I think the basic logic has remain unchanged, at least for a very long time. eg., see this commit where the relevant code was re-written into Go.

A lot of people were complaining about #187 because somehow they ended up with an absolute symbolic link on their hosts.

However, I could be wrong, and symlinks are full of mischief. :)

[debarshiray]

Closing.

I think the problem is caused by the limitations of absolute symbolic links. Please leave a comment or re-open if you have reason to believe otherwise.