Skip to content

Fix DNS queries in images with systemd-resolved on hosts without it#1425

Merged
debarshiray merged 1 commit intocontainers:mainfrom
debarshiray:wip/rishi/issues-1410
Dec 18, 2023
Merged

Fix DNS queries in images with systemd-resolved on hosts without it#1425
debarshiray merged 1 commit intocontainers:mainfrom
debarshiray:wip/rishi/issues-1410

Conversation

@debarshiray
Copy link
Copy Markdown
Member

@debarshiray debarshiray commented Dec 17, 2023

On some Toolbx images with systemd-resolved(8), like the fedora-toolbox
images for Fedora 39 onwards, /etc/resolv.conf can end up being a
symbolic link inside the container that expects the host operating
system to also use systemd-resolved(8):

  $ ls -l /etc/resolv.conf
  lrwxrwxrwx. 1 root root 39 Nov 28 08:50 /etc/resolv.conf ->
    ../run/systemd/resolve/stub-resolv.conf

This happens because systemd-resolved(8) already makes /etc/resolv.conf
a symbolic link inside the image, and, hence, the container's entry
point doesn't change it to point at the host's copy of the file at
/run/host/etc/resolv.conf. Instead, it's left pointing at the host's
copy of the files maintained by systemd-resolved(8) under
/run/systemd/resolve, which happen to be also available inside the
container [1].

If the host OS doesn't use systemd-resolved(8), like Red Hat Enterprise
Linux 9, then this leads to a dangling symbolic link and breaks DNS
queries.

Note that the presence of systemd-resolved(8) in the recent
fedora-toolbox images is a regression caused by the ToolbxReleaseBlocker
Change [2] for Fedora 39 where the image was rewritten in terms of
fedora-kickstarts and pungi-fedora instead of a Container/Dockerfile.
By mistake, systemd crept in as an RPM needed by the image [3], which
in turn pulled in the systemd-resolved RPM as a weak dependency [4].

Hopefully, that will get fixed. However, it's also not practical to
keep track of all the Toolbx images out there in the wild, so it's
wise to make toolbox(1) more resilient to such things.

This will have the downside of overwriting some custom user-made
modifications to the container's /etc/resolv.conf. While that's
unfortunate, it's more important to have Toolbx images produce working
containers on a wide range of host OSes. It will be better to come up
with a more explicit way to support custom user-made modifications to
the container's configuration. Perhaps with a persistent stamp file.

[1] Commit af602c7
af602c7d227617d2
#707

[2] https://fedoraproject.org/wiki/Changes/ToolbxReleaseBlocker

[3] fedora-kickstarts commit 48e2c3b5598de32f
https://pagure.io/fedora-kickstarts/c/48e2c3b5598de32f

[4] fedora-kickstarts commit 49306cb6eada8777
https://pagure.io/fedora-kickstarts/c/49306cb6eada8777

#1410

@debarshiray debarshiray force-pushed the wip/rishi/issues-1410 branch from 623f4e4 to 5777312 Compare December 17, 2023 21:49
@debarshiray debarshiray marked this pull request as draft December 17, 2023 21:53
@debarshiray debarshiray changed the title cmd/initContainer: [WIP] cmd/initContainer: Dec 17, 2023
@softwarefactory-project-zuul
Copy link
Copy Markdown

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/d4d7c02f4149490f8be84e81b8e6214b

✔️ unit-test SUCCESS in 6m 46s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 08s
✔️ unit-test-restricted SUCCESS in 5m 48s
✔️ system-test-fedora-rawhide SUCCESS in 31m 18s
✔️ system-test-fedora-39 SUCCESS in 29m 58s
✔️ system-test-fedora-38 SUCCESS in 29m 44s

@debarshiray debarshiray changed the title [WIP] cmd/initContainer: [WIP] Fix DNS queries in images with systemd-resolved on hosts without it Dec 17, 2023
@debarshiray debarshiray force-pushed the wip/rishi/issues-1410 branch 2 times, most recently from e6fbc82 to 97d9ce3 Compare December 18, 2023 00:27
@softwarefactory-project-zuul
Copy link
Copy Markdown

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/39dd11f1f2484e71bd2d32ec89912cb2

✔️ unit-test SUCCESS in 6m 51s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 09s
✔️ unit-test-restricted SUCCESS in 5m 53s
system-test-fedora-rawhide FAILURE in 28m 53s
system-test-fedora-39 FAILURE in 28m 41s
system-test-fedora-38 FAILURE in 29m 50s

@debarshiray
Copy link
Copy Markdown
Member Author

Cool, the tests do fail if /etc/resolv.conf inside the Toolbx container is not a symbolic link:

fedora-rawhide | not ok 195 network: /etc/resolv.conf inside the default container in 1724ms
fedora-rawhide | # (from function `assert_success' in file test/system/libs/bats-assert/src/assert.bash, line 114,
fedora-rawhide | #  in test file test/system/203-network.bats, line 70)
fedora-rawhide | #   `assert_success' failed
fedora-rawhide | #
fedora-rawhide | # -- command failed --
fedora-rawhide | # status : 1
fedora-rawhide | # output :
fedora-rawhide | # --

I had replaced /etc/resolv.conf with /etc/bashrc to test the tests. :P

@debarshiray debarshiray force-pushed the wip/rishi/issues-1410 branch 2 times, most recently from 79967d3 to 47aafa8 Compare December 18, 2023 12:12
@softwarefactory-project-zuul
Copy link
Copy Markdown

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/958d94ccb32345f29b09a51292716648

✔️ unit-test SUCCESS in 7m 04s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 17s
✔️ unit-test-restricted SUCCESS in 6m 02s
✔️ system-test-fedora-rawhide SUCCESS in 29m 17s
✔️ system-test-fedora-39 SUCCESS in 27m 49s
✔️ system-test-fedora-38 SUCCESS in 28m 07s

On some Toolbx images with systemd-resolved(8), like the fedora-toolbox
images for Fedora 39 onwards, /etc/resolv.conf can end up being a
symbolic link inside the container that expects the host operating
system to also use systemd-resolved(8):
  $ ls -l /etc/resolv.conf
  lrwxrwxrwx. 1 root root 39 Nov 28 08:50 /etc/resolv.conf ->
    ../run/systemd/resolve/stub-resolv.conf

This happens because systemd-resolved(8) already makes /etc/resolv.conf
a symbolic link inside the image, and, hence, the container's entry
point doesn't change it to point at the host's copy of the file at
/run/host/etc/resolv.conf.  Instead, it's left pointing at the host's
copy of the files maintained by systemd-resolved(8) under
/run/systemd/resolve, which happen to be also available inside the
container [1].

If the host OS doesn't use systemd-resolved(8), like Red Hat Enterprise
Linux 9, then this leads to a dangling symbolic link and breaks DNS
queries.

Note that the presence of systemd-resolved(8) in the recent
fedora-toolbox images is a regression caused by the ToolbxReleaseBlocker
Change [2] for Fedora 39 where the image was rewritten in terms of
fedora-kickstarts and pungi-fedora instead of a Container/Dockerfile.
By mistake, systemd crept in as an RPM needed by the image [3], which
in turn pulled in the systemd-resolved RPM as a weak dependency [4].

Hopefully, that will get fixed.  However, it's also not practical to
keep track of all the Toolbx images out there in the wild, so it's
wise to make toolbox(1) more resilient to such things.

This will have the downside of overwriting some custom user-made
modifications to the container's /etc/resolv.conf.  While that's
unfortunate, it's more important to have Toolbx images produce working
containers on a wide range of host OSes.  It will be better to come up
with a more explicit way to support custom user-made modifications to
the container's configuration.  Perhaps with a persistent stamp file.

[1] Commit af602c7
    containers@af602c7d227617d2
    containers#707

[2] https://fedoraproject.org/wiki/Changes/ToolbxReleaseBlocker

[3] fedora-kickstarts commit 48e2c3b5598de32f
    https://pagure.io/fedora-kickstarts/c/48e2c3b5598de32f

[4] fedora-kickstarts commit 49306cb6eada8777
    https://pagure.io/fedora-kickstarts/c/49306cb6eada8777

containers#1410
@debarshiray debarshiray force-pushed the wip/rishi/issues-1410 branch from 47aafa8 to 9289c2d Compare December 18, 2023 13:54
@debarshiray debarshiray marked this pull request as ready for review December 18, 2023 14:03
@debarshiray debarshiray changed the title [WIP] Fix DNS queries in images with systemd-resolved on hosts without it Fix DNS queries in images with systemd-resolved on hosts without it Dec 18, 2023
@softwarefactory-project-zuul
Copy link
Copy Markdown

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/6c1757ec399b40dfb4f647622a5ef1f5

✔️ unit-test SUCCESS in 6m 51s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 08s
✔️ unit-test-restricted SUCCESS in 5m 55s
✔️ system-test-fedora-rawhide SUCCESS in 33m 22s
✔️ system-test-fedora-39 SUCCESS in 32m 17s
✔️ system-test-fedora-38 SUCCESS in 31m 40s

@debarshiray debarshiray merged commit 9289c2d into containers:main Dec 18, 2023
@debarshiray debarshiray deleted the wip/rishi/issues-1410 branch December 18, 2023 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant