[BUG] plugin/forward: not an IP address or file: "/run/systemd/resolve/resolv.conf"

[kalloa]

What happened?

pod/dns-default doesn't start, see:

[cloud-user@microshift ~]$ oc get all -A
NAMESPACE                  NAME                                           READY   STATUS             RESTARTS         AGE
openshift-dns              pod/dns-default-8cqst                          1/2     CrashLoopBackOff   16 (2m18s ago)   30m

What did you expect to happen?

pod/dns-default starts normally

How to reproduce it (as minimally and precisely as possible)?

1. Installing Microshift v4.12 as describe here: https://access.redhat.com/documentation/en-us/red_hat_build_of_microshift/4.12/html/installing/microshift-install-rpm#installing-microshift-from-rpm-package_microshift-install-rpm

Anything else we need to know?

On the host:

[cloud-user@microshift ~]$ cat /run/systemd/resolve/resolv.conf
nameserver 2.2.2.2
nameserver 4.4.2.2
nameserver 8.8.8.8
# Too many DNS servers configured, the following entries may be ignored.
nameserver 1.1.1.1
search example.abc.com

[cloud-user@microshift ~]$ cat /etc/resolv.conf
; Created by cloud-init on instance boot automatically, do not edit.
;
nameserver 2.2.2.2
nameserver 4.4.2.2
nameserver 8.8.8.8

Environment

• MicroShift version (use microshift version):
  MicroShift Version: 4.12.19
  Base OCP Version: 4.12.19

• Hardware configuration:
  2core 4GB ram KVM virtual machine

• OS (e.g: cat /etc/os-release):
  NAME="Red Hat Enterprise Linux"
  VERSION="8.7 (Ootpa)"
  ID="rhel"
  ID_LIKE="fedora"
  VERSION_ID="8.7"
  PLATFORM_ID="platform:el8"
  PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
  HOME_URL="https://www.redhat.com/"
  DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"
  REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
  REDHAT_BUGZILLA_PRODUCT_VERSION=8.7
  REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
  REDHAT_SUPPORT_PRODUCT_VERSION="8.7"

• Kernel (e.g. uname -a):
  Linux microshift.example.abc.com 4.18.0-425.13.1.el8_7.x86_64 Init #1 SMP Thu Feb 2 13:01:45 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

• Others:

Relevant logs

[cloud-user@microshift ~]$ oc -n openshift-dns get pods
NAME                  READY   STATUS             RESTARTS      AGE
dns-default-8cqst     1/2     CrashLoopBackOff   18 (9s ago)   38m
node-resolver-s55pr   1/1     Running            2             47m
[cloud-user@microshift ~]$ oc -n openshift-dns logs dns-default-8cqst
Defaulted container "dns" out of: dns, kube-rbac-proxy
plugin/forward: not an IP address or file: "/run/systemd/resolve/resolv.conf"

oc -n openshift-dns describe pod/dns-default-6p2mw
...
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Normal   Scheduled         68s                default-scheduler  Successfully assigned openshift-dns/dns-default-6p2mw to microshift.example.abc.com
  Normal   Pulled            68s                kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a248d1f647d437307ffeaa8a6ff469ab777becc62b6d0be1da0fe01c22f2a16" already present on machine
  Normal   Created           68s                kubelet            Created container kube-rbac-proxy
  Normal   Started           68s                kubelet            Started container kube-rbac-proxy
  Warning  BackOff           58s (x4 over 66s)  kubelet            Back-off restarting failed container
  Normal   Pulled            47s (x3 over 68s)  kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830f1764d8544d4c01b47142167f9a6fe5b2edd1c34f944cd6c9ff79b268d580" already present on machine
  Warning  DNSConfigForming  46s (x9 over 68s)  kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 2.2.2.2 4.4.2.2 8.8.8.8
  Normal   Created           46s (x3 over 68s)  kubelet            Created container dns
  Normal   Started           46s (x3 over 68s)  kubelet            Started container dns

^^ this "Nameserver limits were exceeded" is just a warning so I guess it can be ignored.
Could you please help me with it?

[ggiguash]

@kalloa , is there any reason you're using a Dev Preview 4.12 version of MicroShift?
We have released MicroShift 4.13 in Tech Preview, so can you use it instead on RHEL 9.2?

[kalloa]

@kalloa , is there any reason you're using a Dev Preview 4.12 version of MicroShift? We have released MicroShift 4.13 in Tech Preview, so can you use it instead on RHEL 9.2?

Hi, in the meantime I've migrated to MicroShift 4.13 where I don't see this issue anymore, so let's close this.
Thank you!

[ggiguash]

@kalloa , thank you for confirming!