kube-apiserver readiness check: stat /var/lib/microshift/resources/kubeadmin/kubeconfig: no such file or directory

[lcarva]

NOTE: we suspect this is because initialization is half way done sometimes, and on next boot the generation of files is not completed

What happened:

Unable to start microshift via systemd. It panics with the following error:

I0311 17:14:22.716602       1 manager.go:112] Starting kube-apiserver
E0311 17:14:22.716772       1 kube-apiserver.go:225] kube-apiserver readiness check: stat /var/lib/microshift/resources/kubeadmin/kubeconfig: no such file or directory
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x30c257a]

goroutine 306 [running]:
k8s.io/client-go/kubernetes.NewForConfig(0x0, 0xc00100efa8, 0x0, 0x0)
        /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/client-go/kubernetes/clientset.go:395 +0x3a
github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run.func1(0xc000728c60, 0xc0017bc180, 0xc001614000)
        /opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:229 +0xa5
created by github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run
        /opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:222 +0xc8

What you expected to happen:

Expected microshift to start successfully.

How to reproduce it (as minimally and precisely as possible):

1. Follow the steps in the Getting Started docs. Failure occurs on this step: sudo systemctl enable microshift --now

Anything else we need to know?:

A few of these warnings early in start process:

WARNING: The same type, major and minor should not be used for multiple devices.

It's unclear if these are related to the issue.

Environment:

• Microshift version (use microshift version): quay.io/microshift/microshift:latest with digest sha256:efea2a587594be185554ea399f532e4d0b6b483ea9b4842c161098b554f9e54f
• Hardware configuration:
• OS (e.g: cat /etc/os-release): Fedora 35
• Kernel (e.g. uname -a): Linux localhost.localdomain 5.16.12-200.fc35.x86_64 #1 SMP PREEMPT Wed Mar 2 19:06:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• Others:

Relevant Logs

Mar 11 12:14:20 localhost.localdomain systemd[1]: Starting MicroShift Containerized...
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: 
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.750306211 -0500 EST m=+0.073820434 container create ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, io.containers.autoupdate=registry, io.buildah.version=1.23.1, io.openshift.expose-services=, build-date=2021-10-26T13:03:06.167238, io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, PODMAN_SYSTEMD_UNIT=microshift.service, architecture=x86_64, com.redhat.component=ubi8-minimal-container, vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=minimal rhel8, version=8.4, vcs-type=git, name=ubi8-minimal, release=212, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Red Hat, Inc., summary=Provides the latest release of the minimal Red Hat Universal Base Image 8., vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public)
Mar 11 12:14:20 localhost.localdomain oci-umount[26772]: umounthook <debug>: prestart container_id:ea62f7801a7c rootfs:/var/lib/containers/storage/overlay/1263437b45ea09594199bc5b502a2cc2611e06699150471d4ab181db0f6c507b/merged
Mar 11 12:14:20 localhost.localdomain oci-umount[26772]: umounthook <warning>: ea62f7801a7c: path /var/lib/containers/storage from /usr/share/oci-umount/oci-umount.d/crio-umount.conf already exists
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.799207333 -0500 EST m=+0.122721541 container init ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=ubi8-minimal-container, name=ubi8-minimal, vendor=Red Hat, Inc., release=212, io.containers.autoupdate=registry, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, distribution-scope=public, io.openshift.tags=minimal rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, summary=Provides the latest release of the minimal Red Hat Universal Base Image 8., maintainer=Red Hat, Inc., io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, PODMAN_SYSTEMD_UNIT=microshift.service, io.openshift.expose-services=, vcs-type=git, build-date=2021-10-26T13:03:06.167238, io.buildah.version=1.23.1, version=8.4)
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.720222708 -0500 EST m=+0.043736935 image pull  quay.io/microshift/microshift:latest
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.808228662 -0500 EST m=+0.131742908 container start ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-type=git, release=212, io.buildah.version=1.23.1, com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, PODMAN_SYSTEMD_UNIT=microshift.service, distribution-scope=public, build-date=2021-10-26T13:03:06.167238, maintainer=Red Hat, Inc., io.openshift.tags=minimal rhel8, io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, io.containers.autoupdate=registry, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, com.redhat.component=ubi8-minimal-container, name=ubi8-minimal, vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, vendor=Red Hat, Inc., architecture=x86_64, version=8.4, summary=Provides the latest release of the minimal Red Hat Universal Base Image 8.)
Mar 11 12:14:20 localhost.localdomain podman[26716]: ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.909574       1 openshift-apiserver.go:88] starting openshift-apiserver 192.168.86.38, args: [--config=/var/lib/microshift/resources/openshift-apiserver/config/config.yaml]
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.912846       1 run.go:117] Starting Microshift
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.912903       1 run.go:122] Started service-manager
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.912963       1 manager.go:112] Starting etcd
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:125","msg":"configuring peer listeners","listen-peer-urls":["https://192.168.86.38:2380"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:477","msg":"starting with peer TLS","tls-info":"cert = /var/lib/microshift/certs/etcd/etcd-peer.crt, key = /var/lib/microshift/certs/etcd/etcd-peer.key, trusted-ca = /var/lib/microshift/certs/ca-bundle/ca-bundle.crt, client-cert-auth = false, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["https://127.0.0.1:2379","https://192.168.86.38:2379"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:311","msg":"starting an etcd server","etcd-version":"3.4.14","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.16.12","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"localhost.localdomain","data-dir":"/var/lib/microshift/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/microshift/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://192.168.86.38:2380"],"listen-peer-urls":["https://192.168.86.38:2380"],"advertise-client-urls":["https://192.168.86.38:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.86.38:2379"],"listen-metrics-urls":["https://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":false,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":""}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"etcdserver/backend.go:80","msg":"opened backend db","path":"/var/lib/microshift/etcd/member/snap/db","took":"94.193µs"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"etcdserver/raft.go:536","msg":"restarting local member","cluster-id":"1fbf54d7fd7fe23a","local-member-id":"129e664cf22c2231","commit-index":162}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"raft/raft.go:1530","msg":"129e664cf22c2231 switched to configuration voters=()"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"raft/raft.go:700","msg":"129e664cf22c2231 became follower at term 81"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"raft/raft.go:383","msg":"newRaft 129e664cf22c2231 [peers: [], term: 81, commit: 162, applied: 0, lastindex: 162, lastterm: 81]"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"warn","ts":"2022-03-11T17:14:20.916Z","caller":"auth/store.go:1366","msg":"simple token is not cryptographically signed"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.918Z","caller":"etcdserver/quota.go:98","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.919Z","caller":"etcdserver/server.go:803","msg":"starting etcd server","local-member-id":"129e664cf22c2231","local-server-version":"3.4.14","cluster-version":"to_be_decided"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.921Z","caller":"etcdserver/server.go:691","msg":"starting initial election tick advance","election-ticks":10}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.921Z","caller":"raft/raft.go:1530","msg":"129e664cf22c2231 switched to configuration voters=(1341622219669512753)"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.921Z","caller":"membership/cluster.go:392","msg":"added member","cluster-id":"1fbf54d7fd7fe23a","local-member-id":"129e664cf22c2231","added-peer-id":"129e664cf22c2231","added-peer-peer-urls":["https://192.168.86.38:2380"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.922Z","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"1fbf54d7fd7fe23a","local-member-id":"129e664cf22c2231","cluster-version":"3.4"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.922Z","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.4"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.922Z","caller":"embed/etcd.go:726","msg":"starting with client TLS","tls-info":"cert = /var/lib/microshift/certs/etcd/etcd-serving.crt, key = /var/lib/microshift/certs/etcd/etcd-serving.key, trusted-ca = /var/lib/microshift/certs/ca-bundle/ca-bundle.crt, client-cert-auth = false, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.923Z","caller":"embed/etcd.go:592","msg":"serving peer traffic","address":"192.168.86.38:2380"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.923Z","caller":"embed/etcd.go:253","msg":"now serving peer/client/metrics","local-member-id":"129e664cf22c2231","initial-advertise-peer-urls":["https://192.168.86.38:2380"],"listen-peer-urls":["https://192.168.86.38:2380"],"advertise-client-urls":["https://192.168.86.38:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.86.38:2379"],"listen-metrics-urls":["https://127.0.0.1:2381"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.923Z","caller":"embed/etcd.go:799","msg":"serving metrics","address":"https://127.0.0.1:2381"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:923","msg":"129e664cf22c2231 is starting a new election at term 81"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:713","msg":"129e664cf22c2231 became candidate at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:824","msg":"129e664cf22c2231 received MsgVoteResp from 129e664cf22c2231 at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:765","msg":"129e664cf22c2231 became leader at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/node.go:325","msg":"raft.node: 129e664cf22c2231 elected leader 129e664cf22c2231 at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.716Z","caller":"etcdserver/server.go:2037","msg":"published local member to cluster through raft","local-member-id":"129e664cf22c2231","local-member-attributes":"{Name:localhost.localdomain ClientURLs:[https://192.168.86.38:2379]}","request-path":"/0/members/129e664cf22c2231/attributes","cluster-id":"1fbf54d7fd7fe23a","publish-timeout":"7s"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: I0311 17:14:22.716380       1 etcd.go:102] etcd is ready
Mar 11 12:14:22 localhost.localdomain microshift[26768]: I0311 17:14:22.716602       1 manager.go:112] Starting kube-apiserver
Mar 11 12:14:22 localhost.localdomain microshift[26768]: E0311 17:14:22.716772       1 kube-apiserver.go:225] kube-apiserver readiness check: stat /var/lib/microshift/resources/kubeadmin/kubeconfig: no such file or directory
Mar 11 12:14:22 localhost.localdomain microshift[26768]: panic: runtime error: invalid memory address or nil pointer dereference
Mar 11 12:14:22 localhost.localdomain microshift[26768]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x30c257a]
Mar 11 12:14:22 localhost.localdomain microshift[26768]: 
Mar 11 12:14:22 localhost.localdomain microshift[26768]: goroutine 306 [running]:
Mar 11 12:14:22 localhost.localdomain microshift[26768]: k8s.io/client-go/kubernetes.NewForConfig(0x0, 0xc00100efa8, 0x0, 0x0)
Mar 11 12:14:22 localhost.localdomain microshift[26768]:         /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/client-go/kubernetes/clientset.go:395 +0x3a
Mar 11 12:14:22 localhost.localdomain microshift[26768]: github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run.func1(0xc000728c60, 0xc0017bc180, 0xc001614000)
Mar 11 12:14:22 localhost.localdomain microshift[26768]:         /opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:229 +0xa5
Mar 11 12:14:22 localhost.localdomain microshift[26768]: created by github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run
Mar 11 12:14:22 localhost.localdomain microshift[26768]:         /opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:222 +0xc8
Mar 11 12:14:22 localhost.localdomain podman[26789]: 2022-03-11 12:14:22.838498262 -0500 EST m=+0.069568919 container died ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift)
Mar 11 12:14:22 localhost.localdomain podman[26789]: 2022-03-11 12:14:22.868233541 -0500 EST m=+0.099304178 container remove ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, architecture=x86_64, io.openshift.expose-services=, version=8.4, io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, maintainer=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, distribution-scope=public, name=ubi8-minimal, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, vcs-type=git, io.containers.autoupdate=registry, com.redhat.component=ubi8-minimal-container, build-date=2021-10-26T13:03:06.167238, PODMAN_SYSTEMD_UNIT=microshift.service, io.openshift.tags=minimal rhel8, io.buildah.version=1.23.1, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, release=212, summary=Provides the latest release of the minimal Red Hat Universal Base Image 8.)
Mar 11 12:14:22 localhost.localdomain systemd[1]: microshift.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 11 12:14:23 localhost.localdomain systemd[1]: microshift.service: Failed with result 'exit-code'.
Mar 11 12:14:23 localhost.localdomain systemd[1]: Failed to start MicroShift Containerized.

[oglok]

Hello @lcarva

What method are you using to install MicroShift? Is it RPM or containerized version?

I'd like to replicate your environment locally.

Thanks

[lcarva]

@oglok, the containerized version with podman. Thanks!

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[ggiguash]

Fixed by #794

[ggiguash]

/close

[openshift-ci]

@ggiguash: Closing this issue.

Details

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[skylabreddy]

Hi Team,

I am trying to install microshift on RHEL-8.7 and I am following Red Hat documentation

https://access.redhat.com/documentation/en-us/red_hat_build_of_microshift/4.12/html/installing/microshift-install-rpm#microshift-install-system-requirements_microshift-install-rpm

Everything went well, but while I am starting the microshift service not starting. I see below issue.

[root@microshift ~]# systemctl start microshift
Job for microshift.service failed because the service did not take the steps required by its unit configuration.
See "systemctl status microshift.service" and "journalctl -xe" for details.
[root@microshift ~]#

Is any one can help on this issue? thanks in advance.

[dhellmann]

@skylabreddy can you share the output of journalctl -u microshift? It might be large, so you may want to use a separate gist or pastebin.

It would also be useful to know more about the environment where you are running. What sort of host (laptop, VM, etc.)? What networking do you have enabled?

[skylabreddy]

@dhellmann Here is the inputs

1. Microshift running on RHEL-8.7 VM
2. If come to network, i did not enabled any decated network for microshift,

[root@microshift ~]# journalctl -u microshift
-- Logs begin at Fri 2023-03-17 20:54:09 IST, end at Fri 2023-03-17 21:09:48 IST. --
Mar 17 20:54:09 microshift systemd[1]: Starting MicroShift...
Mar 17 20:54:09 microshift microshift[649415]: ??? I0317 20:54:09.958519 649415 run.go:116] Starting MicroShift
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958634 649415 certchains.go:122] [admin-kubeconfig-signer] rotate at: 2032-03-19 10:01:>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958659 649415 certchains.go:122] [admin-kubeconfig-signer admin-kubeconfig-client] rota>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958677 649415 certchains.go:122] [aggregator-signer] rotate at: 2023-11-17 10:01:49 +00>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958693 649415 certchains.go:122] [aggregator-signer aggregator-client] rotate at: 2023->
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958707 649415 certchains.go:122] [etcd-signer] rotate at: 2032-03-19 10:01:52 +0000 UTC
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958721 649415 certchains.go:122] [etcd-signer apiserver-etcd-client] rotate at: 2032-03>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958734 649415 certchains.go:122] [ingress-ca] rotate at: 2032-03-19 10:01:50 +0000 UTC
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958748 649415 certchains.go:122] [ingress-ca router-default-serving] rotate at: 2023-11>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958760 649415 certchains.go:122] [kube-apiserver-external-signer] rotate at: 2032-03-19>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958775 649415 certchains.go:122] [kube-apiserver-external-signer kube-external-serving]>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958788 649415 certchains.go:122] [kube-apiserver-localhost-signer] rotate at: 2032-03-1>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958800 649415 certchains.go:122] [kube-apiserver-localhost-signer kube-apiserver-localh>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958813 649415 certchains.go:122] [kube-apiserver-service-network-signer] rotate at: 203>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958827 649415 certchains.go:122] [kube-apiserver-service-network-signer kube-apiserver->
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958840 649415 certchains.go:122] [kube-apiserver-to-kubelet-signer] rotate at: 2023-11->
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958853 649415 certchains.go:122] [kube-apiserver-to-kubelet-signer kube-apiserver-to-ku>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958867 649415 certchains.go:122] [kube-control-plane-signer] rotate at: 2023-11-17 10:0>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958883 649415 certchains.go:122] [kube-control-plane-signer cluster-policy-controller] >
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958896 649415 certchains.go:122] [kube-control-plane-signer kube-controller-manager] ro>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958909 649415 certchains.go:122] [kube-control-plane-signer kube-scheduler] rotate at: >
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958921 649415 certchains.go:122] [kube-control-plane-signer route-controller-manager] r>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958934 649415 certchains.go:122] [kubelet-signer] rotate at: 2023-11-17 10:01:48 +0000 >
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958948 649415 certchains.go:122] [kubelet-signer kube-csr-signer] rotate at: 2023-11-17>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958963 649415 certchains.go:122] [kubelet-signer kube-csr-signer kubelet-client] rotate>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958978 649415 certchains.go:122] [kubelet-signer kube-csr-signer kubelet-server] rotate>
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958990 649415 certchains.go:122] [service-ca] rotate at: 2032-03-19 10:01:49 +0000 UTC
Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.959003 649415 certchains.go:122] [service-ca route-controller-manager-serving] rotate a>
Mar 17 20:54:09 microshift microshift[649415]: ??? I0317 20:54:09.959205 649415 run.go:127] Started service-manager
Mar 17 20:54:09 microshift microshift[649415]: network-configuration I0317 20:54:09.959290 649415 manager.go:114] Starting network-configuration
Mar 17 20:54:09 microshift microshift[649415]: etcd I0317 20:54:09.959326 649415 manager.go:114] Starting etcd
Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.959+0530","caller":"embed/etcd.go:131","msg":"configuring peer li>
Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.959+0530","caller":"embed/etcd.go:481","msg":"starting with peer >
Mar 17 20:54:09 microshift microshift[649415]: network-configuration I0317 20:54:09.959751 649415 netconfig.go:74] "network-configuration" is ready
Mar 17 20:54:09 microshift microshift[649415]: sysconfwatch-controller I0317 20:54:09.959787 649415 manager.go:114] Starting sysconfwatch-controller
Mar 17 20:54:09 microshift microshift[649415]: sysconfwatch-controller I0317 20:54:09.959811 649415 sysconfwatch_linux.go:89] starting sysconfwatch-contr>
Mar 17 20:54:09 microshift microshift[649415]: sysconfwatch-controller I0317 20:54:09.959827 649415 sysconfwatch_linux.go:95] sysconfwatch-controller is >
Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.960+0530","caller":"embed/etcd.go:139","msg":"configuring client >
Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.960+0530","caller":"embed/etcd.go:309","msg":"starting an etcd se>
Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.961+0530","caller":"etcdserver/backend.go:81","msg":"opened backe>
Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.982+0530","caller":"etcdserver/server.go:532","msg":"No snapshot >
Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.998+0530","caller":"etcdserver/raft.go:483","msg":"restarting loc>
Mar 17 20:54:10 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:10.000+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","m>
Mar 17 20:54:10 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:10.000+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","m>
Mar 17 20:54:10 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:10.000+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","m>
lines 1-47

[dhellmann]

You should not need a dedicated network, but we know the CNI plugin does not work with wifi interfaces, yet. As long as your VM has a default route, you should be fine.

I do not see any obvious errors in the log, but it does not look complete, either.

How is the VM configured? How much RAM and CPU, for example?

[skylabreddy]

@dhellmann CPU=4 and RAM=4 and HDD is 200GB .