[BUG] can not run demo Deploying AMQ Broker on MicroShift

[WangFengtu1996]

What happened?

• when I try the demo, I can not access the amqp service in order to execute demo for providing producer and consumer.
• The pod named amq-broker is not running status. It's status is ImagePullBackOff.
• The more detail log

  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  59m                    default-scheduler  Successfully assigned amq-broker/amq-broker-9c78cb94-2p4sh to fedora
  Normal   Pulling    58m (x4 over 59m)      kubelet            Pulling image "registry.redhat.io/amq7/amq-broker:7.8"
  Warning  Failed     58m (x4 over 59m)      kubelet            Failed to pull image "registry.redhat.io/amq7/amq-broker:7.8": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
  Warning  Failed     58m (x4 over 59m)      kubelet            Error: ErrImagePull
  Warning  Failed     58m (x6 over 59m)      kubelet            Error: ImagePullBackOff
  Normal   BackOff    4m45s (x233 over 59m)  kubelet            Back-off pulling image "registry.redhat.io/amq7/amq-broker:7.8"

• But I pull manually image podman pull registry.redhat.io/amq7/amq-broker:7.8 after login is right.

[user@fedora ~]$ podman images
REPOSITORY                          TAG         IMAGE ID      CREATED      SIZE
registry.redhat.io/amq7/amq-broker  7.8         69cd146bb6a8  2 weeks ago  680 MB

• I follow the doc step by step , anyone would give me some guide ?
• Thanks for your time.

Environment

• MicroShift version (use microshift version):
  MicroShift Version: 4.8.0-0.microshift-2022-04-20-141053
  Base OKD Version: 4.8.0-0.okd-2021-10-10-030117

• Hardware configuration:

• OS (e.g: cat /etc/os-release):
  fedora 36 vmware x86

• Kernel (e.g. uname -a):
  Linux fedora 5.19.4-200.fc36.x86_64 Init #1 SMP PREEMPT_DYNAMIC Thu Aug 25 17:42:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Relevant logs

[ggiguash]

@TUT-jiayou, do you have OCP pull secret configured for CRI-O as described in the Runtime Prerequisites?

Also, make sure to configure CRI-O to use that file.

$ sudo grep openshift-pull-secret /etc/crio/crio.conf.d/microshift.conf 
global_auth_file="/etc/crio/openshift-pull-secret"

[WangFengtu1996]

@ggiguash

• I also can not get container image.

oc describe pod  amq-broker-9c78cb94-jmxrk -n amq-broker

Name:         amq-broker-9c78cb94-jmxrk
Namespace:    amq-broker
Priority:     0
Node:         fedora/192.168.10.128
Start Time:   Tue, 13 Sep 2022 10:13:18 +0800
Labels:       app=amq-broker
              pod-template-hash=9c78cb94
Annotations:  <none>
Status:       Pending
IP:           10.42.0.13
IPs:
  IP:           10.42.0.13
Controlled By:  ReplicaSet/amq-broker-9c78cb94
Containers:
  amq-broker:
    Container ID:
    Image:          registry.redhat.io/amq7/amq-broker:7.8
    Image ID:
    Ports:          61616/TCP, 8161/TCP, 5672/TCP, 1883/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      AMQ_USER:      admin
      AMQ_PASSWORD:  admin
      AMQ_PROTOCOL:  amqp,mqtt
      AMQ_DATA_DIR:  /data
    Mounts:
      /data from amq-broker-datavol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rsdtj (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  amq-broker-datavol:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  amq-broker-pvc
    ReadOnly:   false
  kube-api-access-rsdtj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  15m                  default-scheduler  Successfully assigned amq-broker/amq-broker-9c78cb94-jmxrk to fedora
  Normal   Pulling    13m (x4 over 15m)    kubelet            Pulling image "registry.redhat.io/amq7/amq-broker:7.8"
  Warning  Failed     13m (x4 over 15m)    kubelet            Failed to pull image "registry.redhat.io/amq7/amq-broker:7.8": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
  Warning  Failed     13m (x4 over 15m)    kubelet            Error: ErrImagePull
  Warning  Failed     13m (x6 over 15m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff    13m (x7 over 15m)    kubelet            Back-off pulling image "registry.redhat.io/amq7/amq-broker:7.8"
  Normal   Pulling    10m (x4 over 11m)    kubelet            Pulling image "registry.redhat.io/amq7/amq-broker:7.8"
  Warning  Failed     10m (x4 over 11m)    kubelet            Failed to pull image "registry.redhat.io/amq7/amq-broker:7.8": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
  Warning  Failed     10m (x4 over 11m)    kubelet            Error: ErrImagePull
  Warning  Failed     9m29s (x6 over 11m)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    90s (x41 over 11m)   kubelet            Back-off pulling image "registry.redhat.io/amq7/amq-broker:7.8"

• I download the pull secret txt file and move to /etc/crio/openshift-pull-secret
  
• change mode sudo chmod 600 /etc/crio/openshift-pull-secret
• Add global_auth_file="/etc/crio/openshift-pull-secret" to file /etc/crio/crio.conf.d/microshift.conf .
• restart systemd service sudo systemctl restart crio sudo systemctl restart microshift.
• About the service crio and microshift log

[user@fedora ~]$ journalctl -u microshift -f
9月 13 10:29:54 fedora microshift[24643]: W0913 10:29:54.705286   24643 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:29:54 fedora microshift[24643]: W0913 10:29:54.705342   24643 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:29:54 fedora microshift[24643]: W0913 10:29:54.705483   24643 patch_genericapiserver.go:123] Request to "/apis/project.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:29:56 fedora microshift[24643]: W0913 10:29:56.705606   24643 patch_genericapiserver.go:123] Request to "/openapi/v2" (source IP 192.168.10.128:53960, user agent "") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:29:57 fedora microshift[24643]: W0913 10:29:57.597303   24643 patch_genericapiserver.go:123] Request to "/openapi/v2" (source IP 192.168.10.128:53968, user agent "") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:29:58 fedora microshift[24643]: W0913 10:29:58.495262   24643 patch_genericapiserver.go:123] Request to "/openapi/v2" (source IP 192.168.10.128:53974, user agent "") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:29:59 fedora microshift[24643]: W0913 10:29:59.410691   24643 patch_genericapiserver.go:123] Request to "/openapi/v2" (source IP 192.168.10.128:53988, user agent "") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:00 fedora microshift[24643]: W0913 10:30:00.291747   24643 patch_genericapiserver.go:123] Request to "/openapi/v2" (source IP 192.168.10.128:53998, user agent "") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:01 fedora microshift[24643]: W0913 10:30:01.194540   24643 patch_genericapiserver.go:123] Request to "/openapi/v2" (source IP 192.168.10.128:54006, user agent "") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:01 fedora microshift[24643]: E0913 10:30:01.912486   24643 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"amq-broker\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/amq7/amq-broker:7.8\\\"\"" pod="amq-broker/amq-broker-9c78cb94-jmxrk" podUID=a0d624df-a317-4686-a686-f8c4d39e0324
9月 13 10:30:15 fedora microshift[24643]: W0913 10:30:15.832786   24643 patch_genericapiserver.go:123] Request to "/apis/user.openshift.io/v1" (source IP 192.168.10.128:53844, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:resourcequota-controller") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:15 fedora microshift[24643]: W0913 10:30:15.833615   24643 patch_genericapiserver.go:123] Request to "/apis/image.openshift.io/v1" (source IP 192.168.10.128:53968, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:resourcequota-controller") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:15 fedora microshift[24643]: W0913 10:30:15.833867   24643 patch_genericapiserver.go:123] Request to "/apis/project.openshift.io/v1" (source IP 192.168.10.128:53998, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:resourcequota-controller") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:15 fedora microshift[24643]: W0913 10:30:15.834043   24643 patch_genericapiserver.go:123] Request to "/apis/route.openshift.io/v1" (source IP 192.168.10.128:53960, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:resourcequota-controller") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:15 fedora microshift[24643]: W0913 10:30:15.834184   24643 patch_genericapiserver.go:123] Request to "/apis/apps.openshift.io/v1" (source IP 192.168.10.128:53988, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:resourcequota-controller") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:15 fedora microshift[24643]: W0913 10:30:15.834249   24643 patch_genericapiserver.go:123] Request to "/apis/build.openshift.io/v1" (source IP 192.168.10.128:53974, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:resourcequota-controller") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:15 fedora microshift[24643]: W0913 10:30:15.834807   24643 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.10.128:54006, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:resourcequota-controller") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:16 fedora microshift[24643]: W0913 10:30:16.134072   24643 patch_genericapiserver.go:123] Request to "/apis/apps.openshift.io/v1" (source IP 192.168.10.128:53988, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:16 fedora microshift[24643]: W0913 10:30:16.134961   24643 patch_genericapiserver.go:123] Request to "/apis/project.openshift.io/v1" (source IP 192.168.10.128:53998, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:16 fedora microshift[24643]: W0913 10:30:16.135142   24643 patch_genericapiserver.go:123] Request to "/apis/build.openshift.io/v1" (source IP 192.168.10.128:53974, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:16 fedora microshift[24643]: W0913 10:30:16.136004   24643 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.10.128:54006, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:16 fedora microshift[24643]: W0913 10:30:16.136051   24643 patch_genericapiserver.go:123] Request to "/apis/route.openshift.io/v1" (source IP 192.168.10.128:53960, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:16 fedora microshift[24643]: W0913 10:30:16.137364   24643 patch_genericapiserver.go:123] Request to "/apis/user.openshift.io/v1" (source IP 192.168.10.128:53844, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:16 fedora microshift[24643]: W0913 10:30:16.137398   24643 patch_genericapiserver.go:123] Request to "/apis/image.openshift.io/v1" (source IP 192.168.10.128:53968, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce/system:serviceaccount:kube-system:generic-garbage-collector") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:16 fedora microshift[24643]: E0913 10:30:16.912059   24643 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"amq-broker\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/amq7/amq-broker:7.8\\\"\"" pod="amq-broker/amq-broker-9c78cb94-jmxrk" podUID=a0d624df-a317-4686-a686-f8c4d39e0324
9月 13 10:30:20 fedora microshift[24643]: W0913 10:30:20.111698   24643 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1/templateinstances" (source IP 192.168.10.128:54006, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:23 fedora microshift[24643]: W0913 10:30:23.719864   24643 patch_genericapiserver.go:123] Request to "/apis/route.openshift.io/v1/routes" (source IP 192.168.10.128:53960, user agent "microshift/v1.21.1 (linux/amd64) kubernetes/b09a9ce") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704233   24643 patch_genericapiserver.go:123] Request to "/apis/user.openshift.io/v1" (source IP 192.168.10.128:33676, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704344   24643 patch_genericapiserver.go:123] Request to "/apis/user.openshift.io/v1" (source IP 192.168.10.128:33676, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704438   24643 patch_genericapiserver.go:123] Request to "/apis/user.openshift.io/v1" (source IP 192.168.10.128:33676, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704467   24643 patch_genericapiserver.go:123] Request to "/apis/route.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704493   24643 patch_genericapiserver.go:123] Request to "/apis/route.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704765   24643 patch_genericapiserver.go:123] Request to "/apis/image.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704830   24643 patch_genericapiserver.go:123] Request to "/apis/image.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704906   24643 patch_genericapiserver.go:123] Request to "/apis/user.openshift.io/v1" (source IP 192.168.10.128:33676, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704931   24643 patch_genericapiserver.go:123] Request to "/apis/user.openshift.io/v1" (source IP 192.168.10.128:33676, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704468   24643 patch_genericapiserver.go:123] Request to "/apis/route.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704767   24643 patch_genericapiserver.go:123] Request to "/apis/build.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.705407   24643 patch_genericapiserver.go:123] Request to "/apis/image.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.705478   24643 patch_genericapiserver.go:123] Request to "/apis/image.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704783   24643 patch_genericapiserver.go:123] Request to "/apis/route.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.705535   24643 patch_genericapiserver.go:123] Request to "/apis/image.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.704443   24643 patch_genericapiserver.go:123] Request to "/apis/route.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.705707   24643 patch_genericapiserver.go:123] Request to "/apis/build.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.705791   24643 patch_genericapiserver.go:123] Request to "/apis/build.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.705831   24643 patch_genericapiserver.go:123] Request to "/apis/build.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.705881   24643 patch_genericapiserver.go:123] Request to "/apis/build.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.705968   24643 patch_genericapiserver.go:123] Request to "/apis/apps.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.705985   24643 patch_genericapiserver.go:123] Request to "/apis/apps.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706192   24643 patch_genericapiserver.go:123] Request to "/apis/apps.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706254   24643 patch_genericapiserver.go:123] Request to "/apis/apps.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706348   24643 patch_genericapiserver.go:123] Request to "/apis/project.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706413   24643 patch_genericapiserver.go:123] Request to "/apis/apps.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706428   24643 patch_genericapiserver.go:123] Request to "/apis/project.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706453   24643 patch_genericapiserver.go:123] Request to "/apis/project.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706491   24643 patch_genericapiserver.go:123] Request to "/apis/project.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706548   24643 patch_genericapiserver.go:123] Request to "/apis/project.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706646   24643 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706683   24643 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706706   24643 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706729   24643 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:24 fedora microshift[24643]: W0913 10:30:24.706755   24643 patch_genericapiserver.go:123] Request to "/apis/template.openshift.io/v1" (source IP 192.168.10.128:38756, user agent "Go-http-client/2.0") before server is ready, possibly a sign for a broken load balancer setup.
9月 13 10:30:27 fedora microshift[24643]: E0913 10:30:27.911736   24643 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"amq-broker\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/amq7/amq-broker:7.8\\\"\"" pod="amq-broker/amq-broker-9c78cb94-jmxrk" podUID=a0d624df-a317-4686-a686-f8c4d39e0324

Thanks for your time. Please give me more guide.

[WangFengtu1996]

@ggiguash

• I change mode sudo chmod 644 /etc/crio/openshift-pull-secret
• restart service crio microshift
• log info oc describe pod amq-broker-9c78cb94-m7xnb -n amq-broker

[user@fedora ~]$ oc describe pod amq-broker-9c78cb94-m7xnb -n amq-broker
Name:           amq-broker-9c78cb94-m7xnb
Namespace:      amq-broker
Priority:       0
Node:           fedora/192.168.10.128
Start Time:     Tue, 13 Sep 2022 11:49:19 +0800
Labels:         app=amq-broker
                pod-template-hash=9c78cb94
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/amq-broker-9c78cb94
Containers:
  amq-broker:
    Container ID:
    Image:          registry.redhat.io/amq7/amq-broker:7.8
    Image ID:
    Ports:          61616/TCP, 8161/TCP, 5672/TCP, 1883/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      AMQ_USER:      admin
      AMQ_PASSWORD:  admin
      AMQ_PROTOCOL:  amqp,mqtt
      AMQ_DATA_DIR:  /data
    Mounts:
      /data from amq-broker-datavol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5jg29 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  amq-broker-datavol:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  amq-broker-pvc
    ReadOnly:   false
  kube-api-access-5jg29:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  42s   default-scheduler  Successfully assigned amq-broker/amq-broker-9c78cb94-m7xnb to fedora
  Normal  Pulling    42s   kubelet            Pulling image "registry.redhat.io/amq7/amq-broker:7.8"
[user@fedora ~]$ oc describe pod amq-broker-9c78cb94-m7xnb -n amq-broker
Name:         amq-broker-9c78cb94-m7xnb
Namespace:    amq-broker
Priority:     0
Node:         fedora/192.168.10.128
Start Time:   Tue, 13 Sep 2022 11:49:19 +0800
Labels:       app=amq-broker
              pod-template-hash=9c78cb94
Annotations:  <none>
Status:       Pending
IP:           10.42.0.14
IPs:
  IP:           10.42.0.14
Controlled By:  ReplicaSet/amq-broker-9c78cb94
Containers:
  amq-broker:
    Container ID:
    Image:          registry.redhat.io/amq7/amq-broker:7.8
    Image ID:
    Ports:          61616/TCP, 8161/TCP, 5672/TCP, 1883/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Environment:
      AMQ_USER:      admin
      AMQ_PASSWORD:  admin
      AMQ_PROTOCOL:  amqp,mqtt
      AMQ_DATA_DIR:  /data
    Mounts:
      /data from amq-broker-datavol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5jg29 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  amq-broker-datavol:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  amq-broker-pvc
    ReadOnly:   false
  kube-api-access-5jg29:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  43s   default-scheduler  Successfully assigned amq-broker/amq-broker-9c78cb94-m7xnb to fedora
  Normal   Pulling    43s   kubelet            Pulling image "registry.redhat.io/amq7/amq-broker:7.8"
  Warning  Failed     1s    kubelet            Failed to pull image "registry.redhat.io/amq7/amq-broker:7.8": rpc error: code = Unknown desc = error pinging docker registry registry.redhat.io: Get "https://registry.redhat.io/v2/": dial tcp 23.78.216.142:443: connect: connection refused
  Warning  Failed     1s    kubelet            Error: ErrImagePull
  Normal   BackOff    0s    kubelet            Back-off pulling image "registry.redhat.io/amq7/amq-broker:7.8"
  Warning  Failed     0s    kubelet            Error: ImagePullBackOff
[user@fedora ~]$ oc describe pod amq-broker-9c78cb94-m7xnb -n amq-broker.yaml [C^C
[user@fedora ~]$ oc describe pod amq-broker-9c78cb94-m7xnb -n amq-broker
Name:         amq-broker-9c78cb94-m7xnb
Namespace:    amq-broker
Priority:     0
Node:         fedora/192.168.10.128
Start Time:   Tue, 13 Sep 2022 11:49:19 +0800
Labels:       app=amq-broker
              pod-template-hash=9c78cb94
Annotations:  <none>
Status:       Pending
IP:           10.42.0.14
IPs:
  IP:           10.42.0.14
Controlled By:  ReplicaSet/amq-broker-9c78cb94
Containers:
  amq-broker:
    Container ID:
    Image:          registry.redhat.io/amq7/amq-broker:7.8
    Image ID:
    Ports:          61616/TCP, 8161/TCP, 5672/TCP, 1883/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      AMQ_USER:      admin
      AMQ_PASSWORD:  admin
      AMQ_PROTOCOL:  amqp,mqtt
      AMQ_DATA_DIR:  /data
    Mounts:
      /data from amq-broker-datavol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5jg29 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  amq-broker-datavol:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  amq-broker-pvc
    ReadOnly:   false
  kube-api-access-5jg29:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  65s                default-scheduler  Successfully assigned amq-broker/amq-broker-9c78cb94-m7xnb to fedora
  Warning  Failed     23s                kubelet            Failed to pull image "registry.redhat.io/amq7/amq-broker:7.8": rpc error: code = Unknown desc = error pinging docker registry registry.redhat.io: Get "https://registry.redhat.io/v2/": dial tcp 23.78.216.142:443: connect: connection refused
  Warning  Failed     23s                kubelet            Error: ErrImagePull
  Normal   BackOff    22s                kubelet            Back-off pulling image "registry.redhat.io/amq7/amq-broker:7.8"
  Warning  Failed     22s                kubelet            Error: ImagePullBackOff
  Normal   Pulling    10s (x2 over 65s)  kubelet            Pulling image "registry.redhat.io/amq7/amq-broker:7.8"

[WangFengtu1996]

• I also try to create a secret follow link, but I can not pull image.
• oc create secret generic pull-secret -n amq-broker --type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=/etc/crio/openshift-pull-secret

[WangFengtu1996]

• I also login by oc command line, but can not to login in web size. When I use podman login, I can login successfully.

oc login registry.redhat.io

error: couldn't get https://registry.redhat.io:443/.well-known/oauth-authorization-server: unexpected response status 404

[ggiguash]

@TUT-jiayou, can you confirm that the following sequence works for you, i.e. you're able to pull the AMQ image using crictl?

sudo crictl pull registry.redhat.io/amq7/amq-broker:7.8

If the command fails, you can look at the journal for hints on what's wrong

sudo journalctl -n 20 --no-pager -xu crio 

[WangFengtu1996]

@ggiguash thks.

• I check the crio config file/etc/crio/crio.conf.d/microshift.conf, modify the file followed by crio doc

[crio.network]
# cbr0 is the name configured by flannel in /etc/cni/net.d/ config file
# by declaring this crio will wait until that network is configured.
cni_default_network = "cbr0"
# rhel8 crio is configured to only look at /usr/libexec/cni, we override that here
plugin_dirs = [
        "/usr/libexec/cni",
        "/opt/cni/bin"
]

[crio.image]
global_auth_file = "/etc/crio/openshift-pull-secret"

[WangFengtu1996]

Thanks for your time @ggiguash