a5a76d5 panic: interface conversion: runtime.Object is *unstructured.UnstructuredList, not *unstructured.Unstructured [recovered]

[wking]

In a libvirt cluster I just launched using:

$ openshift-install version
openshift-install v0.1.0-52-gedc4d97104f7fefbe6ce778d18aaf53299f8af59
Terraform v0.11.8

I'm seeing:

[core@wking-bootstrap ~]$ kubectl logs -n openshift-cluster-version bootstrap-cluster-version-operator-wking-bootstrap
I1005 21:39:48.036769       1 start.go:67] ClusterVersionOperator v0.0.0-97-ga5a76d51-dirty
I1005 21:39:48.037010       1 start.go:180] Loading kube client config from path "/etc/kubernetes/kubeconfig"
...
E1005 21:42:10.909970       1 event.go:259] Could not construct reference to: '&v1.ConfigMap{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster-version-operator", GenerateName:"", Namespace:"openshift-cluster-version", SelfLink:"/api/v1/namespaces/openshift-cluster-version/configmaps/cluster-version-operator", UID:"d45bde3c-c8e4-11e8-8408-0214269547a8", ResourceVersion:"9259", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63674371377, loc:(*time.Location)(0x1bf25a0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"wking-bootstrap_efc11fa1-5144-45db-a2cb-568952d64f05\",\"leaseDurationSeconds\":90,\"acquireTime\":\"2018-10-05T21:42:10Z\",\"renewTime\":\"2018-10-05T21:42:10Z\",\"leaderTransitions\":8}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Data:map[string]string(nil), BinaryData:map[string][]uint8(nil)}' due to: 'no kind is registered for the type v1.ConfigMap'. Will not report event: 'Normal' 'LeaderElection' 'wking-bootstrap_efc11fa1-5144-45db-a2cb-568952d64f05 became leader'
...
I1005 21:42:12.267560       1 sync.go:24] Running sync for (servicecertsigner.config.openshift.io/v1alpha1, Kind=ServiceCertSignerOperatorConfig) /instance
E1005 21:42:12.534057       1 memcache.go:147] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I1005 21:42:12.556363       1 sync.go:60] Done syncing for (servicecertsigner.config.openshift.io/v1alpha1, Kind=ServiceCertSignerOperatorConfig) /instance
...
I1005 21:42:14.179926       1 sync.go:60] Done syncing for (/v1, Kind=Service) openshift-operator-lifecycle-manager/package-server
I1005 21:42:14.180040       1 sync.go:24] Running sync for (image.openshift.io/v1, Kind=ImageStream) /
I1005 21:42:14.336201       1 request.go:485] Throttling request took 155.956498ms, request: GET:https://wking-api.installer.testing:6443/apis/image.openshift.io/v1/imagestreams
I1005 21:42:14.349166       1 cvo.go:201] Finished syncing operator "openshift-cluster-version/cluster-version-operator" (3.337348292s)
E1005 21:42:14.349428       1 runtime.go:66] Observed a panic: &runtime.TypeAssertionError{interfaceString:"runtime.Object", concreteString:"*unstructured.UnstructuredList", assertedString:"*unstructured.Unstructured", missingMethod:""} (interface conversion: runtime.Object is *unstructured.UnstructuredList, not *unstructured.Unstructured)
/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/iface.go:252
/usr/local/go/src/runtime/iface.go:262
/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/dynamic/simple.go:197
/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/internal/generic.go:31
/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/internal/generic.go:88
/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync.go:51
/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203
/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync.go:33
/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:243
/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:115
/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:173
/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:162
/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:146
/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:2361
panic: interface conversion: runtime.Object is *unstructured.UnstructuredList, not *unstructured.Unstructured [recovered]
	panic: interface conversion: runtime.Object is *unstructured.UnstructuredList, not *unstructured.Unstructured

goroutine 76 [running]:
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x11562c0, 0xc4202d3e00)
	/usr/local/go/src/runtime/panic.go:502 +0x229
github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/dynamic.(*dynamicResourceClient).Get(0xc420e173b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/client-go/dynamic/simple.go:197 +0x90f
github.com/openshift/cluster-version-operator/pkg/cvo/internal.applyUnstructured(0x13e3760, 0xc420e173b0, 0xc4200a6ca0, 0xc4200a6ca0, 0x0, 0x0, 0x13e3760)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/internal/generic.go:31 +0x99
github.com/openshift/cluster-version-operator/pkg/cvo/internal.(*genericBuilder).Do(0xc420576180, 0xc420405b80, 0x29f)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/internal/generic.go:88 +0x72
github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).syncUpdatePayload.func1(0xa, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync.go:51 +0x241
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff(0x2540be400, 0x3ff4cccccccccccd, 0x0, 0x3, 0xc4207e5b20, 0x2c0, 0xc42041ef00)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203 +0x9c
github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).syncUpdatePayload(0xc42043cb00, 0xc420481380, 0xc4205511a0, 0x3b, 0xc4205511a0)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/sync.go:33 +0x749
github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).sync(0xc42043cb00, 0xc4203d4440, 0x32, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:243 +0x49a
github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).(github.com/openshift/cluster-version-operator/pkg/cvo.sync)-fm(0xc4203d4440, 0x32, 0xc4203e3b00, 0x10e38c0)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:115 +0x3e
github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).processNextWorkItem(0xc42043cb00, 0xc4203d2800)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:173 +0xe0
github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).worker(0xc42043cb00)
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:162 +0x2b
github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).(github.com/openshift/cluster-version-operator/pkg/cvo.worker)-fm()
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:146 +0x2a
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc42026fae0)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc42026fae0, 0x3b9aca00, 0x0, 0x1, 0xc42008c900)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc42026fae0, 0x3b9aca00, 0xc42008c900)
	/go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/openshift/cluster-version-operator/pkg/cvo.(*Operator).Run
	/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/cvo.go:146 +0x1d0

[abhinavdahiya]

@wking
https://github.com/operator-framework/operator-lifecycle-manager/blob/0cb39f1e45adbd979a0210849fdea6f06ecbd36f/manifests/30_14-imagestream.yaml
Doesn't have a name specified, therefore the client-go dynamic client gets a list of objects (https://github.com/kubernetes/client-go/blob/9c81604894197dbfe4424c19193068dc5211db9d/dynamic/simple.go#L321) rather than one single object.

And therefore panics here https://github.com/kubernetes/client-go/blob/9c81604894197dbfe4424c19193068dc5211db9d/dynamic/simple.go#L231

[wking]

Is there a way to adjust the CVO to catch this panic and log a complaint about the busted manifest? As it stands, it's not obvious to me how you traced this to https://github.com/operator-framework/operator-lifecycle-manager/blob/0cb39f1e45adbd979a0210849fdea6f06ecbd36f/manifests/30_14-imagestream.yaml#L17 ;).

[wking]

Also, will openshift/origin#21187 catch this sort of thing as well as invalid JSON? I dunno if it's just checking "this is not valid YAML", or if it's also checking "... that the CVO won't choke on for Kubernetes-specific reasons".

[smarterclayton]

No, we’re not checking for schema, just parsing. This needs the gating PR

That said, it should be be checking for name to not make something unstructured list. The only thing with unstructured list should be objects with Kind ending in List with a valid “items” array.

[smarterclayton]

Also, no one should be type casting to unstructured.Unstructured. Cast to runtime.Unstructured only.

[wking]

That said, it should be be checking for name to not make something unstructured list.

That sounds like what #33 is doing.