operator-sdk run --olm reports installation failure after successful installation

[aharbis]

Bug Report

What did you do?

operator-sdk run --olm \
  --olm-namespace=openshift-operator-lifecycle-manager \
  --manifests=deploy/olm-catalog/my-operator \
  --operator-version=0.9.0 \
  --operator-namespace=my-operator-ns

What did you expect to see?

Successful installation.

What did you see instead? Under which circumstances?

The log output from operator-sdk run --olm shows a fatal error stating the operator did not install successully, the the log(s) right before this error show phase: Succeeded:

INFO[0000] Waiting for ClusterServiceVersion "my-operator-ns/my-operator.v0.9.0" to reach 'Succeeded' phase
INFO[0000]   Waiting for ClusterServiceVersion "my-operator-ns/my-operator.v0.9.0" to appear
INFO[0003]   Found ClusterServiceVersion "my-operator-ns/my-operator.v0.9.0" phase: Pending
INFO[0004]   Found ClusterServiceVersion "my-operator-ns/my-operator.v0.9.0" phase: InstallReady
INFO[0005]   Found ClusterServiceVersion "my-operator-ns/my-operator.v0.9.0" phase: Installing
INFO[0022]   Found ClusterServiceVersion "my-operator-ns/my-operator.v0.9.0" phase: Succeeded
FATA[0022] Failed to run operator using OLM: operator my-operator did not install successfully

In addition to this, I can verify in the OCP dashboard as well as via oc that my operator deployment is created and Ready in the cluster:

$ oc get deploy
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
my-operator          1/1     1            1           102s

Looking through my operator pod logs, everything looks fine. I see no errors. This seems to be an operator-sdk error being thrown, as even in the OCP dashboard I cannot find any source of error for the operator installation.

Environment

• operator-sdk version:

$ operator-sdk version
operator-sdk version: "v0.17.0", commit: "2fd7019f856cdb6f6618e2c3c80d15c3c79d1b6c", kubernetes version: "unknown", go version: "go1.13.10 linux/amd64"

• go version:

$ go version
go version go1.13.8 linux/amd64

• Kubernetes version information:

$ oc version
Client Version: openshift-clients-4.3.0-201910250623-88-g6a937dfe
Server Version: 4.3.0
Kubernetes Version: v1.16.2

• Kubernetes cluster kind: OCP 4.3

• Are you writing your operator in ansible, helm, or go? Go

Possible Solution

Additional context

[kraman]

I noticed the same issue. Operator is running fine and csv is in succeeded phase but operator-sdk reports failure to deploy

[estroz]

/assign

[estroz]

This issue has something to do with how we retrieve resource statuses for the installed operator. Packaged objects might exist that aren't being created but are still checked for status. @aharbis @kraman can you link your packages or full operator source?

[aharbis]

@estroz Unfortunately we cannot, our operators are closed source currently.

Can you point to any specific requirements that our CRD would need to implement for this to work? What is meant by resource status of the installed operator?

[estroz]

After waiting for the CSV to install, run --olm will check if all objects in a package exist in the cluster and have positive statuses. In pseudocode:

infos, errors = $(kubectl get -f 0.9.0/*.yaml)
if errors is not empty {
	return "operator has resource errors"
}
if infos contains an error {
	return "operator did not install successfully"
}

It sounds like one of the object manifests in your 0.9.0 package is not being created by OLM. Are there any manifests other than CRDs and the CSV in 0.9.0/?

Also there should be some output following the last log line like:

FATA[0022] Failed to run operator using OLM: operator my-operator did not install successfully
NAME		NAMESPACE	KIND		STATUS
my-object	default		SomeKind	Failed to create
...

Did output that looked like that print?

[aharbis]

@estroz The only files in my 0.9.0/ directory are the CSV and the CRD.

Here's an example of the last log line from my output:

FATA[0079] Failed to run operator using OLM: operator my-operator did not install successfully
NAME                              NAMESPACE          KIND                        STATUS
my-operator.v0.9.0                my-operator-olm    ClusterServiceVersion       Installed
mycrdgroup.company.com            my-operator-olm    CustomResourceDefinition    Installed

It looks like from the output that both the CSV and CRD are installed correctly. Additionally I can confirm from the cluster that both are installed successfully, as I can create CR instances of my CRD type through OLM in the console.

[estroz]

@aharbis I will look into why status is reporting errors erroneously. Hopefully this isn’t blocking you from using the command.