operator reports status Failed when operand version doesn't match operator version

[sjenning]

For debugging/testing, I built a custom release image, overriding the machine-config-controller image pull spec

oc adm release new -s https://api.ci.openshift.org/ --from-image-stream=origin-v4.0 -n openshift --to-image=quay.io/sjenning/origin-release:v4.0 --to-image-base=registry.svc.ci.openshift.org/openshift/origin-v4.0:cluster-version-operator machine-config-controller=quay.io/sjenning/machine-config-controller:latest

This results in a failure of the MCO

$ oc get clusteroperators machine-config -oyaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: 2019-02-13T16:39:19Z
  generation: 1
  name: machine-config
  resourceVersion: "10148"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/machine-config
  uid: e905907c-2fad-11e9-89a3-02f7cc37a53c
spec: {}
status:
  conditions:
  - lastTransitionTime: 2019-02-13T16:39:19Z
    status: "False"
    type: Available
  - lastTransitionTime: 2019-02-13T16:39:19Z
    message: Progressing towards 3.11.0-619-g975a0c93-dirty
    status: "True"
    type: Progressing
  - lastTransitionTime: 2019-02-13T16:50:03Z
    message: 'Failed when progressing towards 3.11.0-619-g975a0c93-dirty because:
      error syncing: timed out waiting for the condition during syncRequiredMachineConfigPools:
      pool master has not progressed to latest configuration: controller version mismatch
      for master-235e3f67d027fee30a461ca75f1f2914 expected 3.11.0-619-g975a0c93-dirty
      has 3.11.0-614-gc3638603-dirty'
    reason: 'error syncing: timed out waiting for the condition during syncRequiredMachineConfigPools:
      pool master has not progressed to latest configuration: controller version mismatch
      for master-235e3f67d027fee30a461ca75f1f2914 expected 3.11.0-619-g975a0c93-dirty
      has 3.11.0-614-gc3638603-dirty'
    status: "True"
    type: Failing
  extension: {}
  relatedObjects: null
  versions: null

It seems that the operator expects each operand to match its own version. This should not be a constraint. All that is required is that the image pull spec match the pull spec given to MCO by the CVO.

[runcom]

we need to relax those checks I guess? or just warn?, taking a stab at this currently

[runcom]

For debugging/testing, I built a custom release image, overriding the machine-config-controller image pull spec

@sjenning did you run an upgrade after the cluster was up or you installed with the custom release payload?

[sjenning]

custom release payload with the machine-config-controller overridden

looking at

machine-config-operator/pkg/operator/status.go

Lines 225 to 231 in f31d48f

	v, ok := mc.Annotations[ctrlcommon.GeneratedByControllerVersionAnnotationKey]
	if t.versionCheckRequired && !ok {
	return fmt.Errorf("%s must be created by controller version %s", t.name, version)
	}
	if ok && v != version {
	return fmt.Errorf("controller version mismatch for %s expected %s has %s", t.name, version, v)
	}

and was added by @abhinavdahiya in #265

seems that it is trying to verify that the MCO version matches the MCC version that created the MachineConfig

@abhinavdahiya why is this constraint needed?

[runcom]

seems that it is trying to verify that the MCO version matches the MCC version that created the MachineConfig

yeah, I was looking at the very same code, we're tagging each MC with a statically built verison of the component itself. Using image specs from the CVO might make sense though (as the only source of truth) . We can tag mc with the image spec, and since the operator has a list of CVO provided images we can match against those (if it makes sense at all)

[runcom]

ok yeah, I get the same as well:

INFO Waiting up to 30m0s for the cluster to initialize... 
DEBUG Still waiting for the cluster to initialize... 
DEBUG Still waiting for the cluster to initialize: Cluster operator machine-config has not yet reported success 
DEBUG Still waiting for the cluster to initialize: Cluster operator machine-config is reporting a failure: Unable to apply 3.11.0-598-g143c1acb: error syncing: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for master-8e32cb9e7734390e7d40ecc06f952efe expected 3.11.0-598-g143c1acb has 3.11.0-624-g6dd0d387-dirty 
FATAL failed to initialize the cluster: Cluster operator machine-config is reporting a failure: Unable to apply 3.11.0-598-g143c1acb: error syncing: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for master-8e32cb9e7734390e7d40ecc06f952efe expected 3.11.0-598-g143c1acb has 3.11.0-624-g6dd0d387-dirty 

[runcom]

@sjenning a workaround I'm trying for testing/developing is bundling the whole MCO like this:

oc adm release new -n openshift --server https://api.ci.openshift.org --from-image-stream "origin-v4.0" --to-image quay.io/runcom/origin-release:v4.0 machine-config-operator=quay.io/runcom/mco-dev:latest machine-config-controller=quay.io/runcom/mcc-dev:latest machine-config-daemon=quay.io/runcom/mcd-dev:latest machine-config-server=quay.io/runcom/mcs-dev:latest 

not sure but I'm testing it out, it works indeed

[cgwalters]

It seems that the operator expects each operand to match its own version. This should not be a constraint. All that is required is that the image pull spec match the pull spec given to MCO by the CVO.

This gets into the fact we really haven't been testing MCO upgrades of itself 😦