Bump timeout for e2e-aws-op

[cgwalters]

We seem to be taking longer here.

[runcom]

I'm fine with this if we're just timing out on the latest test added (quorum guard afaict)

[runcom]

/approve
/lgtm

[hexfusion]

/lgtm

Thanks @cgwalters @runcom !

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, hexfusion, runcom

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [cgwalters,runcom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hexfusion]

/test e2e-aws-upgrade

[cgwalters]

              "degradedMachineCount": 0,
                "machineCount": 3,
                "observedGeneration": 1,
                "readyMachineCount": 2,
                "unavailableMachineCount": 1,
                "updatedMachineCount": 2
            }

[runcom]

@cgwalters that just looks like the worker pool is still rolling and reconciling to me

[openshift-ci-robot]

New changes are detected. LGTM label has been removed.

[kikisdeliveryservice]

120 seems awfully high? i assume we just trying it to see if it's truly a timeout issue?

[cgwalters]

i assume we just trying it to see if it's truly a timeout issue?

Right.
/hold
Since we probably don't want to merge as is.
Also I suspect 2 hours may also be the timeout for Prow jobs in general? Need to investigate that.

[cgwalters]

I find myself getting confused by getUnavailableMachines...it feels like it's supposed to be getMachinesToUpdate but shouldn't it be checking for nodes which are ready?

diff --git a/pkg/controller/node/status.go b/pkg/controller/node/status.go
index 75c16477..dfdd2ee1 100644
--- a/pkg/controller/node/status.go
+++ b/pkg/controller/node/status.go
@@ -192,10 +192,12 @@ func getUnavailableMachines(currentConfig string, nodes []*corev1.Node) []*corev
 			continue
 		}
 
-		nodeNotReady := !isNodeReady(node)
+		if !isNodeReady(node) {
+			continue
+		}
 		if dconfig == currentConfig && (dconfig != cconfig || nodeNotReady) {
 			unavail = append(unavail, node)
-			glog.V(2).Infof("Node %s unavailable: different configs (desired: %s, current %s) or node not ready %v", node.Name, dconfig, cconfig, nodeNotReady)
+			glog.V(2).Infof("Node %s unavailable: different configs (desired: %s, current %s)", node.Name, dconfig, cconfig)
 		}
 	}
 	return unavail

[cgwalters]

Paste from @sjenning

default                                      72s         Warning   MCDBootstrapSyncFailure       node/master-1                                          pending config rendered-master-1eb4fcd28e9f85ea9ac914d1789ad529 bootID a86994cc-fd33-47d4-a3e6-76da3c3d6fa2 matches current! Failed to reboot?
default                                      47s         Warning   MCDBootstrapSyncFailure       node/master-0                                          pending config rendered-master-8e7d5408d19158180e4cd0b6750b434c bootID d1a6236c-df42-412d-bac6-49155ea4ea82 matches current! Failed to reboot?
openshift-machine-config-operator            14s         Warning   FailedScheduling              pod/etcd-quorum-guard-745f7c4d8f-9wstp                 0/6 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 2 node(s) were unschedulable, 3 node(s) didn't match node selector.

$ oc get nodes
NAME       STATUS                     ROLES    AGE   VERSION
master-0   Ready,SchedulingDisabled   master   20h   v1.13.4+e8d1fd69b
master-1   Ready,SchedulingDisabled   master   20h   v1.13.4+e8d1fd69b
master-2   Ready                      master   20h   v1.13.4+e8d1fd69b
worker-0   Ready                      worker   20h   v1.13.4+1f90f9755
worker-1   Ready                      worker   20h   v1.13.4+1f90f9755
worker-2   Ready                      worker   20h   v1.13.4+1f90f9755

[kikisdeliveryservice]

Noticing in the CI artifacts logs that one of the worker daemons has no logs whatsoever

edit: double checked and it was a worker daemon see: openshift-machine-config-operator_machine-config-daemon-gfwj7_machine-config-daemon.log.gz

[openshift-ci-robot]

@cgwalters: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-upgrade	cde17af	link	/test e2e-aws-upgrade
ci/prow/e2e-aws-op	cde17af	link	/test e2e-aws-op

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[runcom]

I find myself getting confused by getUnavailableMachines...it feels like it's supposed to be getMachinesToUpdate but shouldn't it be checking for nodes which are ready?

@cgwalters your patch is reporting unavailable=0 when nodes aren't ready tho, which is what makes progress if everything is available.

[cgwalters]

Closing in favor of #697