ci-operator/step-registry/openshift/e2e/test: Add 2h active_deadline_seconds

[wking]

Looking at this job:

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_cluster-network-operator/785/pull-ci-openshift-cluster-network-operator-master-e2e-upgrade/1308837080670932992/build-log.txt | grep 'Executing\|Pod .* succeeded after\|Process did not'
2020/09/23 18:38:46 Executing "e2e-upgrade-ipi-install-monitoringpvc"
2020/09/23 18:38:52 Pod e2e-upgrade-ipi-install-monitoringpvc succeeded after 4s
2020/09/23 18:38:52 Executing "e2e-upgrade-ipi-install-loki"
2020/09/23 18:38:58 Pod e2e-upgrade-ipi-install-loki succeeded after 4s
2020/09/23 18:38:58 Executing "e2e-upgrade-ipi-conf"
2020/09/23 18:39:25 Pod e2e-upgrade-ipi-conf succeeded after 26s
2020/09/23 18:39:25 Executing "e2e-upgrade-ipi-conf-gcp"
2020/09/23 18:39:31 Pod e2e-upgrade-ipi-conf-gcp succeeded after 4s
2020/09/23 18:39:31 Executing "e2e-upgrade-ipi-install-rbac"
2020/09/23 18:39:36 Pod e2e-upgrade-ipi-install-rbac succeeded after 4s
2020/09/23 18:39:36 Executing "e2e-upgrade-ipi-install-install"
2020/09/23 19:18:00 Pod e2e-upgrade-ipi-install-install succeeded after 38m23s
2020/09/23 19:18:01 Executing "e2e-upgrade-openshift-e2e-test"
{"component":"entrypoint","file":"prow/entrypoint/run.go:165","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 4h0m0s timeout","severity":"error","time":"2020-09-23T22:34:21Z"}
2020/09/23 22:34:22 Executing "e2e-upgrade-gather-loki"
{"component":"entrypoint","file":"prow/entrypoint/run.go:250","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 30m0s grace period","severity":"error","time":"2020-09-23T23:04:21Z"}

That job was moving along fine, but hung up in the openshift-e2e-test step. After around 3h in the step, it hit the cumulative 4h timeout for the ci-operator run, terminated the openshift-e2e-test step, and entered a 30m teardown grace period. The first step of that grace period was the Loki gather, but, presumably because the cluster was dead, the Loki step hung and absorbed the entire grace period. This left no time for the emergency gather-extra collection, cluster teardown, or artifact uploads.

This commit adds a 2h timeout to openshift-e2e-test and a 10m timeout to gather-loki to limit the damage from similar hangs in the future, allowing us to get artifacts that will help understand why the cluster hung in the openshift-e2e-test step.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: stevekuznetsov, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/step-registry/gather/OWNERS [stevekuznetsov,wking]
• ci-operator/step-registry/openshift/OWNERS [stevekuznetsov,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@wking: Updated the following 2 configmaps:

• step-registry configmap in namespace ci at cluster api.ci using the following files:
  • key gather-loki-ref.yaml using file ci-operator/step-registry/gather/loki/gather-loki-ref.yaml
  • key openshift-e2e-test-ref.yaml using file ci-operator/step-registry/openshift/e2e/test/openshift-e2e-test-ref.yaml
• step-registry configmap in namespace ci at cluster app.ci using the following files:
  • key gather-loki-ref.yaml using file ci-operator/step-registry/gather/loki/gather-loki-ref.yaml
  • key openshift-e2e-test-ref.yaml using file ci-operator/step-registry/openshift/e2e/test/openshift-e2e-test-ref.yaml

Details

In response to this:

Looking at this job:

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_cluster-network-operator/785/pull-ci-openshift-cluster-network-operator-master-e2e-upgrade/1308837080670932992/build-log.txt | grep 'Executing\|Pod .* succeeded after\|Process did not'
2020/09/23 18:38:46 Executing "e2e-upgrade-ipi-install-monitoringpvc"
2020/09/23 18:38:52 Pod e2e-upgrade-ipi-install-monitoringpvc succeeded after 4s
2020/09/23 18:38:52 Executing "e2e-upgrade-ipi-install-loki"
2020/09/23 18:38:58 Pod e2e-upgrade-ipi-install-loki succeeded after 4s
2020/09/23 18:38:58 Executing "e2e-upgrade-ipi-conf"
2020/09/23 18:39:25 Pod e2e-upgrade-ipi-conf succeeded after 26s
2020/09/23 18:39:25 Executing "e2e-upgrade-ipi-conf-gcp"
2020/09/23 18:39:31 Pod e2e-upgrade-ipi-conf-gcp succeeded after 4s
2020/09/23 18:39:31 Executing "e2e-upgrade-ipi-install-rbac"
2020/09/23 18:39:36 Pod e2e-upgrade-ipi-install-rbac succeeded after 4s
2020/09/23 18:39:36 Executing "e2e-upgrade-ipi-install-install"
2020/09/23 19:18:00 Pod e2e-upgrade-ipi-install-install succeeded after 38m23s
2020/09/23 19:18:01 Executing "e2e-upgrade-openshift-e2e-test"
{"component":"entrypoint","file":"prow/entrypoint/run.go:165","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 4h0m0s timeout","severity":"error","time":"2020-09-23T22:34:21Z"}
2020/09/23 22:34:22 Executing "e2e-upgrade-gather-loki"
{"component":"entrypoint","file":"prow/entrypoint/run.go:250","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 30m0s grace period","severity":"error","time":"2020-09-23T23:04:21Z"}

That job was moving along fine, but hung up in the openshift-e2e-test step. After around 3h in the step, it hit the cumulative 4h timeout for the ci-operator run, terminated the openshift-e2e-test step, and entered a 30m teardown grace period. The first step of that grace period was the Loki gather, but, presumably because the cluster was dead, the Loki step hung and absorbed the entire grace period. This left no time for the emergency gather-extra collection, cluster teardown, or artifact uploads.

This commit adds a 2h timeout to openshift-e2e-test and a 10m timeout to gather-loki to limit the damage from similar hangs in the future, allowing us to get artifacts that will help understand why the cluster hung in the openshift-e2e-test step.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.