Skip to content

Conversation

@dmage
Copy link
Contributor

@dmage dmage commented Dec 9, 2019

Fixes https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_image-registry/208/pull-ci-openshift-image-registry-release-3.11-integration/42

Shutting down
kill: sending signal to 9 failed: No such process
2019-12-09 13:13:34.468479 N | pkg/osutil: received terminated signal, shutting down...
Gathering artifacts ...
2019/12/09 13:13:35 Container control-plane in pod pull-ci-openshift-image-registry-release-3.11-integration failed, exit code 123, reason Error

xargs returns 123 if any invocation of the command exited with status 1-125. In this case, kill failed. As we use -e, at that moment the entire script stops with the exit code 123.

/assign @smarterclayton

@smarterclayton you've touched this recently, is it ok to just ignore this error?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks correct, can you change all the others as well (i.e. every usage of this in this repo)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@smarterclayton updated, there were two kinds of changes:

-jobs -p | xargs -r kill
+jobs -p | xargs -r kill || true
-jobs -p | xargs kill || true
+jobs -p | xargs -r kill || true

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm, I thought I got all the former. Must have forked after my change.

@smarterclayton
Copy link
Contributor

Looks correct, please apply to all others

@openshift-ci-robot openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Dec 11, 2019
@dmage dmage changed the title sidecar-3: exit with 0 even if kill failed exit with 0 even if kill failed Dec 11, 2019
Copy link
Contributor

@stevekuznetsov stevekuznetsov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From an outside observer this is where the complexity starts to ramp up into unsustainable territory.

@smarterclayton
Copy link
Contributor

Except it doesn't ramp up. We fix it, it goes away, we don't worry about it.

@dmage
Copy link
Contributor Author

dmage commented Dec 12, 2019

/retest

@dmage
Copy link
Contributor Author

dmage commented Dec 12, 2019

/retest ci/rehearse/openshift/cloud-credential-operator/master/e2e-gcp

@dmage
Copy link
Contributor Author

dmage commented Dec 12, 2019

/retest

@dmage
Copy link
Contributor Author

dmage commented Dec 12, 2019

https://prow.svc.ci.openshift.org/pr-history/?org=openshift&repo=release&pr=6295
failures don't seem to be related to the changes

@stevekuznetsov
Copy link
Contributor

This is the third PR for this bash in recent memory... that sounds like ramping up to me :)

@dmage
Copy link
Contributor Author

dmage commented Dec 13, 2019

/retest

@dmage
Copy link
Contributor Author

dmage commented Dec 16, 2019

@smarterclayton @stevekuznetsov can we merge it?

@dmage
Copy link
Contributor Author

dmage commented Dec 18, 2019

/retest

@smarterclayton
Copy link
Contributor

/lgtm

Thanks

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 18, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dmage, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 18, 2019
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

2 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 340c0b1 into openshift:master Dec 18, 2019
@openshift-ci-robot
Copy link
Contributor

@dmage: Updated the following 39 configmaps:

  • prow-job-cluster-launch-installer-src configmap in namespace ci at cluster ci/api-build01-ci-devcluster-openshift-com:6443 using the following files:
    • key cluster-launch-installer-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml
  • prow-job-cluster-launch-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-e2e.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e.yaml
  • prow-job-cluster-launch-e2e-azure configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-e2e-azure.yaml using file ci-operator/templates/openshift/openshift-azure/cluster-launch-e2e-azure.yaml
  • prow-job-master-sidecar-3 configmap in namespace ci at cluster default using the following files:
    • key master-sidecar-3.yaml using file ci-operator/templates/master-sidecar-3.yaml
  • prow-job-endurance-install configmap in namespace ci-stg at cluster default using the following files:
    • key endurance-install.yaml using file ci-operator/templates/openshift/endurance/endurance-install.yaml
  • prow-job-cluster-launch-installer-custom-test-image configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-custom-test-image.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-custom-test-image.yaml
  • prow-job-cluster-launch-installer-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
  • prow-job-cluster-scaleup-e2e-40 configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-scaleup-e2e-40.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-scaleup-e2e-40.yaml
  • prow-job-cluster-launch-installer-metal-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-metal-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-metal-e2e.yaml
  • prow-job-cluster-launch-installer-upi-src configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-upi-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-upi-src.yaml
  • prow-job-cluster-launch-e2e-upgrade configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-e2e-upgrade.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-upgrade.yaml
  • prow-job-cluster-launch-e2e-azure configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-e2e-azure.yaml using file ci-operator/templates/openshift/openshift-azure/cluster-launch-e2e-azure.yaml
  • prow-job-endurance-install configmap in namespace ci at cluster default using the following files:
    • key endurance-install.yaml using file ci-operator/templates/openshift/endurance/endurance-install.yaml
  • prow-job-cluster-launch-installer-metal-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-metal-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-metal-e2e.yaml
  • prow-job-master-sidecar-4.2 configmap in namespace ci at cluster default using the following files:
    • key master-sidecar-4.2.yaml using file ci-operator/templates/master-sidecar-4.2.yaml
  • prow-job-cluster-launch-e2e-openshift-jenkins configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-e2e-openshift-jenkins.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-openshift-jenkins.yaml
  • prow-job-cluster-launch-e2e-upgrade configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-e2e-upgrade.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-upgrade.yaml
  • prow-job-cluster-launch-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-e2e.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e.yaml
  • prow-job-cluster-scaleup-e2e-40 configmap in namespace ci at cluster default using the following files:
    • key cluster-scaleup-e2e-40.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-scaleup-e2e-40.yaml
  • prow-job-master-sidecar-3 configmap in namespace ci-stg at cluster default using the following files:
    • key master-sidecar-3.yaml using file ci-operator/templates/master-sidecar-3.yaml
  • prow-job-cluster-launch-installer-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
  • prow-job-cluster-launch-installer-src configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml
  • prow-job-cluster-launch-installer-libvirt-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-libvirt-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-libvirt-e2e.yaml
  • prow-job-cluster-launch-installer-openstack-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-openstack-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-openstack-e2e.yaml
  • prow-job-cluster-launch-installer-src configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml
  • prow-job-cluster-launch-installer-upi-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-upi-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-upi-e2e.yaml
  • prow-job-cluster-launch-e2e-openshift-ansible configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-e2e-openshift-ansible.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-openshift-ansible.yaml
  • prow-job-cluster-launch-e2e-openshift-jenkins configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-e2e-openshift-jenkins.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-openshift-jenkins.yaml
  • prow-job-master-sidecar-4.2 configmap in namespace ci-stg at cluster default using the following files:
    • key master-sidecar-4.2.yaml using file ci-operator/templates/master-sidecar-4.2.yaml
  • prow-job-endurance-test configmap in namespace ci at cluster default using the following files:
    • key endurance-test.yaml using file ci-operator/templates/openshift/endurance/endurance-test.yaml
  • prow-job-endurance-test configmap in namespace ci-stg at cluster default using the following files:
    • key endurance-test.yaml using file ci-operator/templates/openshift/endurance/endurance-test.yaml
  • prow-job-cluster-launch-installer-custom-test-image configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-custom-test-image.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-custom-test-image.yaml
  • prow-job-cluster-launch-installer-e2e configmap in namespace ci at cluster ci/api-build01-ci-devcluster-openshift-com:6443 using the following files:
    • key cluster-launch-installer-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
  • prow-job-cluster-launch-installer-libvirt-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-libvirt-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-libvirt-e2e.yaml
  • prow-job-cluster-launch-installer-openstack-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-openstack-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-openstack-e2e.yaml
  • prow-job-cluster-launch-installer-upi-e2e configmap in namespace ci at cluster ci/api-build01-ci-devcluster-openshift-com:6443 using the following files:
    • key cluster-launch-installer-upi-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-upi-e2e.yaml
  • prow-job-cluster-launch-installer-upi-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-upi-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-upi-e2e.yaml
  • prow-job-cluster-launch-installer-upi-src configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-upi-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-upi-src.yaml
  • prow-job-cluster-launch-e2e-openshift-ansible configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-e2e-openshift-ansible.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-openshift-ansible.yaml
Details

In response to this:

Fixes https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_image-registry/208/pull-ci-openshift-image-registry-release-3.11-integration/42

Shutting down
kill: sending signal to 9 failed: No such process
2019-12-09 13:13:34.468479 N | pkg/osutil: received terminated signal, shutting down...
Gathering artifacts ...
2019/12/09 13:13:35 Container control-plane in pod pull-ci-openshift-image-registry-release-3.11-integration failed, exit code 123, reason Error

xargs returns 123 if any invocation of the command exited with status 1-125. In this case, kill failed. As we use -e, at that moment the entire script stops with the exit code 123.

/assign @smarterclayton

@smarterclayton you've touched this recently, is it ok to just ignore this error?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link
Contributor

@dmage: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/rehearse/openshift/installer/fcos/e2e-vsphere d0744e5 link /test pj-rehearse
ci/rehearse/openshift/cloud-credential-operator/master/e2e-azure d0744e5 link /test pj-rehearse
ci/rehearse/openshift/cloud-credential-operator/master/e2e-gcp d0744e5 link /test pj-rehearse
ci/rehearse/openshift-priv/ci-experiment-origin/master/e2e-cmd d0744e5 link /test pj-rehearse
ci/rehearse/openshift-priv/ci-experiment-origin/master/e2e-azure d0744e5 link /test pj-rehearse
ci/rehearse/openshift-priv/ci-experiment-origin/master/e2e-vsphere d0744e5 link /test pj-rehearse
ci/rehearse/openshift/installer/master/e2e-metal d0744e5 link /test pj-rehearse
ci/rehearse/openshift/installer/master/e2e-gcp-upi d0744e5 link /test pj-rehearse
ci/rehearse/openshift/cloud-credential-operator/master/e2e-openstack d0744e5 link /test pj-rehearse
ci/rehearse/openshift/installer/master/e2e-aws-proxy d0744e5 link /test pj-rehearse
ci/rehearse/openshift/installer/master/e2e-aws-scaleup-rhel7 d0744e5 link /test pj-rehearse
ci/prow/pj-rehearse d0744e5 link /test pj-rehearse

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

wking added a commit to wking/ci-tools that referenced this pull request Dec 20, 2019
Bringing over a number of changes which have landed in
ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
as of openshift/release@016eb4ed27 (Merge pull request
openshift/release#6505 from hongkailiu/clusterReaders, 2019-12-19).
One series was improved kill logic:

* openshift/release@9cd158adf3 (template: Use a more correct kill
  command, 2019-12-03, openshift/release#6223).
* openshift/release@d0744e520d (exit with 0 even if kill failed,
  2019-12-09, openshift/release#6295)

Another series was around AWS instance console logs:

* openshift/release@e102a16d89
  (ci-operator/templates/openshift/installer/cluster-launch-installer-e2e:
  Gather node console logs on AWS, 2019-12-02,
  openshift/release#6189).
* openshift/release@26fde70045
  (ci-operator/templates/openshift/installer/cluster-launch-installer-e2e:
  Set AWS_DEFAULT_REGION, 2019-12-04, openshift/release#6249).

And there was also:

* openshift/release@cdf97164aa (templates: Add large and xlarge
  variants, 2019-11-25, openshift/release#6081).
* openshift/release@8cbef5e4a7
  (ci-operator/templates/openshift/installer/cluster-launch-installer-e2e:
  Error-catching for Google OAuth pokes, 2019-12-02,
  openshift/release#6190).
* openshift/release@ad29eda8dd (template: Gather the prometheus target
  metadata during teardown, 2019-12-12, openshift/release#6379).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants