Skip to content

ISV Operator Automation#24433

Closed
bandrade wants to merge 1 commit into
openshift:masterfrom
bandrade:isv-operators-automation
Closed

ISV Operator Automation#24433
bandrade wants to merge 1 commit into
openshift:masterfrom
bandrade:isv-operators-automation

Conversation

@bandrade
Copy link
Copy Markdown
Contributor

@bandrade bandrade commented Jan 21, 2020 •
edited
Loading

PR Created to automate ISV Operators test, now it's covering Basic Testing as explained on https://github.com/openshift/osde2e#operator-testing .

Features implemented:

  • It does a basic test for all operators in redhat and certified OperatorSources
  • It creates a subscription based on each packagemanifest description and installModes. For packages that supportes SingleNamespace and OwnNamespace, a specific project is created with specific OperatorGroup. For AllNamespaces it creates a subscription into openshift-operators namespace. After creating the a Subscription and wait for ClusterServiceVersion succeed. And if so, it removes all Operator dependencies to unsure that Operator can be successfully uninstalled.
  • Can be extensible to conver Intermediate and Advanced Testing.
  • Runs 5 Operators test cases in parallel.
  • A specific test suite is created. It does not belong to conformance test suite. Test failures should not interfere at OCP build.

Dry run execution:

./openshift-tests run openshift/isv --dry-run 
Jan 21 16:26:29.640: INFO: Running 'oc --kubeconfig=/home/bandrade/QE/ocp4-aws-ipi-vagrant-4.4/cluster/auth/kubeconfig get packagemanifest -l catalog=redhat-operators -o=jsonpath={range .items[*].metadata}{.name}{'\n'}{end}'
Jan 21 16:26:33.061: INFO: Running 'oc --kubeconfig=/home/bandrade/QE/ocp4-aws-ipi-vagrant-4.4/cluster/auth/kubeconfig get packagemanifest -l catalog=certified-operators -o=jsonpath={range .items[*].metadata}{.name}{'\n'}{end}'
I0121 16:26:50.674041   23391 test_context.go:419] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready
"[Suite:openshift/isv] Operator 3scale-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator akka-cluster-operator-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator amq-broker should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator amq-online should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator amq-streams should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator amq7-cert-manager should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator amq7-interconnect-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator anchore-engine should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator appdynamics-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator appsody-operator-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator aqua-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator businessautomation-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator cam-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator cic-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator citrix-adc-istio-ingress-gateway-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator citrix-cpx-istio-sidecar-injector-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator cluster-logging should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator cockroachdb-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator codeready-workspaces should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator cortex-certifai-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator cortex-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator couchbase-enterprise-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator couchdb-operator-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator cpx-cic-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator crunchy-postgres-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator datagrid should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator dotscience-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator driverlessai-deployment-operator-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator dv-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator eap should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator eddi-operator-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator elasticsearch-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator federatorai-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator fuse-apicurito should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator fuse-camel-k should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator fuse-online should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator hazelcast-enterprise-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator hazelcast-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator ibm-block-csi-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator ibm-spectrum-scale-csi should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator insightedge-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator instana-agent should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator ivory-server-app should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator jaeger-product should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator joget-openshift-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator kiali-ossm should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator kong should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator kube-arangodb should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator kubeturbo-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator kubevirt-hyperconverged should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator local-storage-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator memql-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator metering-ocp should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator mongodb-enterprise should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator neuvector-certified-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator newrelic-infrastructure should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator nfd should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator nuodb-ce-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator ocs-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator oneagent-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator open-enterprise-spinnaker should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator open-liberty-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator openshiftansibleservicebroker should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator openshifttemplateservicebroker should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator openunison-ocp-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator orca should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator perceptilabs-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator percona-server-mongodb-operator-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator percona-xtradb-cluster-operator-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator planetscale-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator portworx-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator presto-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator redis-enterprise-operator-aplpha-cert should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator redis-enterprise-operator-cert should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator robin-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator seldon-operator-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator sematext should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator serverless-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator servicemeshoperator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator sriov-network-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator storageos should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator synopsys-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator sysdig-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator t8c-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator tidb-operator-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator twistlock-certified should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator ubix-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator uma-operator should work properly [Suite:openshift]"
"[Suite:openshift/isv] Operator wavefront-operator should work properly [Suite:openshift]

@ecordell @njhale @kevinrizza. please have a review. Thanks! cc: @scolange @chengzhang1016 @emmajiafan @jianzhangbjz @tbuskey @shawn-hurley

@openshift-ci-robot openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 21, 2020
@shawn-hurley
Copy link
Copy Markdown

shawn-hurley commented Jan 21, 2020

/hold

I think that before we merge this, we need to settle on openshift/enhancements#183

I do not believe that we want these tests to block a kubernetes update.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 21, 2020
@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented Jan 21, 2020

@bandrade: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/verify 343857a link /test verify
ci/prow/e2e-gcp 343857a link /test e2e-gcp
ci/prow/e2e-cmd 343857a link /test e2e-cmd
ci/prow/e2e-aws-serial 343857a link /test e2e-aws-serial
ci/prow/e2e-aws-fips 343857a link /test e2e-aws-fips
ci/prow/e2e-gcp-upgrade 343857a link /test e2e-gcp-upgrade
ci/prow/e2e-gcp-builds 343857a link /test e2e-gcp-builds

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented Feb 11, 2020

@bandrade: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 11, 2020
@openshift-bot
Copy link
Copy Markdown
Contributor

openshift-bot commented May 11, 2020

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci-robot openshift-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 11, 2020
@bandrade
Copy link
Copy Markdown
Contributor Author

bandrade commented Jun 3, 2020

already migrated to openshift-tests repository, so closing this one.

@bandrade bandrade closed this Jun 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bandrade @shawn-hurley @openshift-ci-robot @openshift-bot