From dd860daa92915b339f97b1a827ac916e82f00549 Mon Sep 17 00:00:00 2001 From: Alex Dellapenta Date: Fri, 9 Oct 2020 22:36:16 -0600 Subject: [PATCH] Update OSDK scorecard for 4.7 --- modules/osdk-about-scorecard-tool.adoc | 8 - modules/osdk-running-scorecard.adoc | 32 --- modules/osdk-scorecard-about.adoc | 34 +++ modules/osdk-scorecard-config.adoc | 187 ++++------------ modules/osdk-scorecard-custom-tests.adoc | 155 ++++++++++++++ modules/osdk-scorecard-olm.adoc | 238 --------------------- modules/osdk-scorecard-output.adoc | 78 +++++++ modules/osdk-scorecard-parallel.adoc | 44 ++++ modules/osdk-scorecard-run.adoc | 30 +++ modules/osdk-scorecard-select-tests.adoc | 39 ++++ modules/osdk-scorecard-tests.adoc | 69 ++---- operators/operator_sdk/osdk-scorecard.adoc | 30 +-- 12 files changed, 456 insertions(+), 488 deletions(-) delete mode 100644 modules/osdk-about-scorecard-tool.adoc delete mode 100644 modules/osdk-running-scorecard.adoc create mode 100644 modules/osdk-scorecard-about.adoc create mode 100644 modules/osdk-scorecard-custom-tests.adoc delete mode 100644 modules/osdk-scorecard-olm.adoc create mode 100644 modules/osdk-scorecard-output.adoc create mode 100644 modules/osdk-scorecard-parallel.adoc create mode 100644 modules/osdk-scorecard-run.adoc create mode 100644 modules/osdk-scorecard-select-tests.adoc diff --git a/modules/osdk-about-scorecard-tool.adoc b/modules/osdk-about-scorecard-tool.adoc deleted file mode 100644 index 2158286330ea..000000000000 --- a/modules/osdk-about-scorecard-tool.adoc +++ /dev/null @@ -1,8 +0,0 @@ -// Module included in the following assemblies: -// -// * operators/operator_sdk/osdk-scorecard.adoc - -[id="osdk-about-scorecard-tool_{context}"] -= About the scorecard tool - -To validate an Operator, the scorecard tool provided by the Operator SDK begins by creating all resources required by any related custom resources (CRs) and the Operator. The scorecard then creates a proxy container in the deployment of the Operator which is used to record calls to the API server and run some of the tests. The tests performed also examine some of the parameters in the CRs. diff --git a/modules/osdk-running-scorecard.adoc b/modules/osdk-running-scorecard.adoc deleted file mode 100644 index 069f18785113..000000000000 --- a/modules/osdk-running-scorecard.adoc +++ /dev/null @@ -1,32 +0,0 @@ -// Module included in the following assemblies: -// -// * operators/operator_sdk/osdk-scorecard.adoc - -[id="osdk-running-scorecard_{context}"] -= Running the scorecard - -.Prerequisites - -The following prerequisites for the Operator project are checked by the scorecard tool: - -* Access to a cluster running Kubernetes 1.11.3 or later. -* If you want to use the scorecard to check the integration of your Operator project with Operator Lifecycle Manager (OLM), then a cluster service version (CSV) file is also required. This is a requirement when the `olm-deployed` option is used. -* For Operators that were not generated using the Operator SDK (non-SDK Operators): -** Resource manifests for installing and configuring the Operator and custom resources (CRs). -ifdef::openshift-origin[] -See the link:https://github.com/operator-framework/operator-sdk/blob/v0.15.0/doc/test-framework/writing-e2e-tests.md[Writing E2E Tests] guide for more information on the global and namespaced manifests. -endif::[] -** Configuration getter that supports reading from the `KUBECONFIG` environment variable, such as the `clientcmd` or `controller-runtime` configuration getters. This is required for the scorecard proxy to work correctly. - -.Procedure - -. Define a `.osdk-scorecard.yaml` configuration file in your Operator project. -. Create the namespace defined in the RBAC files (`role_binding`). -. Run the scorecard from the root directory of your Operator project: -+ -[source,terminal] ----- -$ operator-sdk scorecard ----- -+ -The scorecard return code is `1` if any of the executed texts did not pass and `0` if all selected tests passed. diff --git a/modules/osdk-scorecard-about.adoc b/modules/osdk-scorecard-about.adoc new file mode 100644 index 000000000000..09001d6de229 --- /dev/null +++ b/modules/osdk-scorecard-about.adoc @@ -0,0 +1,34 @@ +// Module included in the following assemblies: +// +// * operators/operator_sdk/osdk-scorecard.adoc + +[id="osdk-about-scorecard_{context}"] += About the scorecard tool + +While the Operator SDK `bundle validate` subcommand can validate local bundle directories and remote bundle images for content and structure, you can use the `scorecard` subcommand to run tests on your Operator based on a configuration file and test images. These tests are implemented within test images that are configured and constructed to be executed by the scorecard. + +The scorecard assumes it is run with access to a configured Kubernetes cluster, such as {product-title}. The scorecard runs each test within a pod, from which pod logs are aggregated and test results are sent to the console. The scorecard has built-in basic and Operator Lifecycle Manager (OLM) tests and also provides a means to execute custom test definitions. + +.Scorecard workflow +. Create all resources required by any related custom resources (CRs) and the Operator +. Create a proxy container in the deployment of the Operator to record calls to the API server and run tests +. Examine parameters in the CRs + +The scorecard tests make no assumptions as to the state of the Operator being tested. Creating Operators and CRs for an Operators are beyond the scope of the scorecard itself. Scorecard tests can, however, create whatever resources they require if the tests are designed for resource creation. + +The `scorecard` subcommand has the following syntax: + +[source,terminal] +---- +$ operator-sdk scorecard [flags] +---- + +The scorecard requires a positional argument for either the on-disk path to +your Operator bundle or the name of a bundle image. + +For further information about the flags, run: + +[source,terminal] +---- +$ operator-sdk scorecard -h +---- diff --git a/modules/osdk-scorecard-config.adoc b/modules/osdk-scorecard-config.adoc index 5afe50c264c2..bbe279c0d42f 100644 --- a/modules/osdk-scorecard-config.adoc +++ b/modules/osdk-scorecard-config.adoc @@ -2,160 +2,63 @@ // // * operators/operator_sdk/osdk-scorecard.adoc +:osdk_ver: v1.3.0 + [id="osdk-scorecard-config_{context}"] = Scorecard configuration -The scorecard tool uses a configuration file that allows you to configure internal plug-ins, as well as several global configuration options. - -[id="osdk-scorecard-config-file_{context}"] -== Configuration file - -The default location for the scorecard tool configuration is the `/.osdk-scorecard.*`. The following is an example of a YAML-formatted configuration file: +The scorecard tool uses a configuration that allows you to configure internal plug-ins, as well as several global configuration options. Tests are driven by a configuration file named `config.yaml`, which is generated by the `make bundle` command, located in your `bundle/` directory: -.Scorecard configuration file -[source,yaml] +[source,terminal] ---- -scorecard: - output: json - plugins: - - basic: <1> - cr-manifest: - - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml" - - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml" - - olm: <2> - cr-manifest: - - "deploy/crds/cache.example.com_v1alpha1_memcached_cr.yaml" - - "deploy/crds/cache.example.com_v1alpha1_memcachedrs_cr.yaml" - csv-path: "deploy/olm-catalog/memcached-operator/0.0.3/memcached-operator.v0.0.3.clusterserviceversion.yaml" +./bundle +... +└── tests + └── scorecard + └── config.yaml ---- -<1> `basic` tests configured to test two custom resources (CRs). -<2> `olm` tests configured to test two CRs. - -Configuration methods for global options take the following priority, highest to lowest: - -Command arguments (if available) -> configuration file -> default - -The configuration file must be in YAML format. As the configuration file might be extended to allow configuration of all `operator-sdk` subcommands in the future, the scorecard configuration must be under a `scorecard` subsection. - -[NOTE] -==== -Configuration file support is provided by the `viper` package. For more info on how `viper` configuration works, see the link:https://github.com/spf13/viper/blob/master/README.md[README]. -==== - -[id="osdk-scorecard-config-args_{context}"] -== Command arguments - -While most of the scorecard tool configuration is done using a configuration file, you can also use the following arguments: - -.Scorecard tool arguments -[cols="25%,20%,55%",options="header"] -|=== -|Flag |Type |Description - -|`--bundle`, `-b` -|string -|The path to a bundle directory used for the bundle validation test. - -|`--config` -|string -|The path to the scorecard configuration file. The default is `/.osdk-scorecard`. The file type and extension must be `.yaml`. If a configuration file is not provided or found at the default location, the scorecard exits with an error. - -|`--output`, `-o` -|string -|Output format. Valid options are `text` and `json`. The default format is `text`, which is designed to be a human readable format. The `json` format uses the JSON schema output format used for plug-ins defined later. - -|`--kubeconfig`, `-o` -|string -|The path to the `kubeconfig` file. It sets the `kubeconfig` for internal plug-ins. - -|`--version` -|string -|The version of scorecard to run. The default and only valid option is `v1alpha2`. - -|`--selector`, `-l` -|string -|The label selector to filter tests on. - -|`--list`, `-L` -|bool -|If `true`, only print the test names that would be run based on selector filtering. -|=== - -[id="osdk-scorecard-config-file-options_{context}"] -== Configuration file options - -The scorecard configuration file provides the following options: - -.Scorecard configuration file options -[cols="25%,20%,55%",options="header"] -|=== -|Option |Type |Description - -|`bundle` -|string -|Equivalent of the `--bundle` flag. Operator Lifecycle Manager (OLM) bundle directory path, when specified, runs bundle validation. - -|`output` -|string -|Equivalent of the `--output` flag. If this option is defined by both the configuration file and the flag, the flag value takes priority. - -|`kubeconfig` -|string -|Equivalent of the `--kubeconfig` flag. If this option is defined by both the configuration file and the flag, the flag value takes priority. - -|`plugins` -|array -|An array of plug-in names. -|=== -[id="osdk-scorecard-config-plugins_{context}"] -=== Basic and OLM plug-ins +.Example scorecard configuration file +[source,yaml,subs="attributes+"] +---- +kind: Configuration +apiversion: scorecard.operatorframework.io/v1alpha3 +metadata: + name: config +stages: +- parallel: true + tests: + - image: quay.io/operator-framework/scorecard-test:{osdk_ver} + entrypoint: + - scorecard-test + - basic-check-spec + labels: + suite: basic + test: basic-check-spec-test + - image: quay.io/operator-framework/scorecard-test:{osdk_ver} + entrypoint: + - scorecard-test + - olm-bundle-validation + labels: + suite: olm + test: olm-bundle-validation-test +---- -The scorecard supports the internal `basic` and `olm` plug-ins, which are configured by a `plugins` section in the configuration file. +The configuration file defines each test that scorecard can execute. The +following fields of the scorecard configuration file define the test as follows: -.Plug-in options -[cols="25%,20%,55%",options="header"] +[cols="3,7",options="header"] |=== -|Option |Type |Description - -|`cr-manifest` -|[]string -|The path(s) for CRs being tested. Required if `olm-deployed` is unset or `false`. - -|`csv-path` -|string -|The path to the cluster service version (CSV) for the Operator. Required for OLM tests or if `olm-deployed` is set to `true`. - -|`olm-deployed` -|bool -|Indicates that the CSV and relevant CRDs have been deployed onto the cluster by OLM. - -|`kubeconfig` -|string -|The path to the `kubeconfig` file. If both the global `kubeconfig` and this field are set, this field is used for the plug-in. - -|`namespace` -|string -|The namespace to run the plug-ins in. If unset, the default specified by the `kubeconfig` file is used. - -|`init-timeout` -|int -|Time in seconds until a timeout during initialization of the Operator. +|Configuration field |Description -|`crds-dir` -|string -|The path to the directory containing CRDs that must be deployed to the cluster. +|`image` +|Test container image name that implements a test -|`namespaced-manifest` -|string -|The manifest file with all resources that run within a namespace. By default, the scorecard combines the `service_account.yaml`, `role.yaml`, `role_binding.yaml`, and `operator.yaml` files from the `deploy` directory into a temporary manifest to use as the namespaced manifest. +|`entrypoint` +|Command and arguments that are invoked in the test image to execute a test -|`global-manifest` -|string -|The manifest containing required resources that run globally (not namespaced). By default, the scorecard combines all CRDs in the `crds-dir` directory into a temporary manifest to use as the global manifest. +|`labels` +|Scorecard-defined or custom labels that select which tests to run |=== -[NOTE] -==== -Currently, using the scorecard with a CSV does not permit multiple CR manifests to be set through the CLI, configuration file, or CSV annotations. You must tear down your Operator in the cluster, re-deploy, and re-run the scorecard for each CR that is tested. -==== +:!osdk_ver: diff --git a/modules/osdk-scorecard-custom-tests.adoc b/modules/osdk-scorecard-custom-tests.adoc new file mode 100644 index 000000000000..f28cb0a9765a --- /dev/null +++ b/modules/osdk-scorecard-custom-tests.adoc @@ -0,0 +1,155 @@ +// Module included in the following assemblies: +// +// * operators/operator_sdk/osdk-scorecard.adoc + +[id="osdk-scorecard-custom-tests_{context}"] += Custom scorecard tests + +The scorecard tool can run custom tests that follow these mandated conventions: + +* Tests are implemented within a container image +* Tests accept an entrypoint which include a command and arguments +* Tests produce `v1alpha3` scorecard output in JSON format with no extraneous logging in the test output +* Tests can obtain the bundle contents at a shared mount point of `/bundle` +* Tests can access the Kubernetes API using an in-cluster client connection + +Writing custom tests in other programming languages is possible if the test +image follows the above guidelines. + +The following example shows of a custom test image written in Go: + +.Example custom scorecard test +[%collapsible] +==== +[source,go] +---- +// Copyright 2020 The Operator-SDK Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "encoding/json" + "fmt" + "log" + "os" + + scapiv1alpha3 "github.com/operator-framework/api/pkg/apis/scorecard/v1alpha3" + apimanifests "github.com/operator-framework/api/pkg/manifests" +) + +// This is the custom scorecard test example binary +// As with the Redhat scorecard test image, the bundle that is under +// test is expected to be mounted so that tests can inspect the +// bundle contents as part of their test implementations. +// The actual test is to be run is named and that name is passed +// as an argument to this binary. This argument mechanism allows +// this binary to run various tests all from within a single +// test image. + +const PodBundleRoot = "/bundle" + +func main() { + entrypoint := os.Args[1:] + if len(entrypoint) == 0 { + log.Fatal("Test name argument is required") + } + + // Read the pod's untar'd bundle from a well-known path. + cfg, err := apimanifests.GetBundleFromDir(PodBundleRoot) + if err != nil { + log.Fatal(err.Error()) + } + + var result scapiv1alpha3.TestStatus + + // Names of the custom tests which would be passed in the + // `operator-sdk` command. + switch entrypoint[0] { + case CustomTest1Name: + result = CustomTest1(cfg) + case CustomTest2Name: + result = CustomTest2(cfg) + default: + result = printValidTests() + } + + // Convert scapiv1alpha3.TestResult to json. + prettyJSON, err := json.MarshalIndent(result, "", " ") + if err != nil { + log.Fatal("Failed to generate json", err) + } + fmt.Printf("%s\n", string(prettyJSON)) + +} + +// printValidTests will print out full list of test names to give a hint to the end user on what the valid tests are. +func printValidTests() scapiv1alpha3.TestStatus { + result := scapiv1alpha3.TestResult{} + result.State = scapiv1alpha3.FailState + result.Errors = make([]string, 0) + result.Suggestions = make([]string, 0) + + str := fmt.Sprintf("Valid tests for this image include: %s %s", + CustomTest1Name, + CustomTest2Name) + result.Errors = append(result.Errors, str) + return scapiv1alpha3.TestStatus{ + Results: []scapiv1alpha3.TestResult{result}, + } +} + +const ( + CustomTest1Name = "customtest1" + CustomTest2Name = "customtest2" +) + +// Define any operator specific custom tests here. +// CustomTest1 and CustomTest2 are example test functions. Relevant operator specific +// test logic is to be implemented in similarly. + +func CustomTest1(bundle *apimanifests.Bundle) scapiv1alpha3.TestStatus { + r := scapiv1alpha3.TestResult{} + r.Name = CustomTest1Name + r.State = scapiv1alpha3.PassState + r.Errors = make([]string, 0) + r.Suggestions = make([]string, 0) + almExamples := bundle.CSV.GetAnnotations()["alm-examples"] + if almExamples == "" { + fmt.Println("no alm-examples in the bundle CSV") + } + + return wrapResult(r) +} + +func CustomTest2(bundle *apimanifests.Bundle) scapiv1alpha3.TestStatus { + r := scapiv1alpha3.TestResult{} + r.Name = CustomTest2Name + r.State = scapiv1alpha3.PassState + r.Errors = make([]string, 0) + r.Suggestions = make([]string, 0) + almExamples := bundle.CSV.GetAnnotations()["alm-examples"] + if almExamples == "" { + fmt.Println("no alm-examples in the bundle CSV") + } + return wrapResult(r) +} + +func wrapResult(r scapiv1alpha3.TestResult) scapiv1alpha3.TestStatus { + return scapiv1alpha3.TestStatus{ + Results: []scapiv1alpha3.TestResult{r}, + } +} +---- +==== diff --git a/modules/osdk-scorecard-olm.adoc b/modules/osdk-scorecard-olm.adoc deleted file mode 100644 index 98023611585a..000000000000 --- a/modules/osdk-scorecard-olm.adoc +++ /dev/null @@ -1,238 +0,0 @@ -// Module included in the following assemblies: -// -// * operators/operator_sdk/osdk-scorecard.adoc - -[id="osdk-scorecard-olm_{context}"] -= Running the scorecard with an OLM-managed Operator - -The scorecard can be run using a cluster service version (CSV), providing a way to test cluster-ready and non-Operator SDK Operators. - -.Procedure - -. The scorecard requires a proxy container in the deployment pod of the Operator to read Operator logs. A few modifications to your CSV and creation of one extra object are required to run the proxy _before_ deploying your Operator with Operator Lifecycle Manager (OLM). -+ -This step can be performed manually or automated using bash functions. Choose one of the following methods. -+ --- -* *Manual method:* - -.. Create a proxy server secret containing a local `kubeconfig` file`. - -... Generate a user name using the namespaced owner reference of the scorecard proxy. -+ -[source,terminal] ----- -$ echo '{"apiVersion":"","kind":"","name":"scorecard","uid":"","Namespace":"''"}' | base64 -w 0 <1> ----- -<1> Replace `` with the namespace your Operator will deploy in. - -... Write a `Config` manifest `scorecard-config.yaml` using the following template, replacing `` with the base64 user name generated in the previous step: -+ -[source,yaml] ----- -apiVersion: v1 -kind: Config -clusters: -- cluster: - insecure-skip-tls-verify: true - server: http://@localhost:8889 - name: proxy-server -contexts: -- context: - cluster: proxy-server - user: admin/proxy-server - name: /proxy-server -current-context: /proxy-server -preferences: {} -users: -- name: admin/proxy-server - user: - username: - password: unused ----- - -... Encode the `Config` as base64: -+ -[source,terminal] ----- -$ cat scorecard-config.yaml | base64 -w 0 ----- - -... Create a `Secret` manifest `scorecard-secret.yaml`: -+ -[source,yaml] ----- -apiVersion: v1 -kind: Secret -metadata: - name: scorecard-kubeconfig - namespace: <1> -data: - kubeconfig: <2> ----- -<1> Replace `` with the namespace your Operator will deploy in. -<2> Replace `` with the `Config` encoded as base64. - -... Apply the secret: -+ -[source,terminal] ----- -$ oc apply -f scorecard-secret.yaml ----- - -... Insert a volume referring to the secret into the deployment for the Operator: -+ -[source,yaml] ----- -spec: - install: - spec: - deployments: - - name: memcached-operator - spec: - ... - template: - ... - spec: - containers: - ... - volumes: - - name: scorecard-kubeconfig <1> - secret: - secretName: scorecard-kubeconfig - items: - - key: kubeconfig - path: config ----- -<1> Scorecard `kubeconfig` volume. - -.. Insert a volume mount and `KUBECONFIG` environment variable into each container in the deployment of your Operator: -+ -[source,yaml] ----- -spec: - install: - spec: - deployments: - - name: memcached-operator - spec: - ... - template: - ... - spec: - containers: - - name: container1 - ... - volumeMounts: - - name: scorecard-kubeconfig <1> - mountPath: /scorecard-secret - env: - - name: KUBECONFIG <2> - value: /scorecard-secret/config - - name: container2 <3> - ... ----- -<1> Scorecard `kubeconfig` volume mount. -<2> Scorecard `kubeconfig` environment variable. -<3> Repeat the same for this and all other containers. - -.. Insert the scorecard proxy container into the deployment of your Operator: -+ -[source,yaml] ----- -spec: - install: - spec: - deployments: - - name: memcached-operator - spec: - ... - template: - ... - spec: - containers: - ... - - name: scorecard-proxy <1> - command: - - scorecard-proxy - env: - - name: WATCH_NAMESPACE - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.namespace - image: quay.io/operator-framework/scorecard-proxy:master - imagePullPolicy: Always - ports: - - name: proxy - containerPort: 8889 ----- -<1> Scorecard proxy container. - -* *Automated method:* -+ -The link:https://github.com/operator-framework/community-operators[`community-operators`] repository has several bash functions that can perform the previous steps in the procedure for you. - -.. Run the following `curl` command: -+ -[source,terminal] ----- -$ curl -Lo csv-manifest-modifiers.sh \ - https://raw.githubusercontent.com/operator-framework/community-operators/master/scripts/lib/file ----- - -.. Source the `csv-manifest-modifiers.sh` file: -+ -[source,terminal] ----- -$ . ./csv-manifest-modifiers.sh ----- - -.. Create the `kubeconfig` secret file: -+ -[source,terminal] ----- -$ create_kubeconfig_secret_file scorecard-secret.yaml "" <1> ----- -<1> Replace `` with the namespace your Operator will deploy in. - -.. Apply the secret: -+ -[source,terminal] ----- -$ oc apply -f scorecard-secret.yaml ----- - -.. Insert the `kubeconfig` volume: -+ -[source,terminal] ----- -$ insert_kubeconfig_volume "" <1> ----- -<1> Replace `` with the path to your CSV manifest. - -.. Insert the `kubeconfig` secret mount: -+ -[source,terminal] ----- -$ insert_kubeconfig_secret_mount "" ----- - -.. Insert the proxy container: -+ -[source,terminal] ----- -$ insert_proxy_container "" "quay.io/operator-framework/scorecard-proxy:master" ----- --- - -. After inserting the proxy container, follow the steps in the _Getting started with the Operator SDK_ guide to bundle your CSV and custom resource definitions (CRDs) and deploy your Operator on OLM. - -. After your Operator has been deployed on OLM, define a `.osdk-scorecard.yaml` configuration file in your Operator project and ensure both the `csv-path: ` and `olm-deployed` options are set. - -. Run the scorecard with both the `csv-path: ` and `olm-deployed` options set in your scorecard configuration file: -+ -[source,terminal] ----- -$ operator-sdk scorecard ----- diff --git a/modules/osdk-scorecard-output.adoc b/modules/osdk-scorecard-output.adoc new file mode 100644 index 000000000000..8998c70068dd --- /dev/null +++ b/modules/osdk-scorecard-output.adoc @@ -0,0 +1,78 @@ +// Module included in the following assemblies: +// +// * operators/operator_sdk/osdk-scorecard.adoc + +:osdk_ver: v1.3.0 + +[id="osdk-scorecard-output_{context}"] += Scorecard output + +The `--output` flag for the `scorecard` subcommand specifies the scorecard results output format: either `text` or `json`. + +.Example JSON output snippet +[%collapsible] +==== +[source,json,subs="attributes+"] +---- +{ + "apiVersion": "scorecard.operatorframework.io/v1alpha3", + "kind": "TestList", + "items": [ + { + "kind": "Test", + "apiVersion": "scorecard.operatorframework.io/v1alpha3", + "spec": { + "image": "quay.io/operator-framework/scorecard-test:{osdk_ver}", + "entrypoint": [ + "scorecard-test", + "olm-bundle-validation" + ], + "labels": { + "suite": "olm", + "test": "olm-bundle-validation-test" + } + }, + "status": { + "results": [ + { + "name": "olm-bundle-validation", + "log": "time=\"2020-06-10T19:02:49Z\" level=debug msg=\"Found manifests directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=debug msg=\"Found metadata directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=debug msg=\"Getting mediaType info from manifests directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=info msg=\"Found annotations file\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=info msg=\"Could not find optional dependencies file\" name=bundle-test\n", + "state": "pass" + } + ] + } + } + ] +} +---- +==== + +.Example text output snippet +[%collapsible] +==== +[source,text,subs="attributes+"] +---- +-------------------------------------------------------------------------------- +Image: quay.io/operator-framework/scorecard-test:{osdk_ver} +Entrypoint: [scorecard-test olm-bundle-validation] +Labels: + "suite":"olm" + "test":"olm-bundle-validation-test" +Results: + Name: olm-bundle-validation + State: pass + Log: + time="2020-07-15T03:19:02Z" level=debug msg="Found manifests directory" name=bundle-test + time="2020-07-15T03:19:02Z" level=debug msg="Found metadata directory" name=bundle-test + time="2020-07-15T03:19:02Z" level=debug msg="Getting mediaType info from manifests directory" name=bundle-test + time="2020-07-15T03:19:02Z" level=info msg="Found annotations file" name=bundle-test + time="2020-07-15T03:19:02Z" level=info msg="Could not find optional dependencies file" name=bundle-test +---- +==== + +[NOTE] +==== +The output format spec matches the link:https://pkg.go.dev/github.com/operator-framework/api/pkg/apis/scorecard/v1alpha3#Test[`Test`] type layout. +==== + +:!osdk_ver: diff --git a/modules/osdk-scorecard-parallel.adoc b/modules/osdk-scorecard-parallel.adoc new file mode 100644 index 000000000000..bc22f6ffce1b --- /dev/null +++ b/modules/osdk-scorecard-parallel.adoc @@ -0,0 +1,44 @@ +// Module included in the following assemblies: +// +// * operators/operator_sdk/osdk-scorecard.adoc + +[id="osdk-scorecard-parallel_{context}"] += Enabling parallel testing + +As an Operator author, you can define separate stages for your tests using the scorecard configuration file. Stages run sequentially in the order they are defined in the configuration file. A stage contains a list of tests and a configurable `parallel` setting. + +By default, or when a stage explicitly sets `parallel` to `false`, tests in a stage are run sequentially in the order they are defined in the configuration file. Running tests one at a time is helpful to guarantee that no two tests interact and conflict with each other. + +However, if tests are designed to be fully isolated, they can be parallelized. + +.Procedure + +* To run a set of isolated tests in parallel, include them in the same stage and set `parallel` to `true`: ++ +[source,terminal] +---- +apiVersion: scorecard.operatorframework.io/v1alpha3 +kind: Configuration +metadata: + name: config +stages: +- parallel: true <1> + tests: + - entrypoint: + - scorecard-test + - basic-check-spec + image: quay.io/operator-framework/scorecard-test:v1.3.0 + labels: + suite: basic + test: basic-check-spec-test + - entrypoint: + - scorecard-test + - olm-bundle-validation + image: quay.io/operator-framework/scorecard-test:v1.3.0 + labels: + suite: olm + test: olm-bundle-validation-test +---- +<1> Enables parallel testing ++ +All tests in a parallel stage are executed simultaneously, and scorecard waits for all of them to finish before proceding to the next stage. This can make your tests run much faster. diff --git a/modules/osdk-scorecard-run.adoc b/modules/osdk-scorecard-run.adoc new file mode 100644 index 000000000000..9f231b255566 --- /dev/null +++ b/modules/osdk-scorecard-run.adoc @@ -0,0 +1,30 @@ +// Module included in the following assemblies: +// +// * operators/operator_sdk/osdk-scorecard.adoc + +[id="osdk-scorecard-run_{context}"] += Running the scorecard tool + +A default set of Kustomize files are generated by the Operator SDK after running the `init` subcommand. The default `bundle/tests/scorecard/config.yaml` file that is generated can be immediately used to run the scorecard tool against your Operator, or you can modify this file to your test specifications. + +.Prerequisites + +* Operator project generated by using the Operator SDK + +.Procedure + +. Generate or regenerate your bundle manifests and metadata for your Operator: ++ +[source,terminal] +---- +$ make bundle +---- ++ +This command automatically adds scorecard annotations to your bundle metadata, which is used by the `scorecard` subcommand to run tests. + +. Run the scorecard against the on-disk path to your Operator bundle or the name of a bundle image: ++ +[source,terminal] +---- +$ operator-sdk scorecard +---- diff --git a/modules/osdk-scorecard-select-tests.adoc b/modules/osdk-scorecard-select-tests.adoc new file mode 100644 index 000000000000..822365703a05 --- /dev/null +++ b/modules/osdk-scorecard-select-tests.adoc @@ -0,0 +1,39 @@ +// Module included in the following assemblies: +// +// * operators/operator_sdk/osdk-scorecard.adoc + +[id="osdk-scorecard-select-tests_{context}"] += Selecting tests + +Scorecard tests are selected by setting the `--selector` CLI flag to a set of label strings. If a selector flag is not supplied, then all the tests within the scorecard configuration file are run. + +Tests are run serially with test results being aggregated by the scorecard and written to standard output, or _stdout_. + +.Procedure + +. To select a single test, for example `basic-check-spec-test`, specify the test by using the `--selector` flag: ++ +[source,terminal] +---- +$ operator-sdk scorecard \ + -o text \ + --selector=test=basic-check-spec-test +---- + +. To select a suite of tests, for example `olm`, specify a label that is used by all of the OLM tests: ++ +[source,terminal] +---- +$ operator-sdk scorecard \ + -o text \ + --selector=suite=olm +---- + +. To select multiple tests, specify the test names by using the `selector` flag using the following syntax: ++ +[source,terminal] +---- +$ operator-sdk scorecard \ + -o text \ + --selector='test in (basic-check-spec-test,olm-bundle-validation-test)' +---- diff --git a/modules/osdk-scorecard-tests.adoc b/modules/osdk-scorecard-tests.adoc index 8c17db62db31..b3d3a3cec3dd 100644 --- a/modules/osdk-scorecard-tests.adoc +++ b/modules/osdk-scorecard-tests.adoc @@ -3,72 +3,45 @@ // * operators/operator_sdk/osdk-scorecard.adoc [id="osdk-scorecard-tests_{context}"] -= Tests performed += Built-in scorecard tests -By default, the scorecard tool has a set of internal tests it can run available across two internal plug-ins. If multiple CRs are specified for a plug-in, the test environment is fully cleaned up after each CR so that each CR gets a clean testing environment. +The scorecard ships with pre-defined tests that are arranged into suites: the basic test suite and the Operator Lifecycle Manager (OLM) suite. -Each test has a short name that uniquely identifies the test. This is useful when selecting a specific test or tests to run. For example: - -[source,terminal] ----- -$ operator-sdk scorecard -o text --selector=test=checkspectest ----- - -[source,terminal] ----- -$ operator-sdk scorecard -o text --selector='test in (checkspectest,checkstatustest)' ----- - -[id="osdk-scorecard-tests-basic_{context}"] -== Basic plug-in - -The following basic Operator tests are available from the `basic` plug-in: - -.`basic` plug-in tests -[cols="25%,55%,20%",options="header"] +[id="osdk-scorecard-basic-tests_{context}"] +.Basic test suite +[cols="3,7,3",options="header"] |=== |Test |Description |Short name |Spec Block Exists -|This test checks the custom resources (CRs) created in the cluster to make sure that all CRs have a `spec` block. This test has a maximum score of `1`. -|`checkspectest` - -|Status Block Exists -|This test checks the CRs created in the cluster to make sure that all CRs have a `status` block. This test has a maximum score of `1`. -|`checkstatustest` - -|Writing Into CRs Has An Effect -|This test reads the scorecard proxy logs to verify that the Operator is making `PUT` or `POST`, or both, requests to the API server, indicating that it is modifying resources. This test has a maximum score of `1`. -|`writingintocrshaseffecttest` +|This test checks the custom resource (CR) created in the cluster to make sure that all CRs have a `spec` block. +|`basic-check-spec-test` |=== -[id="osdk-scorecard-tests-olm_{context}"] -== OLM plug-in - -The following Operator Lifecycle Manager (OLM) integration tests are available from the `olm` plug-in: +[id="osdk-scorecard-olm-tests_{context}"] +.OLM test suite -.`olm` plug-in tests -[cols="25%,55%,20%",options="header"] +[cols="3,7,3",options="header"] |=== |Test |Description |Short name -|OLM Bundle Validation -|This test validates the OLM bundle manifests found in the bundle directory as specified by the bundle flag. If the bundle contents contain errors, then the test result output includes the validator log as well as error messages from the validation library. -|`bundlevalidationtest` +|Bundle Validation +|This test validates the bundle manifests found in the bundle that is passed into scorecard. If the bundle contents contain errors, then the test result output includes the validator log as well as error messages from the validation library. +|`olm-bundle-validation-test` |Provided APIs Have Validation -|This test verifies that the CRDs for the provided CRs contain a validation section and that there is validation for each `spec` and `status` field detected in the CR. This test has a maximum score equal to the number of CRs provided by the `cr-manifest` option. -|`crdshavevalidationtest` +|This test verifies that the custom resource definitions (CRDs) for the provided CRs contain a validation section and that there is validation for each `spec` and `status` field detected in the CR. +|`olm-crds-have-validation-test` |Owned CRDs Have Resources Listed -|This test makes sure that the CRDs for each CR provided by the `cr-manifest` option have a `resources` subsection in the `owned` CRDs section of the CSV. If the test detects used resources that are not listed in the `resources` section, it lists them in the suggestions at the end of the test. This test has a maximum score equal to the number of CRs provided by the `cr-manifest` option. -|`crdshaveresourcestest` +|This test makes sure that the CRDs for each CR provided via the `cr-manifest` option have a `resources` subsection in the `owned` CRDs section of the ClusterServiceVersion (CSV). If the test detects used resources that are not listed in the resources section, it lists them in the suggestions at the end of the test. Users are required to fill out the resources section after initial code generation for this test to pass. +|`olm-crds-have-resources-test` |Spec Fields With Descriptors -|This test verifies that every field in the `spec` sections of custom resources have a corresponding descriptor listed in the CSV. This test has a maximum score equal to the total number of fields in the `spec` sections of each custom resource passed in by the `cr-manifest` option. -|`specdescriptorstest` +|This test verifies that every field in the CRs `spec` sections has a corresponding descriptor listed in the CSV. +|`olm-spec-descriptors-test` |Status Fields With Descriptors -|This test verifies that every field in the `status` sections of custom resources have a corresponding descriptor listed in the CSV. This test has a maximum score equal to the total number of fields in the `status` sections of each custom resource passed in by the `cr-manifest` option. -|`statusdescriptorstest` +|This test verifies that every field in the CRs `status` sections have a corresponding descriptor listed in the CSV. +|`olm-status-descriptors-test` |=== diff --git a/operators/operator_sdk/osdk-scorecard.adoc b/operators/operator_sdk/osdk-scorecard.adoc index 813e3b7f363a..d1752a0d33fb 100644 --- a/operators/operator_sdk/osdk-scorecard.adoc +++ b/operators/operator_sdk/osdk-scorecard.adoc @@ -1,30 +1,20 @@ [id="osdk-scorecard"] -= Validating Operators using the scorecard += Validating Operators using the scorecard tool include::modules/common-attributes.adoc[] :context: osdk-scorecard toc::[] -Operator authors should validate that their Operator is packaged correctly and free of syntax errors. As an Operator author, you can use the Operator SDK scorecard tool to validate your Operator packaging and run tests. +As an Operator author, you can use the scorecard tool in the Operator SDK to do the following tasks: -[NOTE] -==== -{product-title} 4.7 supports Operator SDK v1.4.0. -==== +* Validate that your Operator project is free of syntax errors and packaged correctly +* Review suggestions about ways you can improve your Operator -include::modules/osdk-about-scorecard-tool.adoc[leveloffset=+1] +include::modules/osdk-scorecard-about.adoc[leveloffset=+1] include::modules/osdk-scorecard-config.adoc[leveloffset=+1] -.Additional resources - -* You can either set `cr-manifest` or your CSV `metadata.annotations['alm-examples']` to provide CRs to the scorecard, but not both. See xref:../../operators/operator_sdk/osdk-generating-csvs.adoc#osdk-crds-templates_osdk-generating-csvs[CRD templates] for details. - include::modules/osdk-scorecard-tests.adoc[leveloffset=+1] -.Additional resources - -* xref:../../operators/operator_sdk/osdk-generating-csvs.adoc#osdk-crds-owned_osdk-generating-csvs[Owned CRDs] - -include::modules/osdk-running-scorecard.adoc[leveloffset=+1] -include::modules/osdk-scorecard-olm.adoc[leveloffset=+1] -.Additional resources - -* xref:../../operators/operator_sdk/golang/osdk-golang-tutorial.html#osdk-bundle-deploy-olm_osdk-golang-tutorial[Bundling and deploying your Operator by using OLM] +include::modules/osdk-scorecard-run.adoc[leveloffset=+1] +include::modules/osdk-scorecard-output.adoc[leveloffset=+1] +include::modules/osdk-scorecard-select-tests.adoc[leveloffset=+1] +include::modules/osdk-scorecard-parallel.adoc[leveloffset=+1] +include::modules/osdk-scorecard-custom-tests.adoc[leveloffset=+1]