From 098df63396454a938d287af93135829e8817991c Mon Sep 17 00:00:00 2001 From: Alexander Pavel Date: Tue, 25 Sep 2018 10:30:05 -0700 Subject: [PATCH 1/9] doc/test-framework: update test framework walkthrough --- doc/test-framework/writing-e2e-tests.md | 60 ++++++++++++++++++++----- 1 file changed, 50 insertions(+), 10 deletions(-) diff --git a/doc/test-framework/writing-e2e-tests.md b/doc/test-framework/writing-e2e-tests.md index d523f73057..ac70f3a28c 100644 --- a/doc/test-framework/writing-e2e-tests.md +++ b/doc/test-framework/writing-e2e-tests.md @@ -194,31 +194,71 @@ functions will automatically be run since they were deferred when the TestCtx wa ## Running the Tests -To make running the tests simpler, the `operator-sdk` CLI tool has a `test` subcommand that configures some -default test settings, such as locations of the manifest files for your global resource manifest file (by default `deploy/crd.yaml`) and your namespaced manifest file (by defualt `deploy/rbac.yaml` concatenated with `deploy/operator.yaml`), and allows the user to configure these runtime options. To use it, run the -`operator-sdk test` command in your project root and pass the location of the tests using the -`--test-location` flag. You can use `--help` to view the other configuration options and use -`--go-test-flags` to pass in arguments to `go test`. Here is an example command: +To make running the tests simpler, the `operator-sdk` CLI tool has a `test` subcommand that can configure some +default test settings, such as locations of your global resource manifest file (by default +`deploy/crd.yaml`) and your namespaced manifest file (by default `deploy/sa.yaml` concatenated with +`deploy/rbac.yaml` and `deploy/operator.yaml`), and allows the user to configure runtime options. There are 2 ways to use the +subcommand: local and cluster. +### Local +To run the tests locally, run the `operator-sdk test local` command in your project root and pass the location of the tests +as an argument. You can use `--help` to view the other configuration options and use `--go-test-flags` to pass in arguments to `go test`. Here is an example command: ```shell -$ operator-sdk test --test-location ./test/e2e --go-test-flags "-v -parallel=2" +$ operator-sdk test local ./test/e2e --go-test-flags "-v -parallel=2" ``` -For more documentation on the `operator-sdk test` command, see the [SDK CLI Reference][sdk-cli-ref] doc. +For more documentation on the `operator-sdk test local` command, see the [SDK CLI Reference][sdk-cli-ref] doc. For advanced use cases, it is possible to run the tests via `go test` directly. As long as all flags defined in [MainEntry][main-entry-link] are declared, the tests will run correctly. Running the tests directly with missing flags -will result in undefined behavior. This is an example `go test` equivalent to the `operator-sdk test` example above: +will result in undefined behavior. This is an example `go test` equivalent to the `operator-sdk test local` example above: ```shell -# Combine rbac and operator manifest into namespaced manifest -$ cp deploy/rbac.yaml deploy/namespace-init.yaml +# Combine sa, rbac, operator manifest into namespaced manifest +$ cp deploy/sa.yaml deploy/namespace-init.yaml +$ echo -e "\n---\n" >> deploy/namespace-init.yaml +$ cat deploy/rbac.yaml >> deploy/namespace-init.yaml $ echo -e "\n---\n" >> deploy/namespace-init.yaml $ cat deploy/operator.yaml >> deploy/namespace-init.yaml # Run tests $ go test ./test/e2e/... -root=$(pwd) -kubeconfig=$HOME/.kube/config -globalMan deploy/crd.yaml -namespacedMan deploy/namespace-init.yaml -v -parallel=2 ``` +### Cluster + +Another way to run the tests is from within a kubernetes cluster. To do this, you first need to build an image with +the testing binary embedded by using the `operator-sdk build` command and using the `--enable-tests` flag to enable tests: + +```shell +$ operator-sdk build quay.io/example/memcached-operator:v0.0.1 --enable-tests +``` + +Note that the namespaced yaml must be up to date before running this command. The `build` subcommand will warn you +if it finds a deployment in the namespaced manifest with an image that doesn't match the argument you provided. The +`operator-sdk build` command has other flags for configuring the tests that can be viewed with the `--help` flag +or at the [SDK CLI Reference][sdk-cli-ref]. + +Once the image is ready, the tests are ready to be run. To run the tests, make sure you have all global resources +and a namespace with proper rbac configured: + +```shell +$ kubectl create -f deploy/crd.yaml +$ kubectl create namespace memcached-test +$ kubectl create -f deploy/sa.yaml -n memcached-test +$ kubectl create -f deploy/rbac.yaml -n memcached-test +``` + +Once you have your environment properly configured, you can start the tests using the `operator-sdk test cluster` command: + +```shell +$ operator-sdk test cluster quay.io/example/memcached-operator:v0.0.1 --namespace memcached-test + +Example Output: +Test Successfully Completed +``` + +If the tests fail, the command will output the errors in the standard way that `go test` prints errors and logs. + ## Manual Cleanup While the test framework provides utilities that allow the test to automatically be cleaned up when done, From 9107c756799d7d8a837ec5e1844b7b1d7c73b5d3 Mon Sep 17 00:00:00 2001 From: Alexander Pavel Date: Tue, 25 Sep 2018 11:21:51 -0700 Subject: [PATCH 2/9] dock/sdk-cli-reference.md: update for incluster testing support --- doc/sdk-cli-reference.md | 51 ++++++++++++++++++++++++++++++---------- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/doc/sdk-cli-reference.md b/doc/sdk-cli-reference.md index 56c34ddd1c..843cf03bb2 100644 --- a/doc/sdk-cli-reference.md +++ b/doc/sdk-cli-reference.md @@ -9,16 +9,20 @@ Usage: ### Args -* image - is the container image to be built, e.g. "quay.io/example/operator:v0.0.1". This image will be automatically set in the deployment manifests. +* image - is the container image to be built, e.g. "quay.io/example/operator:v0.0.1". ### Flags - +* `-e, --enable-tests` bool - enable in-cluster testing by adding test binary to the image +* `-n, --namespaced` string - path of namespaced resources for tests (default "deploy/operator.yaml") +* `-t, --test-location` string - location of tests (default "./test/e2e") * `-h, --help` - help for build + ### Use -The operator-sdk build command compiles the code, builds the executables, -and generates Kubernetes manifests. After build completes, the image would be built locally in docker. Then it needs to be pushed to remote registry. +The operator-sdk build command compiles the code and builds the executables. After build completes, the image is built locally in docker. Then it needs to be pushed to remote registry. + +If `--enable-tests` is set, the buld command will also build the testing binary and add it to the docker image. ### Example: @@ -158,30 +162,53 @@ Create app-operator/.gitignore ## test -### Flags +### Available Commands + +#### local - Runs the tests locally -* `-t, --test-location` **(required)** string - location of e2e test files +##### Flags * `-k, --kubeconfig` string - location of kubeconfig for kubernetes cluster * `-g, --global-init` string - location of global resource manifest yaml file * `-n, --namespaced-init` string - location of namespaced resource manifest yaml file * `-f, --go-test-flags` string - extra arguments to pass to `go test` (e.g. -f "-v -parallel=2") -* `-h, --help` - help for test +* `-h, --help` - help for local -### Use +##### Use The operator-sdk test command runs go tests built using the Operator SDK's test framework. -### Example: - -#### Test +##### Example: ```bash -operator-sdk test --test-location ./test/e2e/ +$ operator-sdk test local ./test/e2e/ # Output: ok github.com/operator-framework/operator-sdk-samples/memcached-operator/test/e2e 20.410s ``` +#### cluster - Runs the tests within a cluster + +##### Flags +* `-k, --kubeconfig` string - location of kubeconfig for kubernetes cluster +* `-i, --imagePullPolicy` string - set test pod image pull policy. Allowed values: Always, Never (default "Always") +* `-n, --namespace` string - namespace to run tests in (default "default") +* `-p, --pendingTimeout` int - timeout for testing pod in pending state (default 60) +* `-s, --serviceAccount` string - service account to run tests on (default "default") +* `-h, --help` - help for cluster + +##### Use + +The operator-sdk test command runs go tests embedded in an operator image built using the Operator SDK. + +##### Example: + +```bash +$ operator-sdk test cluster quay.io/example/memcached-operator:v0.0.1 + +# Output: +Test Successfully Completed +``` + ## up ### Available Commands From adc00e8cecae0274cabcfea0af60d17da7683172 Mon Sep 17 00:00:00 2001 From: Alexander Pavel Date: Tue, 25 Sep 2018 11:28:43 -0700 Subject: [PATCH 3/9] doc/test-framework/writing-e2e-tests.md: improve wording --- doc/test-framework/writing-e2e-tests.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/test-framework/writing-e2e-tests.md b/doc/test-framework/writing-e2e-tests.md index ac70f3a28c..a836d0a819 100644 --- a/doc/test-framework/writing-e2e-tests.md +++ b/doc/test-framework/writing-e2e-tests.md @@ -194,9 +194,9 @@ functions will automatically be run since they were deferred when the TestCtx wa ## Running the Tests -To make running the tests simpler, the `operator-sdk` CLI tool has a `test` subcommand that can configure some +To make running the tests simpler, the `operator-sdk` CLI tool has a `test` subcommand that can configure default test settings, such as locations of your global resource manifest file (by default -`deploy/crd.yaml`) and your namespaced manifest file (by default `deploy/sa.yaml` concatenated with +`deploy/crd.yaml`) and your namespaced resource manifest file (by default `deploy/sa.yaml` concatenated with `deploy/rbac.yaml` and `deploy/operator.yaml`), and allows the user to configure runtime options. There are 2 ways to use the subcommand: local and cluster. ### Local @@ -257,7 +257,7 @@ Example Output: Test Successfully Completed ``` -If the tests fail, the command will output the errors in the standard way that `go test` prints errors and logs. +If the tests fail, the command will output the errors in the standard `go test` loggin/error format. ## Manual Cleanup From 2218b729b18c79093d318355c2cbe5ab42c273e5 Mon Sep 17 00:00:00 2001 From: Alexander Pavel Date: Tue, 25 Sep 2018 11:50:13 -0700 Subject: [PATCH 4/9] small improvements --- doc/sdk-cli-reference.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/sdk-cli-reference.md b/doc/sdk-cli-reference.md index 843cf03bb2..8b2ac24644 100644 --- a/doc/sdk-cli-reference.md +++ b/doc/sdk-cli-reference.md @@ -20,9 +20,9 @@ Usage: ### Use -The operator-sdk build command compiles the code and builds the executables. After build completes, the image is built locally in docker. Then it needs to be pushed to remote registry. +The operator-sdk build command compiles the code and builds the executables. After build completes, the image is built locally in docker. Then it needs to be pushed to a remote registry. -If `--enable-tests` is set, the buld command will also build the testing binary and add it to the docker image. +If `--enable-tests` is set, the build command will also build the testing binary and add it to the docker image. ### Example: From e9c974a1b94497d3dc783c68712f9daebc8bb57f Mon Sep 17 00:00:00 2001 From: Alexander Pavel Date: Tue, 25 Sep 2018 13:29:41 -0700 Subject: [PATCH 5/9] doc/sdk-cli-reference.md: use new flags --- doc/sdk-cli-reference.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/doc/sdk-cli-reference.md b/doc/sdk-cli-reference.md index 8b2ac24644..f514a3c2e5 100644 --- a/doc/sdk-cli-reference.md +++ b/doc/sdk-cli-reference.md @@ -12,9 +12,9 @@ Usage: * image - is the container image to be built, e.g. "quay.io/example/operator:v0.0.1". ### Flags -* `-e, --enable-tests` bool - enable in-cluster testing by adding test binary to the image -* `-n, --namespaced` string - path of namespaced resources for tests (default "deploy/operator.yaml") -* `-t, --test-location` string - location of tests (default "./test/e2e") +* `--enable-tests` bool - enable in-cluster testing by adding test binary to the image +* `--namespaced-manifest` string - path of namespaced resources manifest for tests (default "deploy/operator.yaml") +* `--test-location` string - location of tests (default "./test/e2e") * `-h, --help` - help for build @@ -167,10 +167,10 @@ Create app-operator/.gitignore #### local - Runs the tests locally ##### Flags -* `-k, --kubeconfig` string - location of kubeconfig for kubernetes cluster -* `-g, --global-init` string - location of global resource manifest yaml file -* `-n, --namespaced-init` string - location of namespaced resource manifest yaml file -* `-f, --go-test-flags` string - extra arguments to pass to `go test` (e.g. -f "-v -parallel=2") +* `--kubeconfig` string - location of kubeconfig for kubernetes cluster +* `--global-manifest` string - location of global resource manifest yaml file +* `--namespaced-manifest` string - location of namespaced resource manifest yaml file +* `--go-test-flags` string - extra arguments to pass to `go test` (e.g. -f "-v -parallel=2") * `-h, --help` - help for local ##### Use @@ -189,12 +189,12 @@ ok github.com/operator-framework/operator-sdk-samples/memcached-operator/test/ #### cluster - Runs the tests within a cluster ##### Flags -* `-k, --kubeconfig` string - location of kubeconfig for kubernetes cluster -* `-i, --imagePullPolicy` string - set test pod image pull policy. Allowed values: Always, Never (default "Always") -* `-n, --namespace` string - namespace to run tests in (default "default") -* `-p, --pendingTimeout` int - timeout for testing pod in pending state (default 60) -* `-s, --serviceAccount` string - service account to run tests on (default "default") -* `-h, --help` - help for cluster +* `--kubeconfig` string - location of kubeconfig for kubernetes cluster +* `--image-pull-policy` string - set test pod image pull policy. Allowed values: Always, Never (default "Always") +* `--namespace` string - namespace to run tests in (default "default") +* `--pending-timeout` int - timeout for testing pod in pending state (default 60) +* `--service-account` string - service account to run tests on (default "default") +* `--help` - help for cluster ##### Use From 510a65cb0b60396fe338829905c5269e0db5c523 Mon Sep 17 00:00:00 2001 From: Alexander Pavel Date: Tue, 25 Sep 2018 15:44:41 -0700 Subject: [PATCH 6/9] commands/.../cluster,doc: suggestions by hasbro17 --- commands/operator-sdk/cmd/test/cluster.go | 2 +- doc/sdk-cli-reference.md | 14 +++++++++++--- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/commands/operator-sdk/cmd/test/cluster.go b/commands/operator-sdk/cmd/test/cluster.go index cdd5cfd050..8c0ae4c790 100644 --- a/commands/operator-sdk/cmd/test/cluster.go +++ b/commands/operator-sdk/cmd/test/cluster.go @@ -56,7 +56,7 @@ func NewTestClusterCmd() *cobra.Command { testCmd.Flags().StringVar(&tcConfig.kubeconfig, "kubeconfig", defaultKubeConfig, "Kubeconfig path") testCmd.Flags().StringVar(&tcConfig.imagePullPolicy, "image-pull-policy", "Always", "Set test pod image pull policy. Allowed values: Always, Never") testCmd.Flags().StringVar(&tcConfig.serviceAccount, "service-account", "default", "Service account to run tests on") - testCmd.Flags().IntVar(&tcConfig.pendingTimeout, "pending-timeout", 60, "Timeout for testing pod in pending state") + testCmd.Flags().IntVar(&tcConfig.pendingTimeout, "pending-timeout", 60, "Timeout in seconds for testing pod to stay in pending state (default 60s)") return testCmd } diff --git a/doc/sdk-cli-reference.md b/doc/sdk-cli-reference.md index f514a3c2e5..366b83350b 100644 --- a/doc/sdk-cli-reference.md +++ b/doc/sdk-cli-reference.md @@ -164,7 +164,11 @@ Create app-operator/.gitignore ### Available Commands -#### local - Runs the tests locally +#### local +Runs the tests locally + +##### Args +* string - location of e2e test files (e.g. "./test/e2e/") ##### Flags * `--kubeconfig` string - location of kubeconfig for kubernetes cluster @@ -186,13 +190,17 @@ $ operator-sdk test local ./test/e2e/ ok github.com/operator-framework/operator-sdk-samples/memcached-operator/test/e2e 20.410s ``` -#### cluster - Runs the tests within a cluster +#### cluster +Runs the e2e tests packaged in an operator image as a pod in the cluster + +##### Args +* string - the operator image that is used to run the tests in a pod (e.g. "quay.io/example/memcached-operator:v0.0.1") ##### Flags * `--kubeconfig` string - location of kubeconfig for kubernetes cluster * `--image-pull-policy` string - set test pod image pull policy. Allowed values: Always, Never (default "Always") * `--namespace` string - namespace to run tests in (default "default") -* `--pending-timeout` int - timeout for testing pod in pending state (default 60) +* `--pending-timeout` int - timeout in seconds for testing pod to stay in pending state (default 60s) * `--service-account` string - service account to run tests on (default "default") * `--help` - help for cluster From faa2a15db73d890062b2a4031385b00d54bbfe9d Mon Sep 17 00:00:00 2001 From: Alexander Pavel Date: Tue, 25 Sep 2018 16:04:44 -0700 Subject: [PATCH 7/9] doc/sdk-cli-reference.md: mention flag defaults [skip ci] --- doc/sdk-cli-reference.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/sdk-cli-reference.md b/doc/sdk-cli-reference.md index 366b83350b..8d98f46b18 100644 --- a/doc/sdk-cli-reference.md +++ b/doc/sdk-cli-reference.md @@ -171,9 +171,9 @@ Runs the tests locally * string - location of e2e test files (e.g. "./test/e2e/") ##### Flags -* `--kubeconfig` string - location of kubeconfig for kubernetes cluster -* `--global-manifest` string - location of global resource manifest yaml file -* `--namespaced-manifest` string - location of namespaced resource manifest yaml file +* `--kubeconfig` string - location of kubeconfig for kubernetes cluster (default "~/.kube/config") +* `--global-manifest` string - path to manifest for global resources (default "deploy/crd.yaml) +* `--namespaced-manifest` string - path to manifest for per-test, namespaced resources (default: combines deploy/sa.yaml, deploy/rbac.yaml, and deploy/operator.yaml) * `--go-test-flags` string - extra arguments to pass to `go test` (e.g. -f "-v -parallel=2") * `-h, --help` - help for local @@ -197,7 +197,7 @@ Runs the e2e tests packaged in an operator image as a pod in the cluster * string - the operator image that is used to run the tests in a pod (e.g. "quay.io/example/memcached-operator:v0.0.1") ##### Flags -* `--kubeconfig` string - location of kubeconfig for kubernetes cluster +* `--kubeconfig` string - location of kubeconfig for kubernetes cluster (default "~/.kube/config") * `--image-pull-policy` string - set test pod image pull policy. Allowed values: Always, Never (default "Always") * `--namespace` string - namespace to run tests in (default "default") * `--pending-timeout` int - timeout in seconds for testing pod to stay in pending state (default 60s) From 4d0c2e2dbbb169fb710c62c3e206eeee8652b2ce Mon Sep 17 00:00:00 2001 From: Alexander Pavel Date: Tue, 25 Sep 2018 16:16:51 -0700 Subject: [PATCH 8/9] doc/sdk-cli-reference.md: mention the test-pod yaml --- doc/sdk-cli-reference.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/sdk-cli-reference.md b/doc/sdk-cli-reference.md index 8d98f46b18..f61b4f87b6 100644 --- a/doc/sdk-cli-reference.md +++ b/doc/sdk-cli-reference.md @@ -22,7 +22,8 @@ Usage: The operator-sdk build command compiles the code and builds the executables. After build completes, the image is built locally in docker. Then it needs to be pushed to a remote registry. -If `--enable-tests` is set, the build command will also build the testing binary and add it to the docker image. +If `--enable-tests` is set, the build command will also build the testing binary, add it to the docker image, and generate +a `deploy/test-pod.yaml` file that allows a user to run the tests as a pod on a cluster. ### Example: From 9480d464449fa40b2b263b6e01447f4693ff6888 Mon Sep 17 00:00:00 2001 From: Alexander Pavel Date: Wed, 26 Sep 2018 12:06:51 -0700 Subject: [PATCH 9/9] doc/test-framework: nits --- doc/test-framework/writing-e2e-tests.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/test-framework/writing-e2e-tests.md b/doc/test-framework/writing-e2e-tests.md index a836d0a819..a08e4356c2 100644 --- a/doc/test-framework/writing-e2e-tests.md +++ b/doc/test-framework/writing-e2e-tests.md @@ -257,7 +257,9 @@ Example Output: Test Successfully Completed ``` -If the tests fail, the command will output the errors in the standard `go test` loggin/error format. +The `test cluster` command will deploy a test pod in the given namespace that will run the e2e tests packaged in the image. +The command will wait until the tests succeed (pod phase=`Succeeded`) or fail (pod phase=`Failed`). +If the tests fail, the command will output the test pod logs which should be the standard go test error logs. ## Manual Cleanup