Operator Image Testing

[AlexNPavel]

Cluster admins need a way to verify that an operator image works correctly before actually deploying it on a live cluster. One way to do this would be to bundle a test binary into the operator's image that can be run to test the operator's functionality. We now have an e2e test framework that users can develop on, which should be a good starting point for this setup. One potential way to set up an image that could work this way:

1. Use go test -c to output the tests in binary form. We can add this to the build script.
2. Add the test binary to the Dockerfile to be included in the image.
3. Allow the user to specify which manifests to use for global and namespaced test resources in the same way that operator-sdk test does. Add these manifests to the Dockerfile.
4. Use a script that would enable the correct flags for the test (project root, manifest paths, etc) that would allow the tests to run correctly when called.

Potential issues:

1. How does a user enable this (new subcommand, flags for the build command, or something else)?
2. How would permissions be handled? The most logical way to run tests from an image would be a pod in a cluster. The e2e tests require admin access however, since they need the ability to create and delete namespaces. Most admins likely wouldn't want to give admin access to a pod for testing purposes.
3. Image size. Due to compression, the image size in repos shouldn't be affected to much, but it is worthwhile to note that a test binary would nearly double the size of the container when decompressed (the test binary would be ~35M; the operator is also ~35M, and the alpine base image is ~5M).

On top of the potential issues listed above, design is another important question. What conventions should be followed when designing this to better match other related products in Openshift?

Original Google Groups request: https://groups.google.com/d/msg/operator-framework/xeQ3VIxf3pQ/epmY67I3CwAJ

[AlexNPavel]

Work to be done
I will add tasks that need to be completed for this based on the original post above and change them as we decide on a final design.

• Add the test binary to the build.sh script
• Add support for an in-cluster kube config in the test framework
• Add the test binary and manifests to the docker image
• Add a script to automatically set correct flags for the test in the image

[AlexNPavel]

Design Update (work will be in PR #469):

Additions to build subcommand: Build will now support building images with the tests inside them and will create a test-pod.yaml file that has the manifest to start the image in testing mode. We assume that the user has created any necessary global resources as well as the rbac in the namespace. The user can start the test pod with kubectl create -f deploy/test-pod.yaml.

Changes to the test subcommand: Will be split into 2 separate subcommands:
test local: Test local will be identical to the current test subcommand, with the exception of the --test-location flag, which we will turn into an argument as it's more logical and consistent with other subcommands in the SDK.
test cluster: Test cluster will take an image name as an argument and then support a --global flag for global resources that should be created before the test and a --kubeconfig flag for the kubeconfig required to access the cluster that the image is being tested on. The test pod will be created using client-go and will not depend on the test-pod.yaml that the build command now generates.

[AlexNPavel]

Update: There will not be a -g flag for the test cluster subcommand as that can lead to a variety of issues, and we will instead require the tester to ensure that the necessary resources exist before running the test

[AlexNPavel]

Completed with PR #469