The custom-scorecard-tests has been tested just with GO

[camilamacedo86]

Feature Request

Describe the problem you need a feature to resolve.

Ensure that Ansible/Helm will also work with this feature

Describe the solution you'd like.

• Add the test to Ansible/Helm e2e tests

operator-sdk/test/e2e/e2e_suite_test.go

Line 230 in 6672d53

By("running custom scorecard tests")

• Note that the suite test in the before we also need to use the helper err = tc.AddScorecardCustomPatchFile()

[camilamacedo86]

in the bug triage meeting, it was discussed and then, shows that we also need to discuss if there's really a need for the custom test example at all in the Go as well. According to @joelanford After all, the basic and olm tests are probably reasonable examples on their own.

[jmccormick2001]

The custom scorecard test was meant to be a small minimal starting point for people wanting to write this sort of custom test image, something they could grab and start using without any use of unexported sdk API code.

The only downside to having people start with the basic/olm tests that I can think of would be they would need to know what to chop out, there might be questions arise when they attempt to do that.

But really I don't think its that big of a deal to remove it, so its fine to me to delete the custom scorecard test image if you want. If thats the direction you want to go, you would just need to update the end user docs to state the changes necessary to refactor the scorecard test image to be something they would implement with their unique tests.

[camilamacedo86]

Hi @jmccormick2001,

It is only about its tests. My suggestion was to ADD the same tests done into the e2e GO with the custom image to Ansible/Helm's e2e as well. Then, in the bug triage meeting, @joelanford and @estroz raised the concern that we might not need to the test to check the custom option in e2e at all and it may do not need to be tested with another type as well.

It shows reasonable, following some questions that might be nice to answer to make the best decision.

• Have we the intention to scaffold the custom scorecard manifests by default on the projects? (I would like to know if we plan that the tool would add the files mocked in scorecard-custom-patch.go )
• When we run operator-sdk scorecard bundle --selector=suite=custom will it use any project manifests or just the image and the files added on scorecard-custom-patch.go ?
• Would possible the custom image works well for a go and not work well for Helm/Ansible?
• What is the downside of not tests the custom scorecard image on e2e? Would really possible unit tests this feature? How we can ensure that the custom test image offered works well as expected if we remove it from the e2e test?

[joelanford]

@camilamacedo86 just for more context about my thought process here. We need to consider the context of each test, what the goal for the test is, and also ask what needs to be tested, and when. So in my mind:

1. e2e tests for the operator types are all about testing that a default scaffolded project works. Since the default scaffold includes the basic and olm tests, we should test those for each operator type. We also test various OLM integrations like packaging generation and execution. Since the custom scorecard tests image is not part of the scaffold for any operator type, it does not make sense to include it in any e2e test for operator scaffolding.
2. We also have (or at least had) subcommand-focused tests for scorecard. I think these have primarily transitioned to the e2e tests and new unit tests. Still in this scenario, we aren't testing the test images, we are testing the scorecard subcommand and its ability to execute tests, collect test output, and interpret it. For this test, we would need something that can return expected and unexpected output. In our case, we have mocks in unit tests and real output from the basic and olm tests. So still in this context, adding the custom tests here would not provide any additional value.
3. Lastly, we ALSO have the custom scorecard tests that we document as an example. It IS desirable to have tests that ensure our custom test example is valid and working with the scorecard APIs. This is similar to our desire to actually test our generated samples.

So having said all of this, I would suggest that we consider combining 1 and 3.

We're already planning to refactor our e2es to test our generated samples. Let's consider re-writing the custom scorecard test image as something that is scaffolded into the generated samples and specific to the example. With this approach, we would have:

1. examples of each operator type and a custom scorecard test for each operator type automatically generated as samples. For example, we could have a simple scorecard test that connects to memcached and ensures it can set and retrieve a value.
2. sanity tests that ensure generated operators and their scorecard custom tests are up-to-date.
3. e2e tests that build the operator and custom scorecard test image in the sample project, verify that the OLM integrations work, that the operator runs and reconciles, and that the custom scorecard test passes.

WDYT @camilamacedo86 @jmccormick2001

Also ping @estroz since you had some thoughts and suggestions as well.

[camilamacedo86]

I liked your idea @joelanford 👍 +1
The only thing that does not actually change the approach, but I'd like to know if we have the intention to scaffold the custom scorecard manifests by default on the projects? Or at least, for now, we have not.

[jmccormick2001]

totally a good idea to try and move the custom scorecard sample to where and how the other samples are done...if that is the case, then you'd only have to scaffold custom scorecard configuration when running/testing those samples since most people building projects won't care to have custom scorecard configs.