Skip to content

scorecard: use TestList for output, exit 1 on test failure#3427

Merged
joelanford merged 2 commits intooperator-framework:masterfrom
joelanford:scorecard-output-api
Jul 15, 2020
Merged

scorecard: use TestList for output, exit 1 on test failure#3427
joelanford merged 2 commits intooperator-framework:masterfrom
joelanford:scorecard-output-api

Conversation

@joelanford
Copy link
Copy Markdown
Member

Description of the change:

  1. Instead of aggregating all test results from ALL scorecard tests under a single v1alpha3.Test status, use a new v1alpha3.TestList to output a list of v1alpha3.Test objects, one for each test defined in the scorecard config file.
  2. Exit with status 1 when any test fails.

Motivation for the change:

  1. The test results should not be disassociated from the test spec (image/entrypoint) that provided the result.
  2. To match existing documentation

Checklist

If the pull request includes user-facing changes, extra documentation is required:

@joelanford joelanford added this to the v1.0.0 milestone Jul 15, 2020
@joelanford joelanford force-pushed the scorecard-output-api branch from 800649e to d8ab436 Compare July 15, 2020 03:26
@jmccormick2001
Copy link
Copy Markdown
Contributor

seems to work for the basic/olm case, the kuttl image needs the updates for that to work again, also the custom test image would need the same updates.

@joelanford
Copy link
Copy Markdown
Member Author

Just tested kuttl, it also works, which is expected because this PR does not change anything about the pod log interface and result gathering.

@joelanford joelanford merged commit 4d8960c into operator-framework:master Jul 15, 2020
@joelanford joelanford deleted the scorecard-output-api branch July 15, 2020 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants