Suite for running upgrade tests in Knative components

[cardil]

Fixes #1638

This introduces a pluggable test suite for running upgrade tests in Knative components. Test suite is fully configurable, so that can be used for upgrade testing in Knative Serving, and Eventing, and Operator. Different configuration can be used by vendors that needs this to be slightly different.

This change will dramatically reduce the needed Bash code there is and let us run Serving and Eventing upgrade tests in the same time.

The idea in this PR is to define reusable plugins in Knative components (Serving, Eventing), use them in test code, and reuse them in integration (Operator) and vendor projects (like OpenShift Serverless). Creation of plugins can be done by implementing Operation or BackgroundOperation interfaces, or by using convenience methods of NewOperation, NewBackgroundOperation (with WaitForStopEvent), or NewBackgroundVerification.

Below are examples of such plugin wrote for Serving continual tests (that will be running in background) and a complete upgrade test:

// file: knative.dev/serving/test/e2e/upgrade/continual.go
package upgrade

import (
  pkgupgrade "knative.dev/pkg/test/upgrade"
  "knative.dev/serving/test/e2e"
)

func ContinualTest() pkgupgrade.BackgroundOperation {
  var (
    namespace string
    service   string
  )
  return upgrade.NewBackgroundVerification("Serving continual test",
    func(c upgrade.Context) {
      namespace = e2e.GenerateTestName("serving-upgrades")
      service = e2e.GenerateTestName("prober")
      e2e.Clients.Knative.Serving.Namespace(namespace).Create(Service{
         // [..]
      })
    },
    func(c upgrade.Context) {
      ksvc := e2e.Clients.Knative.Serving.Namespace(namespace).Get(service)
      // do a checking
    },
  )
}

and example test that uses it:

// +build upgrade
// file: knative.dev/serving/test/e2e/upgrade_test.go
package e2e

import (
  "testing"

  "go.uber.org/zap"
  "knative.dev/pkg/test/upgrade"
  serving "knative.dev/serving/test/e2e/upgrade"
)

func TestServingUpgradesDowngrades(t *testing.T) {
  suite := upgrade.Suite{
    Tests: upgrade.Tests{
      PreUpgrade: []upgrade.Operation{
        serving.SmokeTest(),
      },
      PostUpgrade: []upgrade.Operation{
        serving.SmokeTest(),
      },
      PostDowngrade: []upgrade.Operation{
        serving.SmokeTest(),
      },
      Continual: []upgrade.BackgroundOperation{
        serving.ContinualTest(),
      },
    },
    Installations: upgrade.Installations{
      Base: []upgrade.Operation{
        serving.InstallLatestStable(),
      },
      UpgradeWith: []upgrade.Operation{
        serving.GitHead(),
      },
      DowngradeWith: []upgrade.Operation{
        serving.InstallLatestStable(),
      },
    },
  }
  c := newUpgradeConfig(t)
  suite.Execute(c)
}

func newUpgradeConfig(t *testing.T) upgrade.Configuration {
  log, err := zap.NewDevelopment()
  if err != nil {
    t.Fatal(err)
  }
  return upgrade.Configuration{T: t, Log: log}
}

[knative-prow-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[mattmoor]

Produced via:
gofmt -s -w $(find -path './vendor' -prune -o -path './third_party' -prune -o -name '*.pb.go' -prune -o -type f -name '*.go' -print)
goimports -w $(find -name '*.go' | grep -v vendor | grep -v third_party | grep -v .pb.go | grep -v wire_gen.go)

[mattmoor]

Produced via:
gofmt -s -w $(find -path './vendor' -prune -o -path './third_party' -prune -o -name '*.pb.go' -prune -o -type f -name '*.go' -print)
goimports -w $(find -name '*.go' | grep -v vendor | grep -v third_party | grep -v .pb.go | grep -v wire_gen.go)

[mgencur]

Hey @cardil , I think this looks great! Like it very much. I've put some comments here and there. There's probably some room for improvement to make it slightly cleaner or easier for the implementor of the background operations. Thanks!

[knative-metrics-robot]

The following is the coverage report on the affected files.
Say /test pull-knative-pkg-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
test/upgrade/functions.go	Do not exist	100.0%
test/upgrade/steps.go	Do not exist	95.5%
test/upgrade/suite_execution.go	Do not exist	94.3%
test/upgrade/vars.go	Do not exist	N/A

[mgencur]

@cardil thanks for all the changes! I think it looks good now. Looking forward to using that.

[mgencur]

/lgtm

[chizhg]

/approve

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cardil, chizhg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• test/OWNERS [chizhg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment