Implement tests with plugable upgrade mechanism

[mgencur]

Describe the feature

The current upgrade/downgrade tests are orchestrated from Bash. It runs pre-upgrade Golang tests, calls Bash functions to perform the upgrade, runs post-upgrade Golang tests, calls Bash functions for downgrade, runs post-downgrade Golang tests.

This is a proposal for changing the orchestration where the Golang test suite including pre-upgrade, post-upgrade and post-downgrade tests would run as a monolithic test suite, performing upgrades and downgrades using a plugable mechanism.

The scenario would look like this:

• Start the single Golang test suite for upgrades/downgrades
• The test suite runs pre-upgrade tests
• A plugin is called to perform an upgrade
• The test suite runs post-upgrade tests
• A plugin is called to perform a downgrade
• The test suite runs post-downgrade tests
• The Golang test suite ends

Upstream tests would include a default plugin that would call bash scripts for upgrading/downgrading Knative.
Downstream tests could provide their own mechanism for upgrades/downgrades, possibly using Golang API. If no extra plugin is provided the test suite would execute the default plugin.

Advantages of this approach:

• makes it possible to run the whole test suite from Golang and step through the program/debug it (this would be more convenient in downstream where upgrades are done using Golang API but it can be used in upstream as well)
• removes the need for the system to communicate with the prober test via pipe in order to end it
• more cohesive upgrade/downgrade test suite
• easier execution in environments other than upstream

[mgencur]

/assign

[cardil]

I would go even further. We got upgrade tests in eventing as well, and also merger of those two in knative/operator upgrade tests. That's not ideal.

I would propose to make it fully pluggable test suite. Test suite should be located in some common place, like new knative.dev/pkg/test/upgrade package.

The modified scenario would look like this:

• Start the single Golang test suite for upgrades/downgrades
• Register appropriate plugins for each step
• The test suite runs pre-upgrade tests from plugins
• Plugins are called to perform an upgrade
• The test suite runs post-upgrade tests from plugins
• Plugins are called to perform a downgrade
• The test suite runs post-downgrade tests from plugins
• The Golang test suite ends

In that way we can do it properly, once. And then reuse that code in Serving, Eventing, Operator and in many places downstream as well.

/cc @houshengbo

[coryrc]

We should use Tekton for orchestration of the various test components.

Supporting a custom-written test orchestrator in Go might be less work than maintaining one in bash, but it's still a custom test orchestrator and we should use a standard tool instead of reinventing our own.

[cardil]

That would require to have working Tekton installation. Additional dependency, additional time to set up.

I think, here we should have a single upgrade test logic, but pluggable, that we can adjust to perform different operations (plugins) on specific steps. In example, we on downstream would like to execute serving & eventing upgrade tests in the same run, go upgrading Serverless Operator and OCP cluster, instead of upgrading serving or eventing separately.

[chizhg]

kntest is under development, we can potentially make it as a subcommand.

Also for a bit more context, we are currently using kubetest2 as the test orchestrator, and it supports plugins just like kn and kubectl. A customer knative-upgrade-tester plugin implementation should also fit in here.

[cardil]

@chizhg PTAL at my refined proposition at knative/pkg#1638

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[cardil]

/reopen
/remove-lifecycle stale

PR #10216 should fix this