Use K8s etcd database to publish and fetch upgrade tests reports

[cardil]

Problem
PR #2388 introduced upgrade tests for eventing. In those I fetch execution report after the test. It's done with simple REST invocation. Drawback of this approach is the need to somehow connect to that REST endpoint from outside the cluster. It's done by getting external address of one of the nodes and usage of NodePort. @vaikas during PR review propose that maybe we can use K8s infrastructure itself, especially etcd database. I like this idea, as it will be K8s native and work in any K8s installation.

I think we can rewrite the REST endpoint code to be asynchronous, and work by requesting report being produced as a simple ConfigMap that contains JSON report. This request can be done, also by creating specific ConfigMap or maybe K8s Event object. Also, specific for upgrade test, request can also be omitted as there is already Finished event being propagated, so we can create a report after receiving it. I think that it might be useful to create a general purpose request-response framework, as it can be used to collect data from other types of e2e tests, not being related to eventing, even.

Persona:
Which persona is this feature for? Contributors

Exit Criteria
REST endpoint for upgrade tests is removed, as well as node IP resolver code. Execution report is fetch as a ConfigMap from the cluster.

Time Estimate (optional):
10d

[vaikas]

@slinkydeveloper Are there any other remaining tasks, or can we call this good?

[slinkydeveloper]

@vaikas WDYM?

@cardil I think you can reuse recordevents as is, without any changes to the test image itself. When the report is ready, send the result wrapped in a cloudevent to the recordevents image brought up and then assert on the other side using the eventTracker (just look at one of our usages of recordevents inside an e2e and you'll figure it out)

[vaikas]

@slinkydeveloper when I asked (offline the other day) if we had changed all uses, I mistook it to mean you had converted these as well :) My bad.

[slinkydeveloper]

@vaikas ah no i left the upgrade tests untouched, these don't use recordevents, transformevents and friends but wathola-* images

[cardil]

At @slinkydeveloper

When the report is ready

I actually notify cluster (HTTP test runner outside cluster -> wathola-receiver service) that tests has been finished, and that produce the report that I parse back at test runner. How I would achieve the same with recordevents? It communication is outside of the cluster, as test are executed outside.

[slinkydeveloper]

@cardil with recordevents the idea is pretty simple: recordevents push k8s events, test runner watches for k8s events and parse them

[cardil]

The problem is to notify service from client, from outside cluster, that test should end.

Can I create such k8s event from outside cluster?

I was thinking about using configmap. So, that receiver would wait for specific configmap to tell it should end. If so it will generate report k8s event, which could be read from client.

[slinkydeveloper]

Can I create such k8s event from outside cluster?

Sure you can, the test runner always has "admin" access to the cluster