Make the eventstore pods more resilient to failures. Causing flakes in our tests.

[vaikas]

Describe the bug
Many recent flakes have had errors that looks like so (just choosing one example, but there are others). These cause flaky tests, so we should really figure out what's going on here so that our tests stability improves.

    TestChannelBasedBrokerWithManyTriggers/Channel-messaging.knative.dev/v1/test_default_broker_with_many_attribute_triggers_using_v1beta1_trigger: event_info_store.go:111: Failed to start the EventInfoStore associated to pod 'dumper-testany-testany--extname2-extvalue2': timeout waiting for recordevents pod to come up: http get error: Get "http://localhost:40258/minmax": dial tcp 127.0.0.1:40258: connect: connection refused

Expected behavior
Test infra to be more robust towards failures.

To Reproduce
Look at some recent flakey tests failures and grep for this string.

Knative release version
HEAD

Additional context
Add any other context about the problem here such as proposed priority

[grantr]

@ericlem @slinkydeveloper @cardil I think you're all involved in this new testing infrastructure. Any idea what could be causing this?

[slinkydeveloper]

I had a look a while ago and I added a wait for the event record pod to be ready before starting invoking it. That change seemed to have fixed the issue, but now it's back again here...

[slinkydeveloper]

I start having the feeling the problem is the port forwarding, i'm working on increasing the logging there: knative/pkg#1531

[lionelvillard]

@slinkydeveloper maybe you can look for inspiration there: https://github.com/knative/pkg/blob/master/test/logstream/kubelogs.go

[lberk]

/assign @slinkydeveloper