"Wait" is overly used in e2e/conformance

[mattmoor]

The conformance (and possibly e2e) tests have a number of checks that poll for success. This masks problems where we say we're Ready but aren't actually.

We should change the e2e test to use Wait methods a bit more sparingly and then concretely check the state of the system.

If this is too flaky, we can consider adding retries, but the idea of "Wait" feels wrong to me.

/assign @mattmoor

I'll see how flaky this is after we get it re-enabled with Wait in place.

[mattmoor]

Sent out a PoC (linked above).

[mattmoor]

It is notable that the small change I made in the PoC above caught a bug I almost introduced in #1322.
The Revision's status would have converged to what we expected after a resync, but because we were "Check"ing vs. "Wait"ing it blocked my PR. I'd like to see us tighten up more of this checking to hopefully squeeze out other issues we might be missing.

/kind API
/kind spec
/assign @jonjohnsonjr