Metrics: E2E Test for InMemoryChannel

[nachocano]

Problem
In order to ensure that the InMemoryChannel generates metrics correctly, we need an E2E test.

Persona:
Contributor

Exit Criteria
A test in the e2e suite that runs before every PR merge. The test verifies that an event sent from an InMemoryChannel sends at least event_count metrics.

Additional Context
This depends on #1818 being done

[Harwayne]

/project Observability To do

[akashrv]

/milestone v0.10.0

[anishj0shi]

I'd like to work on this

/assign

[nachocano]

@anishj0shi this still needs for the IMC to report data plane metrics....
I'm doing a big refactor here #1933 to allow easy addition of data plane metrics to the InMemoryChannel, and all channels in general... but until that gets merged, I think it might be hard to start this test...

This main issue has all the pending items we have currently for metrics #1693. If you want to start with some e2e test, I'd suggest for example adding to the ApiServerSource #1815...

But any of those e2e tests will need some infrastructure setup, i.e., we will need to write some code to startup a prometheus instance, configure the observability configmap to send metrics to prometheus, and collect metrics from prometheus... This issue tracks that #1920
It will be somewhat similar to what @Harwayne has done for Zipkin tracing, starting up zipkin, and then being able to collect traces from there.

If you want to pursue the infra setup, then you can follow some of the zipkin tracing e2e setup, and try to add something "similar" for prometheus... Does it make sense?

[anishj0shi]

/unassign
@nachocano Absolutely, I'm assigning myself to #1920