E2E Metrics for Importer/Broker/Trigger model

[nachocano]

Problem
This is a placeholder feature for metrics-related tasks that will be driven by the eventing-observability subgroup and scoped down to 0.9, 0.9+ releases.
Here are some initial tasks we identified:

• Infrastructure setup, scraping of metrics - Prometheus

  • Add prometheus operator issue #1925
    • knative-observability changes #52
    • documentation in eventing and docs
  • Control plane
    • Broker Ingress/Filter #5378
    • Eventing Controller #5378
    • ApiServerSource #1747
    • CronJobSource #1747
    • ContainerSource #1747
    • KafkaSource
    • GitHubSource
    • PullSubscription
    • Storage Source #246
    • AWSSqsSource
    • InMemory Channel #1747
    • Kafka Channel #617
    • NATSS Channel #614
    • Channel (from knative-gcp)

• Infrastructure setup and scraping of metrics - Stackdriver

  • Configurable metrics backend #1726
  • Broker ingress/filter changes metrics refactoring / Use stats reporter #1739, #1735
  • pkg changes #612
  • Eventing changes to support newer pkg #1779
  • CloudEvents SDK changes to surface HTTP status codes #177
  • event_processing_latencies not working #1753
  • Tag broker metrics by type #1423
  • Use responseCode tag in Broker ingress/filter metrics instead of result #1784
  • Add broker label selector in Triggers #1806
  • Serialization/Deserialization of configMaps in pkg #656
  • Source stats_reporter in pkg #648

• Data plane metrics

  • Broker (event_count)
  • Trigger (event_count, event_processing_latencies, event_dispatch_latencies)
  • Importers (event_count)
    • ApiServerSource #1786, #1864, #1872
    • CronJobSource #1790
    • KafkaSource
    • GitHubSource
    • AWSSqsSource
    • PullSubscription #250
    • Storage #250
  • Channels (event_count)
    • InMemoryChannel (issue Metrics: Data Plane metrics for InMemoryChannel #1818) (needs big refactor --> done in #1933)
    • Channel (from knative-gcp)
    • Kafka (needs refactor --> migrate to CE sdk)
    • NATSS (needs refactor --> migrate to CE sdk)

• Graphana Dashboards

  • Broker/ Trigger (issue Metrics: Graphana Dashboards for Broker/Trigger #1821) #1966
  • Importers (issue Metrics: Graphana Dashboards for Source metrics #1822) #1966
  • Channels (issue Metrics: Graphana Dashboards for Channels metrics #1823)

• E2E tests for metrics

  • Infrastructure setup (prometheus) (issue Metrics: E2E Tests Setup for Prometheus #1920)
  • Broker (issue Metrics: E2E Test for Broker #1819)
  • Trigger (issue Metrics: E2E Test for Triggers #1820)
  • Sources
    • ApiServerSource (issue Metrics: E2E Test for ApiServerSource #1815)
    • CronJobSource (issue Metrics: E2E Test for CronJobSource #1816)
    • KafkaSource
    • GitHubSource
    • AWSSqsSource
    • PubSub (from knative-gcp, issue Metrics: E2E test for PubSub google/knative-gcp#289)
    • Storage (from knative-gcp, issue Metrics: E2E test for Storage google/knative-gcp#288)
  • Channels
    • InMemoryChannel (issue Metrics: E2E Test for InMemoryChannel #1817)
    • Channel (from knative-gcp, issue E2E test for Channel google/knative-gcp#287)
    • Kafka
    • NATSS

• Documentation

  • Metrics available (Broker/Trigger, Core Sources) #1891
  • Accessing metrics #1891
    • prometheus
    • stackdriver
  • Instrumentation
    • Source
    • Channel
  • Doc in knative/docs issue #1792

Persona:
Which persona is this feature for?
Event consumer, Event producer, and Operator.

Exit Criteria
Users should be able to see metrics in the grafana/prometheus UI and stackdriver. Documentation should be available in knative.dev, indicating what metrics are defined and how to add new ones, as well as how to configure the metrics infrastructure. E2E tests should be in place as well.

Additional context (optional)
Some notes on the eventing-observability subgroup https://docs.google.com/document/d/178iKE5_qtnXQfYcXp-NeKWKguoM_GLKuz3E2w96F0gI/edit?usp=sharing

[nachocano]

/assign @nachocano
/assign @sayanh

[knative-prow-robot]

@nachocano: GitHub didn't allow me to assign the following users: sayanh.

Note that only knative members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

/assign @nachocano
/assign @sayanh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nachocano]

fyi @sayanh
apparently we cannot have two persons assigned.

[nachocano]

fyi @Harwayne

[akashrv]

/milestone v0.9.0

[akashrv]

The way we defined the metrics for Broker, could we define for Importers and add it to the data plane contract of the importer (@n3wscott @nachocano ) ?

[nachocano]

The way we defined the metrics for Broker, could we define for Importers and add it to the data plane contract of the importer (@n3wscott @nachocano ) ?

I think we can do that.

[sayanh]

Infrastructure setup and scraping of metrics (e.g., prometheus operator, or via configMaps, as with serving)

Working on a first prototype with configmaps. After discussion with @nachocano, few more concrete stuff came up:

• Have a working Prometheus scrape config for eventing pods
• Have a reconciler dashboard
• Create a common repo for dashboards ( as patching from serving is not ideal)

[nachocano]

• Create a common repo for dashboards ( as patching from serving is not ideal)

I think this one refers to something like have a common repo for cross-observability things. There is one called knative-observability, but not sure about the work being done there.
I think for 0.9 we'll have to patch, but we can start the discussions on what is the right approach.

Thanks for working on this @sayanh

[akashrv]

The way we defined the metrics for Broker, could we define for Importers and add it to the data plane contract of the importer (@n3wscott @nachocano ) ?

I think we can do that.

Then lets add it to the list of tasks (checkboxes) and follow-up with @n3wscott to add to importer spec.

[nachocano]

Current documentation: https://github.com/knative/eventing/blob/master/docs/metrics.md