Send observability related configmaps to destination namespaces

[sayanh]

Problem
A short explanation of the problem, including relevant restrictions.

Right now, the config-observability and config-tracing configmaps are watched by different broker filters/ingress deployments across namespaces. E.g. a broker is created in namespace A then the corresponding broker filters/ingresses watch the observability configmaps in knative-eventing namespace while living in namespace A.

Pros:

• Dynamic changes in configmaps are picked by the deployments instantaneously.

Cons:

• Need strong RBAC policies for cross namespace access, which is not ideal.

The community and @n3wscott favored the other way where a copy of the configmaps are base64 encoded and pushed down to the final deployments. Here is a sample implementation.

Pros:

• No cross namespace RBAC policies required.

Cons:

• Need to kill the old deployments to reflect the dynamic changes in configmaps.

Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.
Following components should be using the 2nd approach

• broker filter
• broker ingress
• apiserver source
• container source
  ...

Time Estimate (optional):
How many developer-days do you think this may take to resolve?

Additional context (optional)
Add any other context about the feature request here.

[grantr]

A general solution to this problem could be a controller that watches specific configmaps and replicates them to other namespaces. Then we don't need to setup unsual cross-namespace RBAC, and pods can get new config without restarting.

[lionelvillard]

I like the idea of having a generic controller for replicating configmaps. In the future, it will then be easy to extend this controller for instance to support namespace-scoped configuration by merging the knative-eventing configmaps with the one provided in the namespace.

[Harwayne]

/project Observability To do
/milestone v0.10.0

[capri-xiyue]

/assign @capri-xiyue

[grantr]

I've written a design doc describing a general ConfigMap propagation solution that solves this problem for Knative Eventing and other projects with this need. It suggests creating a namespaced CRD ConfigMapPropagation that signals the intent to copy ConfigMaps from one namespace to another:

kind: ConfigMapPropagation
metadata:
  name: knative-eventing
  namespace: default
spec:
  originalNamespace: knative-eventing
  selector:
    knative.dev/config-category: eventing

More details in https://docs.google.com/document/d/19kubGE8RQWlPFxz38G7hAtcqfihHosXCr6zXnWWOfOI/edit.

There are certainly other solutions, but this one seems cleanest to me (so far). Please comment either here or in the doc.