Pull NetworkConfig logic out of Revision controller

[mattmoor]

I'd rather have the controllers' use of informers focused on reconciliation. In the case of the NetworkConfig, I feel like we could do this by:

1. Exposing SetNetworkConfig on the revision controller
2. In cmd/controller/main.go doing something like:

    configMapInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
        UpdateFunc: func(_, obj interface{}) {
            configMap := obj.(*corev1.ConfigMap)
            if isNetworkConfig(configMap) {
                revisionController.SetNetworkConfig(NewNetworkConfigFromConfigMap(configMap))
                return
            }
            // handle other configmaps being routed to other controllers
        })

[mattmoor]

DomainConfig uses a similar pattern.

I have some thoughts on what this might look like, so I may take a look at it this week (if @mdemirhan has not already started on it).

/unassign @mdemirhan
/assign @mattmoor

[mattmoor]

cc @josephburnett

It sounds (even from the docs) like ConfigMap updates through volumes can be very laggy:

As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period + ttl of ConfigMaps cache in kubelet.

tl;dr I think I'd propose a simple informer-based configuration model.

I think we'd have something like pkg/configurationmanager (I'll resist the urge to name this pkg/conformer), which exposes:

package configurationmanager

type Watcher func(*corev1.ConfigMap)

type Interface interface {
   Watch(name string, cb Watcher)
}

func New(informer) Interface {...}

Then Controller's would watch the ConfigMaps that they care about:

package route

func New(..., cm configurationmanager.Interface) controller.Interface {
   ...
   // Parse the ConfigMap into DomainConfig, see what's changed and do what setDomainConfig does today.
   cm.Watch("domain-config", c.receiveDomainConfig)
}

package revision

func New(..., cm configurationmanager.Interface) controller.Interface {
   ...
   // Parse the ConfigMap into DomainConfig, see what's changed and do what setDomainConfig does today.
   cm.Watch("network-config", c.receiveNetworkConfig)
}

This is in essence a generalization of what exists through informers directly today, but hides some of that complexity behind a higher-level interface (that we can fake for testing).

This model should avoid the lag of depending on the ConfigMap volumes, and also enable us to be aware of when we've picked up updates to trigger additional work. For example, when we pick up new sidecar images, we may want to trigger a global reconciliation to roll it out (assuming a world where we reconcile the underlying Deployment).

[vaikas]

Yep, this makes sense to me.

[mattmoor]

I put together the linked PR that implements what's outlined above for {Network,Domain}Config.

Ideally we'd have a single method for handling our Controller configs (including config-autoscaler, config-observability and config-logging), so please consider this through that lens.