Allow eventing to make use of knative serving

[yolocs]

Problem
In eventing there are places where we'd like to make use of knative serving capabilities, e.g. from #2154, there are attempts to run broker ingress/filter as knative services so that they can autoscale.
This doesn't imply a runtime dependency on serving (as we can make it configurable for whether to use kn service), but it will likely introduce code dependency on serving. And there are concerns about introducing serving as a code dependency.

I want to start a discussion on what options we have to allow eventing make use of serving. Some options I have heard so far:

1. Just introduce serving as a code dependency. As @grantr pointed out, we have automated process updating pkg in eventing/serving, at least pkg is more manageable now.
2. As @slinkydeveloper pointed out, we can update to go model for better transitive dependency management.
3. Duck? @lionelvillard could you elaborate this?

Persona:
Maintainer mostly?

Exit Criteria
Eventing being able to make use serving.

[lionelvillard]

I was just referring to the fact that both serving and eventing have a dependency on pkg/apis including pkg duck types.

AFAICT, we just need to introduce a code dependency on the apis (and generated client/injection), which is much more manageable compare to having a dependency on internal code.

[yolocs]

AFAICT, we just need to introduce a code dependency on the apis (and generated client/injection), which is much more manageable compare to having a dependency on internal code.

@grantr I remember you mentioned that it (separating serving api out) was considered before but somehow got dropped. Can you still find any history discussion, or who from the serving team could help?

[bbrowning]

Does Eventing just want request-based autoscaling for Eventing pods? Or does something in Eventing actually want the Service/Revision/Configuration APIs of Serving?

[yolocs]

Does Eventing just want request-based autoscaling for Eventing pods? Or does something in Eventing actually want the Service/Revision/Configuration APIs of Serving?

Currently I'm looking at the autoscaling capability. But I'm not sure if other folks have different potential usage of serving.

[lionelvillard]

I think we only care about request-based autoscaling.

[yolocs]

@vaikas @mattmoor for suggestions.

[lionelvillard]

Ideally revision is the only thing we need but AFAIK it does not expose a service. I think we should probably consume Knative Serving the way it's supposed to be consumed, meaning starting with Knative Service.

[mattmoor]

I think that so long as we have a clean out-of-box install story for eventing that doesn't require serving, then having a dependency is fine. eventing-contrib depends on serving, and that's fine.

Whether we choose to separate things by repository or by release artifact is worth discussing, but it should certainly be possible for folks to take advantage of serving if it's there.

My $0.02.

[mattmoor]

cc @vaikas

[yolocs]

Another idea I have is to have some kind of dynamic injection, like what we have for the default broker channel. For example, in the config we can specify a default service provider.

apiVersion: v1
kind: ConfigMap
metadata:
  name: default-svc-webhook
  namespace: knative-eventing
data:
  # By default, we always use k8s service and deployment
  default-svc-override: |
    clusterDefault:
      apiVersion: serving.knative.dev/v1
      kind: Service
      spec:
        template:
          spec: {{.podSpec}}
    namespaceDefaults:
      ...

And eventing control plane just generate the pod spec and fill into the {{.podSpec}}.

[vaikas]

As discussed in the meeting, my thinking is that we should not take a hard dependency on serving as in that being the only thing that works with Eventing. Better would be to find a way to make it pluggable, since some pieces might require things like queue based scaling which Serving currently has no answer to. There's for example some work going on with KEDA project to understand where we can learn from each other and not reimplement things, so making sure that there's way to plug in alternate scale strategies is the key here. Currently everything is push based and request driven scaling that Serving gives is a natural fit, but it's not the only thing we have to worry about.

[yolocs]

I have put more thinking on this topic.

Better would be to find a way to make it pluggable

My assumption is that "pluggable" here means having pluggable autoscaling mechanism for eventing workloads, e.g. use ksvc for some while KEDA for some others.

My conclusion: there is no natural way to make it pluggable. To make things pluggable, we usually need to find a common interface first so that different autoscaling mechanism can implement the interface. But in this case I found none. Initially I thought the interface could be as simple as a podTemplate, and the autoscaling mechanism will scale it automatically - that can work for sure with ksvc. Then I realized that most KEDA autoscaling requires some external metrics and different queue requires different external metrics. To demonstrate the problem:

A pubsub channel requires a receiver (a deployment) to pull messages from the topic. In order to make use of KEDA, a ScaledObject is needed with multiple pubsub specific parameters (e.g. topic name, key, etc.) to access the stackdriver metric for the queue size. And only the channel controller knows these parameters. If the controller is going to create a KEDA ScaledObject, then eventing now has a dependency on KEDA (because the controller needs to use the KEDA API) - assuming this is as bad as dependency on ksvc. If we're going to provide an interface, then the controller couldn't possibly know the implementation is KEDA and thus not know what parameters to provide.

To move forward, I think we probably should lock down to several well-known ways to run autoscalable workloads such as ksvc, KEDA. For the current repo, there is no pull-based workload. So taking a code dependency on serving is sufficient.

[aslom]

There should be some way to detect if KEDA (knative serving) is installed? If it is not installed then controller would log error and not use KEDA (knative serving) and switch to fallback without scaling?