Decide on the future of the Broker and Trigger

[devguyio]

Background
After working with the Channel/Subscription models and gathering feedback from users, there was an agreement that the event consumers for the MVP usecases want to consume events based on the CloudEvents attributes without a knowledge of how it gets routed or produced (i.e. which channel they need to create a subscription for etc) which lead to the existinig Broker and Trigger model ( See #815 , #814 , #862 )

Problem
After the initial basic implementation of the Broker and Trigger and collectiing feedback, it was clear that there are multiple shortcomings:

• Broker requires unnecessary event deliveries Broker requires unnecessary event deliveries #2288
• Brokers persist messages unnecessarily Brokers persist messages unnecessarily #2287
• Broker-to-broker event routing across namespaces Broker-to-broker event routing across namespaces #2050
• Broker needs to be a blackbox and not expose its internal design (e.g. Channels)

Which clearly shows the Broker and Trigger need to evolve into a better model and/or design.

Exit Critirea
There isi a clear set of improvments on the Broker and Trigger spec and/or implementation(s) which address the problems above and allows its wide adaption

[devguyio]

The absence of a definition of what a Broker is and does and how we evaluate if something is a Broker or not lead to @grantr working on Broker Conformance Spec

Issue #2306

[devguyio]

@n3wscott started a discussion around alternate Broker implementations with a proposal

Issue #2274

[mikehelmick]

Can we discuss this issue in the v1beta1 task force on Wednesday?

[matzew]

IMO another short coming is that the trigger/filter is incredibly limited and covers only a small amount of use cases. That said there is currently no good way to "plug in" some custom filtering implementation.

This makes the user fallback to "traditional" (or stable) channel/subscription model...

[aslom]

We should record the goals for broker such as that in-memory version is not for production use. And optimized implementations may not be using channels. Other changes?

[davyodom]

I've tried to compile an initial list of what should be there in order to use the current broker implementation in a production context for the project I am working on. Not all of these issues would apply to everyone and not all are completely mandatory, but hopefully this helps start a discussion. I think they are general enough that they shouldn't dictate a separate broker implementation though, unless the default broker will only be intended for POCs and development (like the in memory channel). Some of the points are repeated from above or have some related issues already (with both partial and complete solutions proposed). I would have created this as a google doc for easier collaboration but don't think I have access to create them in the knative community space there.

• High throughput use cases
  • Ingress and filter must be horizontally scalable (Using HPA to scale out default broker #2461, KSVC would also be a good solution for us)
  • Ingress and filter must support resource requests and limit specification (Support configuring resource requests and limits for broker-filter and broker-ingress #2321)
  • Ingress and filter must support additional annotations to configure aspects outside of knative eventing (e.g. istio annotations to define envoy CPU request overrides without changing full service mesh defaulting)
• Efficiency
  • Current broker can not consume any events without re-persisting them. (Brokers persist messages unnecessarily #2287)
    • E.g. events are in a kafka topic already, they must now be read by a source, then sent to the broker ingress, re-persisted in a different kafka topic, and finally read for each trigger. Note that this problem is the same for all source + channel approaches and is not unique for the broker.
  • Current broker must read all events for each trigger. High volume and low volume event streams mixed in a single broker or a large number of different event types in a single broker are problematic cases here. (Broker requires unnecessary event deliveries #2288)
    • To keep the broker as channel based, event partitioning strategies to use multiple channels could be introduced, possibly in a pluggable way. A list of some ways this could be used : Regex to support use cases where semantics of event types are well know, fixed count of types per channel to support generic use cases, and user specified grouping when characteristics such as event volume or requirements such as critical near real-time for an event type are known. (Regex or user specified would meet our initial requirements)
• Security
  • Different consumers need to be authorized independently. All events in a broker can not be made uniformly available to all subscribers. (Add trigger identity so that events from different triggers could be authenticated differently #2277)
• Functional
  • Cross namespace delivery
    • Users may wish to isolate subscription workloads by namespace without having to send all events to each of those namespaces. It should not be required to create a new broker and re-persist all events in that namespace in order to consume them there. Allowing a namespace in the object ref of a trigger would help here. Authorization policies could be applied around trigger creation and namespace usage externally.

[akashrv]

IMO another short coming is that the trigger/filter is incredibly limited and covers only a small amount of use cases. That said there is currently no good way to "plug in" some custom filtering implementation.

This makes the user fallback to "traditional" (or stable) channel/subscription model...

#2275

[grantr]

/assign

[davyodom]

The questions which seem to be floating around without clear consensus are:

1. Should a default broker exist in eventing-core?
2. Should this default broker have production-like qualities or should a different one be created in eventing-contrib?
3. Should the prod-like broker be channel based and expand on the current impl in eventing?
4. Is the full set of features/qualities in the comments above in scope?
5. Should alternate broker implementations be supported? (I think this already has a 'yes' consensus, but just listing here in case there is still some debate)

[akashrv]

My 2 cents:

The questions which seem to be floating around without clear consensus are:

1. Should a default broker exist in eventing-core?

[akashrv] Yes. Or at least part of default install (release yaml). A customer should not be forced into a vendor-written broker and should have a Knative OSS alternative.

2. Should this default broker have production-like qualities or should a different one be created in eventing-contrib?
3. Should the prod-like broker be channel based and expand on the current impl in eventing?

[akashrv] What does a "production-like broker mean" is debatable. Example certain delivery semantics could be transport specific, but we can always define some minimum spec for something to be called as Broker and #2306 tries to define this. So, IMO, if we have a Broker that is backed by channels (not saying the current implementation should be the one) and channels abstract out delivery semantics and guarantees, then we can have a default broker and an operator can decide to leverage channels to get different delivery semantics.

4. Is the full set of features/qualities in the comments above in scope?
5. Should alternate broker implementations be supported? (I think this already has a 'yes' consensus, but just listing here in case there is still some debate)

[akashrv:] Yes different vendors should be able to write their custom Broker that are conformant with with Broker spec. This could be due to variety of reasons such as leverage capabilities of vendor infra, or current channel based broker design doesn't work out.

[TristonianJones]

Regarding Triggers, Tekton implemented CEL-based triggers using the Knative filtering proposal from @grantr as a reference. FYI @wlynch