Broker internal design improvements

[mikehelmick]

This is broken out from #1381

Problem

The current implementation of the broker uses a single internal channel for delivery and another internal channel for replies. This is not ideal for a scalable workload with dozens or hundreds of triggers and thousands of events per second flowing through the broker.

Persona:

Event Consumer, Systems Integrator

Exit Criteria

The broker meets an agreed upon throughput goal (to be determined)

Time Estimate (optional):
10-15d

Implementation could be complicated as it involves channel management, potentially new configuration points.

Additional context (optional)

In order to meet the proposed design of #1381 - these additional design points are needed.

The selector option (As ObjectReference) will be added to the TriggerFilter portion of the Trigger spec. The selector relates back to the metadata.uid of the object being selected against.

If a trigger contains a selector filter, this filter will be applied with exact match before delivery. The selector is an ObjectReference and will cause the trigger to have an exact match filter on the UID of the referenced object in the sourceid attribute. (See #1554 )

When using the selector filter, we can update the status of the trigger with a failed condition if the expected importer / event source is not present and ready.

[lionelvillard]

From a logical point of view, is it fair to say that a broker is a Choice/Routing (see #1525 and #1529) object, where the Filter object handles at least sourceid? Also see #1436 to give you an idea on how routing can be optimized.

[aslom]

Mike, did you consider creating in-memory channels implementation that is based on knative serving? That should make brokers scalable? If we are concerned about cold start we can keep min-instances=1?

[mikehelmick]

From a logical point of view, is it fair to say that a broker is a Choice/Routing (see #1525 and #1529) object, where the Filter object handles at least sourceid? Also see #1436 to give you an idea on how routing can be optimized.

@lionelvillard - I don't think that broker + trigger +sourceid covers the choice CRD as defined in #1525 - in particular it doesn't provide the mutually exclusive branches condition. If you set up your triggers correctly, you would get that behavior, but we haven't introduce anything that allows for single subscriber only.

I think it would be interesting to work through the user experience of choice being backed by the broker+trigger model.

Mike, did you consider creating in-memory channels implementation that is based on knative serving? That should make brokers scalable? If we are concerned about cold start we can keep min-instances=1?

@aslom - I'm more concerned about the current design using a single channel by default. Depending on the actual channel used this could lead to weird blocking/starvation issues. In-memory channels, or channels that are auto scaled are a potential solution, but I also want us to consider solutions that give better durability guarantees.

[lionelvillard]

It's funny how some people see eventing being built on top of messaging and some other people see the exact opposite.

[n3wscott]

/close
We added conformance tests and additional implementations of Broker, the broker inside of knative eventing is MT and should be used as a reference, but likely there are more robust and production ready brokers available to install.