Broker-to-broker event routing across namespaces

[devguyio]

Problem
Using the Broker objects, it's not straight forward how to deliver a set of events across namespaces. Specifically when you have a many-to-many relationship between Event Sources and Brokers. Using existing primitives, one would need to funnel in all events through channels/subscriptions across namespaces to the namespace broker, where they will be finally filtered by the triggers, which is very inefficient in resource utilization given the potential that thousands of events might get unnecessarily routed and persisted without real subscribers.

Persona:
Which persona is this feature for?

• Event Consumers (efficient events routing)
• Event Producers
• Systems integrators

Exit Criteria
Given brokers A and B are in two different namespaces, Events can be routed from A to B given that there are Trigger object for Broker B with matching filters.

Time Estimate (optional):
7 days

Additional context (optional)
In Kyma, there exists many event sources in a single, admin controller namespace due to admin provided credentials etc, and there are many interested subscribers (functions and microservices) across multiple namespaces. To solve this, we have an Event Bus with a RESTful API and that leverages Knative Channel objects for fanning out events to subscribers across multiple namespaces using URIs if and only if access control per namespace allows it (we use an EventActivation CRD) and there're existing subscribers to this event type.

Trying to solve this using existing primitives, we had attempts such as the following

which ends up in doing the filtering at a very late stage.

[devguyio]

This is another attempt (which is not very accurate in details) to solve it using existing primitives to achieve a closer to source filtering using Trigger objects. Problem here is that there should be a sort of controller or manager that creates a Trigger per event type per source whenever one is needed, as well as the original Trigger object which the user creates for his serverless function.

[sayanh]

The double trigger approach is good and would be one way to solve the problem. I have a concern based on the current implementation of the broker:

• Currently, I'm skeptical about using a broker with an event source because a broker comes with 2 channels i.e. ingress and filter. Filter channel stores all the events irrespective of a trigger. This implies cost(storage, CPU, network, etc) incurred on the persistence layer. So in this approach, an event(with no trigger) will be at least be stored in 1 channel and if it has a trigger, then it will be stored in 4 channels. If I speak generically, the event will be stored in 2+2 times N(N is the number of brokers). This is also the problem with the channel approach mentioned in the issue description.

[sayanh]

I have another approach in my mind where I'm introducing a new component called BrokerManager(BM). This lies between an Event Source and other brokers. It maintains a map(in-memory or redis sort of key-value store) that stores filter criteria mapped to a broker URL. The functions of a BM are as follows:
BM watches for all the brokers in the cluster and reconciles on the Broker custom resource:

• If a broker is found, it searches for triggers using the brokers in that namespace.
  • If a trigger is found, it creates an entry in the map containing filter criteria in the trigger to the broker URL.
  • If a trigger is deleted, the entry in the map pertaining to this trigger is deleted.
  • If a broker is deleted, the entries in the map with the URL of the same broker get deleted.

When an event reaches BM from an event source, the BM fans out the event to all the brokers matching the filter criteria based on the in-memory map.

• Does BM queue all the events from EventSource and gives 2xx back to Event Source irrespective of the event reaching the concerned brokers?
  It may queue in internal memory. Although the Event Source gets 2xx after publishing to BM, there is no guarantee that a valid broker ( then, in turn, a subscriber) will get this message. What if the BM dies before sending the events to Brokers? The messages will be lost.
  Then it should be a synchronized request(event) from EventSouce to BM then to all Brokers and BM should send 5xx if any one or more brokers send a non-2xx response. Then it means the publisher suffers due to the requests reaching brokers. There can be a retry policy or a dead letter queue in BM.

• Can we scale such a BM? What happens to the ordering of the events?
  Yes we can scale, each request reaches only one instance of the BM. Ordering is not guaranteed if there are multiple instances of BM.
  

Another approach would be bottom-up as here #934.

[n3wscott]

Try to use a Parallel component instead of building a new BrokerManager. The filter function could even live in the namespace of the BrokerN using uris of those functions and the broker ingress uri to link it all together.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[matzew]

/reopen

[knative-prow-robot]

@matzew: Reopened this issue.

Details

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.