Make it easier to consume events directly from specific sources

[lindydonna]

I’m a PM at Google, working on the CLI and UI experience for GCP products built on Knative Eventing. The current system has a lot of granular objects that can be combined in very flexible ways. Unfortunately, this makes it quite difficult to create a user experience for some common cases. I think these considerations would apply to most vendors who are building products on top of Knative, so I’d like to get feedback from both the community and especially other vendors doing their own implementations.

I think there are two key developer scenarios that our product needs to support:

1. [FaaS scenario] (Similar to: AWS Lambda and Google Cloud Functions) As a developer, I want to deliver from a managed cloud service, to a target Service (or Addressable). Managed cloud services are services like AWS S3, Google Cloud Storage, Pub/Sub, etc.
2. [Event-Driven scenario] (Similar to: Kafka, Google Pub/Sub, AWS SNS, etc)
   1. As an event-producer developer, I want to define custom on-cluster events and send them to a central location.
   2. As an event-consumer developer, I want to consume these custom on-cluster events, without directly specifying the sender of the event (i.e., the Service generating the event).

The Event-Driven scenario is very easy to express in Knative Eventing. The FaaS scenario is easy as well, but only if you just use Importer directly.

NOTE: Here I am assuming there is value in having a single model for both scenarios (see @sixolet's comment below for options along with the pros and cons). This may not be a valid assumption.

---

A good UI and CLI for the FaaS experience would involve customers configuring an upstream sender and target Service in one logical step. Since we don’t currently have an object that does exactly this, I’ll use the placeholder name Storm.

Note: Trigger could be modified to have the properties we need in Storm. It just doesn’t have them today.

Storm could be implemented with a number of objects, but the key is that ideally, FaaS customers just think about Storm. If they list all Storms in a namespace, they should have a complete view of what is happening.

Unfortunately, the current model has the following issues:

1. To create a Storm, you’d have to create an EventSource and a Trigger. Creation works, but after that there’s no clear way to list all Storms in the system. (Trigger could have all the fields that are currently in the EventSource, but then the developer would have to keep both in sync.)
2. Suppose Alice instantiates an EventSource for PubSub/awesome-topic (exposing event type AwesomeEvent) and a Trigger for AwesomeEvent -> Service1. Bob, not knowing the state of the cluster, also instantiates PubSub/awesome-topic and creates a Trigger for AwesomeEvent -> Service2. Now, Service1 and Service2 will get duplicate events.

Problem 1 makes it very hard to create a UI or CLI that lists all Storms in the system.

Problem 2 could even cause semantic issues if both EventSources publicize the same event type, without realizing that there are other subscribers.

[duglin]

Great topic. First a question, on your 2, why would Service1 and Service2 get dup events? Wouldn't Service1 only see events driven by the first EventSource+Trigger?

I haven't had a chance to fully think through how this would look, but in the times when I did ponder it, I would think about what I would want to say as an end using trying to hook these things up. And it usually comes down to something like:

• I want x,y,z events from event producer A sent to my KnService C
• e.g. I want the "push" event from github repo myApp sent to my "testPR" KnService

And in order to express that I would probably create a KnEventing equivalent to the KnService. Meaning, some single entity (a Storm) that allowed me to express everything and under the covers it split that data up into smaller (more targeted) resources to get the job done. The way a KnService is used to create revisions, routes, etc...

So, in the above example, it might do this:

• create a github event source, and set the sink to the "testPR" KnService, or
• create a github event source, set the sink to a broker, and create a trigger for that broker that has the "testPR" as the target/sink, or
• based on what's already in the system, it might just need to create a new trigger due to an event source and broker already existing

Whichever the system chooses is an impl detail, but could be influence by optional properties on Storm. For example, an option that indicates the QoS which would control whether things are persisted or not. Or, another one that indicates whether this event is a candidate to be shared with other parts of the system (meaning goes thru a broker that allows for multiple triggers) or just one.

The net of it to me is that I think the user should express what the want as an end result, and then magic happen via internal resources that they shouldn't need to think about.

[sixolet]

Two unrelated and contradictory views of this problem, neither of which introduces a new object:

1. "Storm" is and has always been the Event Importer. Event importers can send to any target, including directly to a Service. Event Importers should include in their spec whatever users tend to need to subscribe to the right events for their problem domain, including whatever filtering they need. Brokers and triggers are more for your in-cluster event mesh. An importer might say something like "give me Slack events on channel foo matching the regex (eggs)+ spam".

Properties:

• Importers can be simple
• Triggers can be simple
• Importers get complicated when you start trying to share work between requests for the same off-cluster events.
• You do similar tasks (asking for a subset of on-cluster events (create a Trigger), asking for a subset of off-cluster events (create an Importer)) in wildly different ways.
• You have to understand which category of thing you are doing before doing it. If you try to use the wrong model, ex, you end up firehosing your entire slack server into your broker, just cause you want eggs and spam from one channel.
• Two loosely coupled models for eventing

2. "Storm" is Trigger. An Importer represents the potential to get these events onto the cluster and into the broker. Event Importers should be smart enough to look at their relevant triggers and adjust the events they ask for from off-cluster to only pull in the events that someone's asking for by making a Trigger; the Trigger functions as "here is a request for these events". Creating the Trigger tells the Importer to realize that its potential in a particular way. In this world creating the Importer is a setup task, but you only have to do it once per credential-access-realm. Ex, once per github org, once per slack server, once per GCP project, etc. An Importer might say something like "connect to the Slack server chat.foo.com"; a Trigger might say something like "give me Slack events on channel foo matching the regex (eggs)+ spam".

Properties:

• Importers need to look at Triggers if they want to be efficient. Your average importer that does all this properly is a little less simple.
• Most users only ever need to interact with Triggers.
• You do all eventing tasks that aren't "configure a connection to a service" by creating a Trigger.
• An overly-broad trigger ("all events in the broker") may not work as expected.
• One model for eventing.

[deissnerk]

@duglin I like the overall vision. A while ago I wrote this document. Terminology is still kind of messy, but the overall idea is to define Broker as something that takes a Trigger (My service wants to get events of type a from source b) as input and configures so-called Routers and Sources (now Importers) accordingly.

[mikehelmick]

/assign mikehelmick

[lindydonna]

@duglin said:

Great topic. First a question, on your 2, why would Service1 and Service2 get dup events? Wouldn't Service1 only see events driven by the first EventSource+Trigger?

We answered this in the WG meeting this week, but so that the discussion is recorded: note that Trigger is always used with Broker. So, if a customer does EventSource -> Trigger it always goes to Broker. If the EventSource/Importer is connected directly to the sink, as mentioned by @sixolet, then the duplicate problem is sidestepped entirely.

I'm going to create a doc so that we can collaborate on the ideas easily, but will not get to that for a few more days.

[grantr]

I tend to generalize the two scenarios thusly:

1. I wish to express interest by specifying the attributes of the event source.
2. I wish to express interest by specifying the attributes of the event.

In some cases, the specification of the event source attributes and the event attributes are identical. For example:

org: knative
repo: eventing
type: com.github.pull.create

Assuming GitHub credentials are identified by some other mechanism, this is sufficient to specify the attributes of the source connection and the attributes of the event to filter.

Other cases may not allow this overlap, mainly when the upstream source supports a filtering language that is not supported downstream. For example:

type: mysql.row.update
query: select * from myapp.users where login_time > date_sub(now(), interval 60 seconds)

In this case the downstream filter does not support querying the event stream via SQL, and cannot reliably derive a supported filter from the query. The query is an event source attribute, not an event attribute.

[vaikas]

For completeness sake, can we jot down why FaaS case is not answered by wiring a Source.directly. I.E. not using Broker/Trigger.

[duglin]

+1 to what I think is the broader point @vaikas-google is making.... we shouldn't assume people are using (or want to use) all of the features of KnEventing. Meaning, they may not want/need brokers, channels, triggers, etc... in all scenarios - they should be used only when they need to be used.

And this may not be possible, or too much magic is involved, but if we can get to the point where a user expresses their interest in what events they want (along with the minimal data they think we should need to achieve that goal for them), we should them be able to put together the wiring under the covers for them. And part of that work means knowing when (or not) to use the more advanced features. And the infrastructure used should change as the user's requirements change (scale up/down in complexity).

[lindydonna]

For completeness sake, can we jot down why FaaS case is not answered by wiring a Source.directly. I.E. not using Broker/Trigger.

I'll answer here, since I haven't yet started a doc.

Some folks on the Google side feel that it is best to have a single user model across both scenarios. So, per @sixolet's comment above, her option # 1 would present a different model, depending on which case you are in.

As she put it:

You do similar tasks (asking for a subset of on-cluster events (create a Trigger), asking for a subset of off-cluster events (create an Importer)) in wildly different ways.

My personal view is that these two scenarios do not necessarily need to have the same objects, but obviously it would be preferable there were just one model (all else being equal).

If we decide that there should be one set of objects, we need a design for how Importer -> Broker -> Trigger can work in a reasonable way.

Great question, and I'll clarify in the original issue.

[mikehelmick]

I am of the opinion that there is value in being able to have a consistent trigger writing experience, without regard to where the events are sourced from (off cluster producer talking directly to broker, off producer through importer, on cluster producers).

There are great building blocks and we shouldn't prevent people from assembling them in whichever way they see fit. However, for building a product out of the components of eventing, there is value in providing consistent, high level abstractions to developers and leaving lower level config to operators.

I think it would be difficult to build a UX over a collection of related, but different abstractions. Since event source is not a type in itself, it makes it extremely difficult to list all of the importers that might be pointing to a function, or event to list the importers one might have. Assuming you could list all importers, you need to cross that with all triggers and all subscriptions.

This is where I think there is value in using trigger as the common abstraction for understanding how events are intended to be delivered in the system. There's a single thing to list, I can envision building a CLI or UI out of it with trigger as an entry point. From there, we can provide deeper analysis of how the events are actually getting there through the broker.

As I said before - people can set up events to flow another way, but for people choosing to use broker + trigger, I would like to open up all scenarios to them.

[lindydonna]

I've started a doc in the shared drive so we can collaborate more easily. See High-level eventing scenarios (June 2019)

[mikehelmick]

I wanted to give an update here and I would be happy to discuss this at the next WG meeting.

We are currently exploring an extension to Trigger to support the scenarios described for the storm object and the scenarios in the high-level eventing scenarios doc.

What we are looking at is

• Extending the registry to [optionally] describe how to create an importer for a given source type
• Extend trigger so that when creating a trigger, it is possible to instantiate an importer that will feed events to that trigger, again optional.
• To enable the previous one, we'll make Trigger implement Addressable contract, the URL will actually point to the broker. something like http://default-broker.default.svc.local/triggers/trigger-uid

This makes supporting the FaaS scenario straight forward, create a trigger for a specific producer. When the trigger is deleted, the importer is deleted.

The Event-Driven scenarios can be solved by adding events to the event registry that don't specific how to create an importer.

This allows a platform provider to have a pre-configured registry (or even an immutable registry).

Developers can interact with the system simply through triggers. This makes listing all triggers or listing triggers that go to a specific service easy to do. I believe this makes CLI/UI scenarios easier to develop.

[lionelvillard]

when you say:

an importer that will feed events to that trigger,

through the broker or directly? I think you mean through the broker, is that correct?