Add content-based event filtering

[lionelvillard]

Problem
Event subscribers want to receive only events they are interested in. They don't want to be overloaded with unwanted events. They also don't know what to do when they don't want an event (return an error?)

Persona:
Event consumer

Exit Criteria
An event subscriber does not receive events when their content matches a certain condition and vice versa.

Time Estimate (optional):
1-2 days?

Additional context (optional)
One possible way to solve this is to implement a new kind of channel, called Filter.
Another way is to extend the in-memory channel with filtering
Either way should work with Pipeline

[Harwayne]

Can you give a few examples of the kind of filtering you are interested in?

[lionelvillard]

Here some examples using JsonPath + comparison operators:

$.foo == 1
'annotation' in $.annotations
$.result

[Harwayne]

Here some examples using JsonPath + comparison operators:

$.foo == 1
'annotation' in $.annotations
$.result

To be sure I understand, these are checks against the data attribute of the Cloud Event, which we (implicitly?) assume is JSON. Is taht correct?

[lionelvillard]

That's correct. JsonPath could be used when the data is JSON. Somewhere we will have to specify want kind of data the event subscriber is expected. Maybe there should be one Filter per content type (JsonFilter, XmlFilter, ...)

[srvaroa]

Can you give a few examples of the kind of filtering you are interested in?

We maintain an event bus that collects events from multiple systems (e.g. GitHub, Travis, Spinnaker, Kubernetes, and a few others). A number of tools (mostly for automation, and analytics) listen on these events, filter for a particular type, and act on them. A typical case are GitHub bots, which is a good use case for this feature.

In our infra, a bot would be listening GitHub stream, but is interested only in a particular type. One bot that automagically rebases PRs when they become out of date with master, so it only cares about pushes to master. Another bot labels PRs based on configuration stored in the repo, so it only cares about PR events that represent a create / update, but not a close. And so on. Typically this filtering involves looking at the GitHub event payload and matching on some values of $path.

We are currently doing this filtering client-side (all events get to every bot, and a simple SDK provides the filtering functionality so the bot writer doesn't really have to do the work). At this point we have enough volume (our GH has ~1.5k developers), and we'd also prefer to simplify that SDK (among other reasons so the SDK doesn't restrict language choices.) We implemented one PoC based on Openfaas where:

• One bot ("Router") processes all events from the stream (let's assume only the GitHub stream)
• A engineer writes a bot A (in our case, for OpenFaas). In the repo, they place a file that declares a matching rule for JSON payloads (we'd probably go for something jq friendly) that means "this bot wants to receive events that match on $filter".
• When bot A is merged, we ensure that:
  • The Bot is installed in $runtime (e.g. OpenFaas in our case)
  • The Router registers the new mapping $filter -> bot, so that any event that matches on $filter is routed to Bot's endpoint.
• Profit.

Content-event based filtering would match perfectly well this case, offering the Router component, and a mechanism to declare filters. I would stress that offering the ability to perform rich filtering (e.g. support a jq query, JsonPath, whatever else) would be ideal as it gives sufficient flexibility to handle event streams that can be quite heterogeneous (think GH events).

Does that make sense?

[lionelvillard]

@srvaroa thanks for sharing this use case. Very interesting!

I would stress that offering the ability to perform rich filtering (e.g. support a jq query, JsonPath, whatever else) would be ideal as it gives sufficient flexibility to handle event streams that can be quite heterogeneous (think GH events).

AFAIK, jq is much more expressive compare to JsonPath. Do you have filtering expressions that can't be expressed in JsonPath?

In any case, we want to be flexible in the language used to filter events, either to support different content-type (eg XPath for XML) or various levels of filtering complexity and/or expertise (eg. jq vs jsonpath).

[srvaroa]

I think JsonPath will go a long way, although as you say remaining flexible is a big pro. In my mind the idea would be to support some form of configuration such as:

config:
  matcher: jsonpath
  routes:
    - expression: $SOME_JSONPATH_EXPR
      target: $URI
    - expression:  $SOME_OTHER_JSONPATH_EXPR
      target: $URI

The matcher node would apply to all events passing through this filter, and the Router component would rely on a suitable implementation, applying the expressions in the language that the user is more comfortable with. There might be cases where a single event stream might have mixed content (e.g. json / xpath) although this feels overcomplicated.

By the way, is there a relationship between this issue and #930 (cc @nachocano)?

[lionelvillard]

@vaikas-google What do you think about this (in the context of pipelining)?

[vaikas]

this seems like it would be useful general purpose function. You pass the filter expression as config into it and it could be the first stage in the pipeline. Then if we get some experience in there could see on generalizing that and make it easier to streamline the user experience.