GitHub demo events should be usable with any action (currently hard-coded to ElaRoutes)

[inlined]

Expected Behavior

GitHub events should be useable with any aciton

Actual Behavior

GitHub event source directly wires a GitHub webhook to an externally visible Ela route.

Steps to Reproduce the Problem

1. Run GitHub sample

Additional Info

To correctly fix this, the GitHub proxy will be responsible for exposing a "ReceiveAdapter" that will act as the gateway between GitHub and the event system. It should reformat webhook responses as Events and send them to the delivery service.

[markfisher]

Would the "ReceiveAdapter" be its own process?

Is there any documentation describing the "delivery service"?

[inlined]

@vaikas-google and I have a lot of brain-dumping to do. Now that KubeCon is over I can focus on getting some of the code hacked together for the demo cleaned up and checked in as well as putting design ideas into a docs folder where we can debate and approve proposals before implementing.

The general thought process was that we will need pluggable "proxy" services for receiving events from outside the cluster. To support software that may be deployed on-prem, the proxy services should be abstracted from the actual event sources for which they're proxying.

Ville broke the idea of a proxy into two key services:

• Subscriber: handles the translation of control plane commands. The BindController would delegate to the Subscriber to ensure that an event subscription was created & use the BindController's response to update the Bind's status.
• Receive Adapter: An optional data-plane component of the proxy. In a push-based system (e.g. GitHub webhooks), the Receive Adapter would verify AuthZ and make the raw payload conform to a CloudEvent format. In a pull based systems (e.g. Events from a database, logs, etc) the Receive adapter would be stateful; it would be responsible from reading some raw pipeline of data and constructing events.

We would need similar plugins if we want to support multiple kinds of event handlers (aka Actions in the CloudEvents vocab). This area has a reasonably high level of overlap with the "Case for Invokers". My goal is that Elafros interop is very smooth and we can work on ways to cut out extra hops, but we should have a general enough interface that Elafros isn't a hard-coded dependency (or worse, the only supported Action).

[markfisher]

It's still not clear to me if you're describing plugins that will be running in the BindController process, or are these proxies going to be running as separate pods?

[mbehrendt]

fwiw -- here you can find some information on how event sources are plugged into OpenWhisk:

https://github.com/apache/incubator-openwhisk/blob/master/docs/feeds.md

[markfisher]

Thanks @mbehrendt - I like the idea that a Feed is just another type of Action. We have a similar idea in riff, except that our "Function" maps to the OpenWhisk "Action" (so a lifecycle trigger would be the input to such a Function, but its output is a stream of events that other Functions can consume).

[deissnerk]

In kubeless there is a concept that seems to be very similar to Subscriber and Receive Adapter. So far the focus is on different communication protocols like kafka, http etc. For each of these we defined a CRD. A controller (subscriber?) configures and launches then triggers (receive adapters?) that pass the received message to the according function.
For handling events inside kubeless, I am currently thinking about an event routing component that gets events as input (via gRPC) and invokes the according functions. This way the transport of events is decoupled from the wiring of events and functions.

[evankanderson]

@mbehrendt -- thanks for that doc. In particular, breaking down the three different implementation options for data plane event delivery (what @inlined is calling a Receive Adapter) is very helpful.

I'm wondering if you consider the OpenWhisk implementation of the control plane lifecycle (firing lifecycle events at an OpenWhisk action) to be a necessity, or an optional implementation strategy? I can see a few different ways of layering this:

1. Bind Controller calls a well-defined set of Subscriber APIs
2. Each Subscriber Controller does a Watch for Bindings which apply to that data source
3. Bind Controller fires events (OpenWhisk-style) to a subscriber.

Personally, I'm somewhat partial to option 2; I think 2 could be converted to 3 fairly trivially with something like https://github.com/GoogleCloudPlatform/metacontroller

We should probably also have a glossary where we can standardize some of this terminology, i.e. #27

[markfisher]

+1 for option 2

IMO option 1 violates the guideline of avoiding imperative operations within controllers. I guess option 3 is borderline by that same measure.

[n3wscott]

out of date.
/close

[knative-prow-robot]

@n3wscott: Closing this issue.

Details

In response to this:

out of date.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.