Be able to define a simple pipeline

[vaikas]

Problem
It would be nice to define a simple pipeline F1->F2->F3 in a user friendly way.
You can do this today manually by specifying the channels and subscriptions. It would be nice to just define a sequence of functions (and likely a ChannelProvisioner) and the Pipeline controller would then create the Channels and Subscriptions for them. This should also be able to be used in a Trigger object.

While you can also do this with the Broker Trigger model by specifying the Triggers, it does have couple of downsides (without additional work):

1. exposes intermediate events that can then be seen by other triggers
2. need to understand immediate event types so that you can create the necessary triggers

I believe this is a pretty low hanging fruit and would be helpful for users.

While this has been talked about in the past, I couldn't find an issue for it, if there is, let's dupe this, but I wanted an issue that we could hopefully get in the voting block so that we can see if there's any interest in this from the community.

Persona:
Which persona is this feature for?
Event Consumer

Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.
When a user can easily create a pipeline and plug it into the broker.

Time Estimate (optional):
How many developer-days do you think this may take to resolve?

Additional context (optional)
Add any other context about the feature request here.

[lionelvillard]

There is a proposal here: https://docs.google.com/document/d/1vG11SMg3rkvcwEyI_jblsYPfzptfzAxJgJGL_eX5Hn0/edit#heading=h.kdgu3otgt0my

[scothis]

I implemented this concept as a PoC a few months ago. Feel free to use as much (or little) as is helpful. https://github.com/scothis/eventing/blob/sequence/pkg/apis/eventing/v1alpha1/sequence_types.go

[akashrv]

#1039 sounds similar.

[lionelvillard]

I'm going to take a stab at it. I started to gather some requirements listed below. Not a complete list by no mean, and it goes beyond defining simple pipelines.

• being able to orchestrate the flow of function using connectors like sequence, conditional, error handling and iteration.
• being able to connect functions returning non-cloudevent compatible response
• being able to directly connect functions (non-brokered vs brokered)

@vaikas-google what do you think?

[vaikas]

I'd prefer to start with a simple sequence as described in the proposal.
non-cloudevents seems bad as the entire assumption has been to standardize on cloud events, so it would be odd to wire a pipeline in and get something else out of there.
I think one of the important aspects of this work would be indeed to wire functions in without involving the broker so that the intermediate events are not visible through the broker (for example).
If you haven't gotten too far, I can share what I started with last week (and put aside while I worked on the in-memory CRD conversion), and perhaps we can compare notes for the starting point on that.
I'll share it in just a few minutes once I rebase, etc.

[lionelvillard]

@vaikas-google yes we should start simple. I agree that wiring functions directly is important for various reasons. I also think there should be an option to use a broker to persist intermediate state.

I didn't go very far, so please share what you have so far, thx.

[vaikas]

#1239