Define an importer specification

[mikehelmick]

This is broken out from #1381

Problem
There is currently no specification for how a behaved importer should be written.

Persona:
Event producer and System Integrator

Exit Criteria

1. Agreed upon importer specification and documentation

2. Conformance test so importers can be tested for their level of conformance

Time Estimate (optional):
5d

Additional context (optional)

This is part of the overall design proposal for issue #1381 and the following is a necessary part of that, assuming the overall design moves forward.

The primary design point here is introducing the concept of importers identifying themselves. We introduce a CloudEvents extension attribute called sourceid that allows for events to be filtered based on this attribute if present. This is needed to disambiguate between importers of events that might be connected to the same upstream producer.

The convention here is that a well-behaved importer will populate the sourceid attribute with the metadata.uid value for the object that a trigger writer would select on. This could be the importer itself, or could be a knative service. It is important to note that objects may need to coordinate.

The GCSSource actually creates a Cloud PubSub importer to bring the events into the broker. The PubSub importer should use the uid of the GCSSource since that is what the user will be configuring against.

This introduces a new data plane contract for importers. When going through an event mesh, this is unavoidable if we wish to support the FaaS scenarios as defined in #1381 . This solution makes it very important for an importer author, but it is also optional. Not all importers / event producers will have to implement the sourceid contract, but failing to do so, prevents the FaaS scenario.

Conformance tests should make it clear to a user if the importer they are using supports the sourceid contract or not.

[n3wscott]

/assign

Actively working on this.

[duglin]

As I mentioned in #1381 - this should be importerId since it is not a "source" in the CloudEvents perspective.

Also, I'm not clear on how, or who, will use this information. Can a concrete example be provided?

[mikehelmick]

sourceid vs importerid doesn't matter much to me.

As for how the information will be used, a trigger might look like this (using importerid in this case)

apiVersion: eventing.knative.dev/v1alpha1
kind: Trigger
metadata:
  name: bound-trigger
  namespace: default
spec:
  filter:
    importerId: f7ee9a93-a28e-11e9-9c8b-42010a80002b
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1alpha1
      kind: Service
      name: event-handler

[duglin]

@mikehelmick can you elaborate on why someone would do this? I thought that part of this UX design was to try to hide some of the complexities, and asking the user to find this UUID doesn't seem to align with that - especially if the importer can choose the value it'll use and it's not required to be the same thing for all importers.

If the user wants to specify the importer then I kind of wonder why they're not just talking to it directly, or setting up a dedicate broker (if they want the QoS features). What's the advantage of having a common broker in these scenarios where the user seems to want to filter on the importer? It seems that due to performance of being shared (or the added complexity above) might out-weigh the benefits.

[mikehelmick]

Sorry - I totally messed that example up. No, I don't expect users to go look up the UID of their importer and that doesn't work with a gitops scenario. I would expect something like this instead

apiVersion: eventing.knative.dev/v1alpha1
kind: Trigger
metadata:
  name: bound-trigger
  namespace: default
spec:
  broker: default
  filter:
    importer:
     ref:
       apiVersion: sources.knative.dev/v1alpha1
       kind: AwesomeSource
       name: awesome-source-importer
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1alpha1
      kind: Service
      name: event-handler
status:
  importerId: f7ee9a93-a28e-11e9-9c8b-42010a80002b

As for one broker, vs multiple brokers. I think that should be an operator concern. If expected throughput of certain event types is low, you get better resource utilization by putting them on the same broker.

[grantr]

The GCSSource actually creates a Cloud PubSub importer to bring the events into the broker. The PubSub importer should use the uid of the GCSSource since that is what the user will be configuring against.

This is an important point. Several of our current importers delegate data plane responsibilities to another object with a different name and uid. This internal object must know what filter attribute(s) the broker will expect, or the Broker needs to know how resolve the correct filter attribute(s) on the internal object. Seems like something that should be part of the importer specification.

[duglin]

@mikehelmick ok, that objRef makes more sense from a syntax perspective. Thanks

I have more comments, but I think they're more on the overall design than just this one point so I'll make them over in the other issue.