Document few messages from current event sources

[nachocano]

By documenting few examples of how our current event sources are sending cloud events into the eventing mesh, we might be able to better understand what we actually need for a first cut of Registry (and also some of the filtering for Broker/Trigger)

Below there are few dumps from the message_dumper. Only the HTTP headers are included. Most of the events below were sent using cloud events spec v0.1. Since then, most of the sources in eventing-sources repo have been changed to send cloud events v0.2. Note how the HTTP header names (Ce-*) change.

GitHub Event Source

pull_request

Host: message-dumper.default.svc.cluster.local
Accept-Encoding: gzip
Ce-Cloudeventsversion: 0.1
Ce-Eventid: 95481390-3bbc-11e9-9399-36e044325046
Ce-Eventtime: 2019-03-01T00:54:45.218614027Z
Ce-Eventtype: dev.knative.source.github.pull_request
Ce-Knativehistory: t.default.triggers.cluster.local
Ce-Knativehistory: default-broker.default.svc.cluster.local
Ce-Source: https://github.com/nachocano/eventing/pull/2
Ce-X-Github-Delivery: "95481390-3bbc-11e9-9399-36e044325046"
Ce-X-Github-Event: "pull_request"
Content-Length: 19741
Content-Type: application/json
User-Agent: Go-http-client/1.1
X-B3-Parentspanid: 15c7392dc989d476
X-B3-Sampled: 1
X-B3-Spanid: fe270d91f8c7b851
X-B3-Traceid: 80d46a89f5060f1a
X-Envoy-Expected-Rq-Timeout-Ms: 600000
X-Envoy-Internal: true
X-Forwarded-For: 127.0.0.1, 127.0.0.1
X-Forwarded-Proto: http
X-Request-Id: 98ed55ab-6f50-9d8f-898d-5ce800ab7a8b

BitBucket Event Source

repo:push

Host: message-dumper.default.svc.cluster.local
Accept-Encoding: gzip
Ce-Event-Key: "repo:push"
Ce-Id: d65b2f88-11b6-4588-b1ba-d5095a160235
Ce-Knativehistory: default-broker-x46lv.default.channels.cluster.local
Ce-Knativehistory: t.default.triggers.cluster.local
Ce-Request-Uuid: "d65b2f88-11b6-4588-b1ba-d5095a160235"
Ce-Source: https://api.bitbucket.org/2.0/repositories/nachocano/eventing-test/commits?include=3f909319ef2732dd86c448947a7d396faf6917af&exclude=b8d1aded3b8c3d6c32b563113a1388d7c99cc35a
Ce-Specversion: 0.2
Ce-Time: 2019-03-06T01:16:40.667029098Z
Ce-Type: dev.knative.source.bitbucket.repo:push
Content-Length: 4437
Content-Type: application/json
User-Agent: Go-http-client/1.1
X-B3-Parentspanid: 36c826a98b911016
X-B3-Sampled: 1
X-B3-Spanid: 1fe8b6975b2b6c2d
X-B3-Traceid: 0573724c260219df
X-Envoy-Expected-Rq-Timeout-Ms: 600000
X-Envoy-Internal: true
X-Forwarded-For: 127.0.0.1, 127.0.0.1
X-Forwarded-Proto: http
X-Request-Id: 81b4603c-3f2b-9e0b-ace4-fe3a7aeb2047

GCP PubSub source

publish message to new_testing topic

Host: message-dumper.default.svc.cluster.local
Accept-Encoding: gzip
Ce-Cloudeventsversion: 0.1
Ce-Eventid: 414566320791303
Ce-Eventtime: 2019-02-28T23:04:45.406Z
Ce-Eventtype: google.pubsub.topic.publish
Ce-Knativehistory: t.default.triggers.cluster.local
Ce-Knativehistory: default-broker.default.svc.cluster.local
Ce-Source: //pubsub.googleapis.com/knative-project-228222/topics/new_testing
Content-Length: 121
Content-Type: application/json
User-Agent: Go-http-client/1.1
X-B3-Parentspanid: 3270f8840314fa7d
X-B3-Sampled: 1
X-B3-Spanid: 5fdd60dab6d58725
X-B3-Traceid: 18f61c3161ef8df2
X-Envoy-Expected-Rq-Timeout-Ms: 600000
X-Envoy-Internal: true
X-Forwarded-For: 127.0.0.1, 127.0.0.1
X-Forwarded-Proto: http
X-Request-Id: bcb0b147-22e0-9a0d-86ce-f475d2ba151d

Kubernetes Event Source

busybox

Host: message-dumper.default.svc.cluster.local
Accept-Encoding: gzip
Ce-Cloudeventsversion: 0.1
Ce-Eventid: 7270f0c7-3ba0-11e9-842c-42010a8a0103
Ce-Eventtime: 2019-02-28T21:33:11Z
Ce-Eventtype: dev.knative.k8s.event
Ce-Knativehistory: t.default.triggers.cluster.local
Ce-Knativehistory: default-broker.default.svc.cluster.local
Ce-Source: /apis/v1/namespaces/default/pods/busybox
Content-Length: 745
Content-Type: application/json
User-Agent: Go-http-client/1.1
X-B3-Parentspanid: 9849ef63f21db180
X-B3-Sampled: 1
X-B3-Spanid: e94380a55e480ae4
X-B3-Traceid: 55ea296f18e053c8
X-Envoy-Expected-Rq-Timeout-Ms: 600000
X-Envoy-Internal: true
X-Forwarded-For: 127.0.0.1, 127.0.0.1
X-Forwarded-Proto: http
X-Request-Id: 70fc878d-006f-9ec4-9c94-fa92b30c9747

CronJob Event Source

schedule = "*/2 * * * *"

Host: message-dumper.default.svc.cluster.local
Accept-Encoding: gzip
Ce-Cloudeventsversion: 0.1
Ce-Eventid: f6aa3f90-3ba0-11e9-842c-42010a8a0103
Ce-Eventtime: 2019-02-28T21:36:53Z
Ce-Eventtype: dev.knative.k8s.event
Ce-Knativehistory: default-broker.default.svc.cluster.local
Ce-Knativehistory: t.default.triggers.cluster.local
Ce-Source: /apis/v1/namespaces/default/pods/cronjob-test-cronjob-source-gq9xb-58fd74f85c-dfbp6
Content-Length: 891
Content-Type: application/json
User-Agent: Go-http-client/1.1
X-B3-Parentspanid: 4de786ce3dad7798
X-B3-Sampled: 1
X-B3-Spanid: bc907c9fd868dd27
X-B3-Traceid: d6b4c733c2fcc30f
X-Forwarded-For: 127.0.0.1
X-Forwarded-Proto: http
X-Request-Id: e1ea514e-d7e0-9e14-8189-4cb6f05a8885

[nachocano]

/kind doc

[nachocano]

FYI
@Harwayne
@grantr

[nachocano]

/assign

[nachocano]

My takeaways so far from this are:

1. We cannot use the Ce-Eventtype or Ce-Type header values as the name of potential Custom Objects of Kind EventType in the Registry, as they may not conform to Kubernetes naming formats. An option could be:

   • Given that we are changing the types right now (e.g., from repo:push to dev.knative.source.bitbucket.repo:push), we might just change it in a way that complies with k8s naming conventions. And maybe add an optional alias field in the EventType CRD, with the original type name).
   • ....

2. It seems that it might be good enough to support exact matching of Ce-Eventtype or Ce-Type attributes for a first cut of Broker/Trigger.

3. We should try to support prefix match for the cloud events Ce-Source attribute in Broker/Trigger MVP, otherwise source matching will be completely useless. No consumer will match on, for example, Update t4.yaml to match instructions in doc nachocano/eventing#12. Instead, they will be interested in matching things like any pull request in the nachocano/eventing repo, e.g., https://github.com/nachocano/eventing/pull/
   If people don't want regex matches, that's fine, but IMO at least simple prefix matching should be included.

[nachocano]

fyi @akashrv

[grantr]

I think my preferred solution to problem #1 is to add a type field to EventType that is the authoritative CloudEvent type, making the EventType meta.name field non-authoritative (but ideally as close to the actual type as possible).

(uninteresting fields removed from the following examples for clarity)

kind: EventType
meta:
  # this is advisory
  name: com.github.pull_request
spec:
  # this is authoritative
  type: com.github.pull_request

kind: EventType
meta:
  # This name is generated by stripping invalid chars from the type name and
  # setting generateName on creation
  name: type-with-invalid-chars-5ha6
spec:
  type: "type with invalid chars 😀"

The user story for "list all event types" is equivalent because the event type can still be displayed using custom columns.

IMO this also simplifies (but does not solve) the problem of multiple conflicting definitions for a single type. By allowing conflicts to exist, the user (or entity creating the EventType, which may be a Broker) is not immediately responsible for resolving the conflict at creation time. Conflict resolution to be postponed to a later time, and perhaps more importantly, a resolution strategy can be selected when necessary based on the specific need. For example, an ingress validator on the data plane may want to allow an event if ANY schema matches. Separately, a control-plane conflict checker may want to generate a Kubernetes event if any two EventTypes conflict.

This data model recognizes that conflicts are expected occurrences rather than invalid edge cases.

For example:

kind: EventType
meta:
  name: com.github.pull_request
spec:
  type: com.github.pull_request
  schema: https://github.com/schemas/pull_request.v3.json

This type was created when the GitHub source adapter was installed. It specifies the com.github.pull_request type with the pull_request.v3 schema.

Later the GitLab source adapter is installed. This adapter uses the com.github.pull_request type
name also to make replacing GitHub with GitLab easier, which would normally be fine but it also specifies GitHub's newer pull_request.v4 schema. The GitHub source adapter hasn't been upgraded yet so it still specifies the v3 schema.

kind: EventType
meta:
  name: gitlab-pull-request
spec:
  type: com.github.pull_request
  schema: https://github.com/schemas/pull_request.v4.json

This state may not actually cause errors: v4 might be backward compatible with v3, allowing consumers to work with either.

[nachocano]

@grantr Makes sense! Let me recap.

The type attribute here (what I called alias) should be what it comes in the cloud event Ce-Eventtype or Ce-Type headers, right? There were some discussions on not changing the types in the receive adaptors, as we are currently doing. But for this discussion, whatever comes in those headers, would be our authoritative type. And based on that, we can generate a valid though advisory name for the CO.

I'm OK with having what you are proposing for a first cut, and we cover the user story of listing the types as you said.

[Harwayne]

FYI @vaikas-google, @evankanderson. I think these real world examples are key to understanding what the registry should look like. Furthermore, any proposal we have should show what these events look like in that proposal.

Thanks @nachocano!

[n3wscott]

Also, don't think of type as headers, they are context attributes on the cloudevent, the headers are just that projection onto binary http.

[nachocano]

Also, don't think of type as headers, they are context attributes on the cloudevent, the headers are just that projection onto binary http.

Makes sense. Should called them context attributes instead. Thanks for the clarification

[nachocano]

This help fixing #929