From 9c8f86a8a986b4c1387c335eca1d59bce18a229d Mon Sep 17 00:00:00 2001 From: nachocano Date: Mon, 1 Apr 2019 10:12:03 -0700 Subject: [PATCH 01/20] Registry proposal --- docs/registry/README.md | 193 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 docs/registry/README.md diff --git a/docs/registry/README.md b/docs/registry/README.md new file mode 100644 index 00000000000..24f8966027b --- /dev/null +++ b/docs/registry/README.md @@ -0,0 +1,193 @@ +# Registry Proposal + +## Objective + +Design an **initial** version of the Registry that can support discoverability of +the different event types that can be consumed from the eventing mesh. + +## Requirements + +Our design revolves around the following core requirements: + +1. We should have a Registry per namespace to enforce isolation. +2. The Registry should contain the event types that can be consumed from +the eventing mesh. If an event type is not ready for consumption, we should explicitly indicate so. +3. The event types stored in the Registry should contain all the required information +for a consumer to create a Trigger without resorting to some other OOB mechanism. + +## Design Ideas + +### EventType CRD + +We propose having a namespaced EventType CRD. Here is an example of how a CR would look like: + +```yaml +apiVersion: eventing.knative.dev/v1alpha1 +kind: EventType +metadata: + name: repopush +spec: + type: repo:push + source: my-user/my-repo + schema: my-schema + broker: default +``` + + +- The `name` of the EventType is advisory, non-authoritative. Given that Cloud Event types can +contain characters that may not comply with Kubernetes naming conventions, we will (slightly) +modify those names to make them K8s-compliant, whenever we need to generate them. + +- `type` is authoritative. This refers to the Cloud Event type as it enters into the eventing mesh. + +- `source`: an identifier of where we receive the event from. This might not necessarily be the Cloud Event source +attribute. If we receive the event from our receive adaptors, the info might come in a Cloud Event custom extension (e.g., from). + +- `schema` is a URI with the EventType schema. It may be a JSON schema, a protobuf schema, etc. It is optional. + +- `broker` refers to the Broker that can provide the EventType. + +### EventType Instantiation + +We foresee the following ways of populating the Registry: + +**1. Event Source CR installation** + +Upon installation of an Event Source CR, the source will register its EventTypes. + +Example: + +```yaml +apiVersion: sources.eventing.knative.dev/v1alpha1 +kind: GitHubSource +metadata: + name: github-source-sample + namespace: default +spec: + eventTypes: + - push + - pull_request + ownerAndRepository: my-other-user/my-other-repo + accessToken: + secretKeyRef: + name: github-secret + key: accessToken + secretToken: + secretKeyRef: + name: github-secret + key: secretToken + sink: + apiVersion: eventing.knative.dev/v1alpha1 + kind: Broker + name: default + +``` + +By applying this file, two EventTypes will be registered, with types `push` and `pull_request`, +source `my-other-user/my-other-repo`, for the `default` Broker in the `default` namespace. + +**2. Manual User Registration** + +A user manually `kubectl applies` an EventType CR. + +Example: + +```yaml +apiVersion: eventing.knative.dev/v1alpha1 +kind: EventType +metadata: + name: repofork + namespace: default +spec: + type: repo:fork + source: my-other-user/my-other-repo + broker: dev +``` + +This would register the EventType named `repofork` with type `repo:fork`, source `my-other-user/my-other-repo` +in the `dev` Broker of the `default` namespace. + + +**3. Broker Auto Registration Policy** + +Upon arrival of a non-registered EventType to a Broker ingress, and in case the Broker +ingress policy allows auto-registration of EventTypes, the Broker will create the EventType. + +Example: + +Set up a Broker with `autoAdd` enabled. + +```yaml +apiVersion: eventing.knative.dev/v1alpha1 +kind: Broker +metadata: + name: auto-add-demo +spec: + ingressPolicy: + autoAdd: true +``` + +Now if someone emits a non-registered event (e.g., `dev.knative.foo.bar`) +into the Broker `auto-add-demo`, an EventType will be created upon the event arrival. +Note that the Broker's address is well-known, it will always be +`-broker..svc.`. In this case case, it is +`auto-add-demo-broker.default.svc.cluster.local`. + +We can send the event manually. While SSHed into a `Pod` with the Istio sidecar, run: + +```shell +curl -v "http://auto-add-demo-broker.default.svc.cluster.local/" \ + -X POST \ + -H "X-B3-Flags: 1" \ + -H "CE-CloudEventsVersion: 0.1" \ + -H "CE-EventType: dev.knative.foo.bar" \ + -H "CE-EventTime: 2018-04-05T03:56:24Z" \ + -H "CE-EventID: 45a8b444-3213-4758-be3f-540bf93f85ff" \ + -H "CE-Source: dev.knative.example" \ + -H 'Content-Type: application/json' \ + -d '{ "much": "wow" }' +``` + +The Broker `auto-add-demo` will then create the EventType with type `dev.knative.foo.bar` in the Registry. + +## Discoverability Use Case + +By adding the spec.* fields of EventType as custom columns in the CRD we can fulfill the discoverability +use case: + +*As an Event Consumer I want to be able to discover the different event types that I can consume +from the different Brokers, without resorting to any OOB mechanism.* + +First, Event Consumers will list the EventTypes registered in the system + +`$ kubectl get eventtypes -n default` + +``` +NAME TYPE SOURCE SCHEMA BROKER READY REASON +dev.knative.foo.bar-55wcn dev.knative.foo.bar dev.knative.example auto-add-demo True +repofork repo:fork my-other-user/my-other-repo dev False BrokerIsNotReady +repopush repo:push my-other-user/my-other-repo my-schema default True +dev.knative.source.github.push-34cnb dev.knative.source.github.push my-user/my-repo default True +dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request my-user/my-repo default True +``` + +Then, they will be able to *easily* create the appropriate Trigger(s) + + +```yaml +apiVersion: eventing.knative.dev/v1alpha1 +kind: Trigger +metadata: + name: my-service-trigger + namespace: default +spec: + filter: + sourceAndType: + type: dev.knative.foo.bar + source: dev.knative.example + subscriber: + ref: + apiVersion: serving.knative.dev/v1alpha1 + kind: Service + name: my-service +``` From 5f7e91a9135ccedb3d397308d2f60add24e6e729 Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 2 Apr 2019 11:29:51 -0700 Subject: [PATCH 02/20] Updates after comments --- docs/registry/README.md | 70 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 5 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index 24f8966027b..053cfe106ca 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -26,10 +26,11 @@ apiVersion: eventing.knative.dev/v1alpha1 kind: EventType metadata: name: repopush + namespace: default spec: type: repo:push source: my-user/my-repo - schema: my-schema + schema: /my-schema broker: default ``` @@ -41,7 +42,16 @@ modify those names to make them K8s-compliant, whenever we need to generate them - `type` is authoritative. This refers to the Cloud Event type as it enters into the eventing mesh. - `source`: an identifier of where we receive the event from. This might not necessarily be the Cloud Event source -attribute. If we receive the event from our receive adaptors, the info might come in a Cloud Event custom extension (e.g., from). +attribute. + +If we have control over the entity emitting the Cloud Event, as is the case of many of our receive adaptors, +then we propose to add a Cloud Event custom extension (e.g., from) with this information, to ease the creation of filters +on Triggers later on. +As the Cloud Event source attribute is somewhat useless (e.g., github pull requests are populated with `https://github.com///pull/`), +there is no way of doing exact matching of Cloud Event sources on Triggers. Thus, we propose adding this custom extension +to Cloud Events whenever we can. If the extension is not present, then we fallback to the Cloud Event source. +Note that when we start supporting more advanced filtering mechanisms on Triggers, we might not need this. + - `schema` is a URI with the EventType schema. It may be a JSON schema, a protobuf schema, etc. It is optional. @@ -83,8 +93,40 @@ spec: ``` -By applying this file, two EventTypes will be registered, with types `push` and `pull_request`, -source `my-other-user/my-other-repo`, for the `default` Broker in the `default` namespace. +By applying the above file, two EventTypes will be registered, with types `dev.knative.source.github.push` and +`dev.knative.source.github.pull_request`, source `my-other-user/my-other-repo`, for the `default` Broker in the `default` + namespace, and with owner `github-source-sample`. + +In YAML, the EventTypes would look something like these: + +```yaml +apiVersion: eventing.knative.dev/v1alpha1 +kind: EventType +metadata: + generateName: dev.knative.source.github.push- + namespace: default + owner: # Owned by github-source-sample +spec: + type: dev.knative.source.github.push + source: my-other-user/my-other-repo + broker: default +--- +apiVersion: eventing.knative.dev/v1alpha1 +kind: EventType +metadata: + generateName: dev.knative.source.github.pullrequest- + namespace: default + owner: # Owned by github-source-sample +spec: + type: dev.knative.source.github.pull_request + source: my-other-user/my-other-repo + broker: default +``` + +Two things to notice: +- We generate the names by stripping invalid characters from the original type (e.g., `_`) +- The `spec.type` adds the prefix `dev.knative.source.github.` This is a separate discussion on whether we should +change the (GitHub) types or not. **2. Manual User Registration** @@ -112,6 +154,10 @@ in the `dev` Broker of the `default` namespace. Upon arrival of a non-registered EventType to a Broker ingress, and in case the Broker ingress policy allows auto-registration of EventTypes, the Broker will create the EventType. +Note that the creation of the EventType is done asynchronously, i.e., the Cloud Event is accepted and sent +to the appropriate Trigger(s) in parallel of the EventType creation. If the creation fails, on a subsequent arrival +there will be a new creation attempt. + Example: @@ -149,6 +195,20 @@ curl -v "http://auto-add-demo-broker.default.svc.cluster.local/" \ ``` The Broker `auto-add-demo` will then create the EventType with type `dev.knative.foo.bar` in the Registry. +In YAML, the EventType would look something like these: + +```yaml +apiVersion: eventing.knative.dev/v1alpha1 +kind: EventType +metadata: + generateName: dev.knative.foo.bar- + namespace: default + owner: # Owned by auto-add-demo? +spec: + type: dev.knative.foo.bar + source: dev.knative.example + broker: auto-add-demo +``` ## Discoverability Use Case @@ -166,7 +226,7 @@ First, Event Consumers will list the EventTypes registered in the system NAME TYPE SOURCE SCHEMA BROKER READY REASON dev.knative.foo.bar-55wcn dev.knative.foo.bar dev.knative.example auto-add-demo True repofork repo:fork my-other-user/my-other-repo dev False BrokerIsNotReady -repopush repo:push my-other-user/my-other-repo my-schema default True +repopush repo:push my-other-user/my-other-repo /my-schema default True dev.knative.source.github.push-34cnb dev.knative.source.github.push my-user/my-repo default True dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request my-user/my-repo default True ``` From a5e4762cd6537e52eebe1a1e432e77fa55877865 Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 2 Apr 2019 11:50:56 -0700 Subject: [PATCH 03/20] adding broker ingress policies for reference --- docs/registry/README.md | 56 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 5 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index 053cfe106ca..c92886df46a 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -223,12 +223,12 @@ First, Event Consumers will list the EventTypes registered in the system `$ kubectl get eventtypes -n default` ``` -NAME TYPE SOURCE SCHEMA BROKER READY REASON -dev.knative.foo.bar-55wcn dev.knative.foo.bar dev.knative.example auto-add-demo True -repofork repo:fork my-other-user/my-other-repo dev False BrokerIsNotReady +NAME TYPE SOURCE SCHEMA BROKER READY REASON +dev.knative.foo.bar-55wcn dev.knative.foo.bar dev.knative.example auto-add-demo True +repofork repo:fork my-other-user/my-other-repo dev False BrokerIsNotReady repopush repo:push my-other-user/my-other-repo /my-schema default True -dev.knative.source.github.push-34cnb dev.knative.source.github.push my-user/my-repo default True -dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request my-user/my-repo default True +dev.knative.source.github.push-34cnb dev.knative.source.github.push my-user/my-repo default True +dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request my-user/my-repo default True ``` Then, they will be able to *easily* create the appropriate Trigger(s) @@ -251,3 +251,49 @@ spec: kind: Service name: my-service ``` + +## Broker Ingress Policies + +We propose adding an `ingressPolicy` field to the Broker's spec CRD. +Here are three CR examples with different policies. Note that we are omitting +Broker's fields irrelevant for this discussion. + +**1. Allow Any** + +By not specifying an ingress policy, the default will be to accept any event. + +```yaml +apiVersion: eventing.knative.dev/v1alpha1 +kind: Broker +metadata: + name: broker-allow-any +``` + +**2. Allow Registered** + +By setting the ingress policy to not allow any, the broker will accept only events with EventTypes in the Registry. + +```yaml +apiVersion: eventing.knative.dev/v1alpha1 +kind: Broker +metadata: + name: broker-allow-registered +spec: + ingressPolicy: + allowAny: false +``` + +**3. Auto Add** + +By setting the ingress policy to auto add, the broker will accept any event and will add its EventType +to the Registry (in case is not present). + +```yaml +apiVersion: eventing.knative.dev/v1alpha1 +kind: Broker +metadata: + name: broker-auto-add +spec: + ingressPolicy: + autoAdd: true +``` From 269205e87eb3a3632b761107cef1f226a7962579 Mon Sep 17 00:00:00 2001 From: nachocano Date: Mon, 8 Apr 2019 15:13:32 -0700 Subject: [PATCH 04/20] More clarifications. Restructuring the document a bit. Referring to the User Stories document, which seems that have been forgotten. Mainly emphasizing that this is intended for a Registry MVP. --- docs/registry/README.md | 105 ++++++++++++++++++++++++++-------------- 1 file changed, 69 insertions(+), 36 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index c92886df46a..991edf652b5 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -1,9 +1,27 @@ # Registry Proposal +## Problem + +As an `Event Consumer` I want to be able to discover the different event types that I can consume +from the different Brokers, without resorting to any OOB mechanism. +This is also known as the `discoverability` use case, and is the main focus of this proposal. + ## Objective -Design an **initial** version of the Registry that can support discoverability of -the different event types that can be consumed from the eventing mesh. +Design an **initial** version of the **Registry** for the **MVP** that can support discoverability of +the different event types that can be consumed from the eventing mesh. For details on the different user stories +that this proposal enables, please refer to the +[User stories and personas for Knative eventing](https://docs.google.com/document/d/15uhyqQvaomxRX2u8s0i6CNhA86BQTNztkdsLUnPmvv4/edit?usp=sharing) document. + +#### Out of scope + +- Registry to Registry communication. This doesn't seem needed for an MVP. +- Security-related matters. Those are handled offline by the `Cluster Configurator`, e.g., the `Cluster Configurator` +takes care of setting up Secrets for connecting to a GitHub repo, and so on. +- Registry synchronization with `Event Producers`. We assume that if new GitHub events are created by +GitHub after our cluster has been configured (i.e., our GitHub CRD Source installed) and the appropriate webhooks +have been created, we will need to create new webhooks (and update the GitHub CRD Source) if we want to listen for +those new events. Such task will again be in charge of the `Cluster Configurator`. ## Requirements @@ -11,15 +29,18 @@ Our design revolves around the following core requirements: 1. We should have a Registry per namespace to enforce isolation. 2. The Registry should contain the event types that can be consumed from -the eventing mesh. If an event type is not ready for consumption, we should explicitly indicate so. +the eventing mesh in order to support the `discoverability` use case. +If an event type is not ready for consumption, we should explicitly indicate so (e.g., if the Broker +is not ready). 3. The event types stored in the Registry should contain all the required information for a consumer to create a Trigger without resorting to some other OOB mechanism. -## Design Ideas + +## Proposed Ideas ### EventType CRD -We propose having a namespaced EventType CRD. Here is an example of how a CR would look like: +We propose introducing a namespaced-EventType CRD. Here is an example of how a CR would look like: ```yaml apiVersion: eventing.knative.dev/v1alpha1 @@ -34,7 +55,6 @@ spec: broker: default ``` - - The `name` of the EventType is advisory, non-authoritative. Given that Cloud Event types can contain characters that may not comply with Kubernetes naming conventions, we will (slightly) modify those names to make them K8s-compliant, whenever we need to generate them. @@ -57,13 +77,14 @@ Note that when we start supporting more advanced filtering mechanisms on Trigger - `broker` refers to the Broker that can provide the EventType. -### EventType Instantiation +### Typical Flow -We foresee the following ways of populating the Registry: +`1.` A `Cluster Configurator` configures the cluster in a way that allows the population of EventTypes in the Registry. +We foresee the following three ways of populating the Registry so far: -**1. Event Source CR installation** +`1.1` Event Source CR installation -Upon installation of an Event Source CR, the source will register its EventTypes. +Upon installation of an Event Source CR by a `Cluster Configurator`, the Source will register its EventTypes. Example: @@ -97,7 +118,11 @@ By applying the above file, two EventTypes will be registered, with types `dev.k `dev.knative.source.github.pull_request`, source `my-other-user/my-other-repo`, for the `default` Broker in the `default` namespace, and with owner `github-source-sample`. -In YAML, the EventTypes would look something like these: +Note that the `Cluster Configurator` is the person in charge of taken care of authentication-related matters. E.g., if a new `Event Consumer` +wants to listen for events from a different GitHub repo, the `Cluster Configurator` will take care of the necessary secrets generation, +and new Source instantiation. + +In YAML, the above EventTypes would look something like these: ```yaml apiVersion: eventing.knative.dev/v1alpha1 @@ -125,12 +150,12 @@ spec: Two things to notice: - We generate the names by stripping invalid characters from the original type (e.g., `_`) -- The `spec.type` adds the prefix `dev.knative.source.github.` This is a separate discussion on whether we should +- The `spec.type` adds the prefix `dev.knative.source.github.` This is a **separate discussion** on whether we should change the (GitHub) types or not. -**2. Manual User Registration** +`1.2.` Manual Registration -A user manually `kubectl applies` an EventType CR. +The `Cluster Configurator` manually `kubectl applies` an EventType CR. Example: @@ -149,15 +174,21 @@ spec: This would register the EventType named `repofork` with type `repo:fork`, source `my-other-user/my-other-repo` in the `dev` Broker of the `default` namespace. +As under the hood, `kubeclt apply` just makes a REST call to the API server with the appropriate RBAC permissions, +the `Cluster Configurator` can give EventType `create` permissions to trusted parties, so that they can register +their EventTypes. -**3. Broker Auto Registration Policy** -Upon arrival of a non-registered EventType to a Broker ingress, and in case the Broker -ingress policy allows auto-registration of EventTypes, the Broker will create the EventType. +`1.3` Broker Auto Registration Policy + +The `Cluster Configurator` configures the Broker ingress policy to allow auto-registration of EventTypes. +Upon arrival of a non-registered EventType to the Broker ingress, the Broker will then create that type. Note that the creation of the EventType is done asynchronously, i.e., the Cloud Event is accepted and sent to the appropriate Trigger(s) in parallel of the EventType creation. If the creation fails, on a subsequent arrival -there will be a new creation attempt. +there will be a new creation attempt. +Although sub-optimal (as `Event Consumers` find out about EventTypes upon first arrival), we believe this mechanism is needed +for Sources where the `Cluster Configurator` does not know in advance the EventTypes they can produced (e.g., a ContainerSource). Example: @@ -210,15 +241,9 @@ spec: broker: auto-add-demo ``` -## Discoverability Use Case - -By adding the spec.* fields of EventType as custom columns in the CRD we can fulfill the discoverability -use case: +`2.` An `Event Consumer` checks the Registry to see what EventTypes it can consume from the mesh. -*As an Event Consumer I want to be able to discover the different event types that I can consume -from the different Brokers, without resorting to any OOB mechanism.* - -First, Event Consumers will list the EventTypes registered in the system +Example: `$ kubectl get eventtypes -n default` @@ -231,8 +256,9 @@ dev.knative.source.github.push-34cnb dev.knative.source.github.push dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request my-user/my-repo default True ``` -Then, they will be able to *easily* create the appropriate Trigger(s) +`3.` The `Event Consumer` creates a Trigger to listen to an EventType in the Registry. +Example: ```yaml apiVersion: eventing.knative.dev/v1alpha1 @@ -252,13 +278,16 @@ spec: name: my-service ``` -## Broker Ingress Policies +### Broker Ingress Policies -We propose adding an `ingressPolicy` field to the Broker's spec CRD. -Here are three CR examples with different policies. Note that we are omitting -Broker's fields irrelevant for this discussion. +We briefly mentioned configuring an ingress policy in the Broker so that it can auto-register EventTypes. +In order to support that, we propose adding an `ingressPolicy` field to the Broker's spec CRD. + +Below are three CR examples with Brokers configured using different policies. +We are omitting Broker's fields irrelevant to this discussion. +Note that such configuration is done by the `Cluster Configurator`. -**1. Allow Any** +- Allow Any By not specifying an ingress policy, the default will be to accept any event. @@ -269,9 +298,9 @@ metadata: name: broker-allow-any ``` -**2. Allow Registered** +- Allow Registered -By setting the ingress policy to not allow any, the broker will accept only events with EventTypes in the Registry. +By setting the ingress policy to not allow any, the Broker will accept only events with EventTypes in the Registry. ```yaml apiVersion: eventing.knative.dev/v1alpha1 @@ -283,9 +312,9 @@ spec: allowAny: false ``` -**3. Auto Add** +- Auto Add -By setting the ingress policy to auto add, the broker will accept any event and will add its EventType +By setting the ingress policy to auto add, the Broker will accept any event and will add its EventType to the Registry (in case is not present). ```yaml @@ -297,3 +326,7 @@ spec: ingressPolicy: autoAdd: true ``` + +Note that more policies should probably need to be configured, e.g., allow auto-registration of EventTypes received +from within the cluster (e.g., from a response of a Service running in the cluster) as opposed to external services, and so on. +We just enumerate three simple cases here. From 928fd2e3b8574de31129ba13d44776c673ba28be Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 9 Apr 2019 10:11:03 -0700 Subject: [PATCH 05/20] started a FAQ list at the bottom, for further clarifications. --- docs/registry/README.md | 68 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 64 insertions(+), 4 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index 991edf652b5..2be01107ad1 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -10,7 +10,7 @@ This is also known as the `discoverability` use case, and is the main focus of t Design an **initial** version of the **Registry** for the **MVP** that can support discoverability of the different event types that can be consumed from the eventing mesh. For details on the different user stories -that this proposal enables, please refer to the +that this proposal touches, please refer to the [User stories and personas for Knative eventing](https://docs.google.com/document/d/15uhyqQvaomxRX2u8s0i6CNhA86BQTNztkdsLUnPmvv4/edit?usp=sharing) document. #### Out of scope @@ -21,7 +21,8 @@ takes care of setting up Secrets for connecting to a GitHub repo, and so on. - Registry synchronization with `Event Producers`. We assume that if new GitHub events are created by GitHub after our cluster has been configured (i.e., our GitHub CRD Source installed) and the appropriate webhooks have been created, we will need to create new webhooks (and update the GitHub CRD Source) if we want to listen for -those new events. Such task will again be in charge of the `Cluster Configurator`. +those new events. Until doing so, those new events shouldn't be listed in the Registry. +Such task will again be in charge of the `Cluster Configurator`. ## Requirements @@ -118,7 +119,7 @@ By applying the above file, two EventTypes will be registered, with types `dev.k `dev.knative.source.github.pull_request`, source `my-other-user/my-other-repo`, for the `default` Broker in the `default` namespace, and with owner `github-source-sample`. -Note that the `Cluster Configurator` is the person in charge of taken care of authentication-related matters. E.g., if a new `Event Consumer` +Note that the `Cluster Configurator` is the person in charge of taking care of authentication-related matters. E.g., if a new `Event Consumer` wants to listen for events from a different GitHub repo, the `Cluster Configurator` will take care of the necessary secrets generation, and new Source instantiation. @@ -315,7 +316,7 @@ spec: - Auto Add By setting the ingress policy to auto add, the Broker will accept any event and will add its EventType -to the Registry (in case is not present). +to the Registry (in case it is not present). ```yaml apiVersion: eventing.knative.dev/v1alpha1 @@ -330,3 +331,62 @@ spec: Note that more policies should probably need to be configured, e.g., allow auto-registration of EventTypes received from within the cluster (e.g., from a response of a Service running in the cluster) as opposed to external services, and so on. We just enumerate three simple cases here. + +## FAQ + +Here is a list of frequently asked questions that may help clarify the scope of the Registry. + +- Is the Registry meant to be used just for creating Triggers or for also setting up Event Sources? + + It's mainly intended for helping the `Event Consumer` with the `discoverability` use case. Therefore, + it is meant for helping out creating Triggers. + +- If I have a simple use case where I'm just setting up an Event Source and my KnService is its Sink (i.e., no Triggers involved), +is there a need/use for the Registry? + + In this case, we believe there is no need for the Registry. As you can see in the EventType CRD, there is a mandatory + `broker` field. If you are not sending events to a Broker, then there is no need to use a Registry. + Implementation-wise, we can check whether the Source's sink kind is `Broker`, and if so, then register its EventTypes. + +- Is the Registry meant to be used in a single-user environment where the same person is setting up both the Event Source and +the destination Sink? + + We believe is mainly intended for multi-user environment. A `Cluster Configurator` persona is in charge of setting up + the Sources, and `Event Consumers` are the ones that create the Triggers. Having said that, it can also be used in + a single-user environment, but the Registry might not add much value compared to what we have right now in terms of + `discoverability`, but it surely does in terms of, for example, `admission control`. + +- Does a user need to know which type of environment they're in before they should know if they should look at the Registry? +In other words, is a Registry always going to be there and if not under what conditions will it be? + + If there are Sources pointing to Brokers, then there should be a Registry. `Event Consumers` will always be able to + `kubectl get eventtypes -n `. + +- Once an Event Source is created, how is a new one created with different auth in an env where the user is really just meant +to deal with Triggers? This may not be a Registry specific question but if one of the goals of the Registry is to make it +so that the user only deals with Triggers using the info in the Registry, I think this aspect comes into play. + + We believe the Event Source instantiation with different credentials should be handled by the `Cluster Configurer`. If the + `Cluster Configurer` persona happens to be the same person as the `Event Consumer` persona, then it will have to take care + of creating the Source. This is related to the question of a single-user, multi-user environment above. + +- I've heard conflicting messages around whether the Registry is just a list of Event Types or it will also be a list of +Event Sources so that the user doesn't need to query the CRDs to get the list. We need to be clear about this. + + The Registry is a list of EventTypes. Having said that, the `Event Consumer` could also (if it has the proper RBAC permissions) + list Event Sources (e.g., `kubectl get crds -l eventing.knative.dev/source=true`), but that list is not part of what we call + Registry here. The idea behind the fields in the EventType CRD is to have all the necessary information there in + order to create a Triggers, thus, in most cases, the `Event Consumer` shouldn't have to list Sources. + +- I wonder if the Event Source populating the Registry should happen when the Event Source is loaded into the system, +meaning when the Event Source's CRD is installed (not when an instance of the CRD is created). + +The problem with that is that you don't have a namespace (nor Broker, user/repo, etc.) at that point. +Which namespace the EventType should be created on? Pointing to which Broker? +Implementation-wise, one potential solution is to have a controller for source CRDs, whenever one is installed, search for all the namespaces with +eventing enabled (`kubectl get namespaces -l knative-eventing-injection=enabled`), and adding all the possible EventTypes from that CRD to each of +the Brokers in those namespaces. A downside of this is that the Registry information is not "accurate", in the sense that it only has info about EventTypes +that may eventually flow in the system. But actually, they will only be able to flow when a CR is created. + +- ... + From d4e5a7db643873639ea65d94387ee42221d79026 Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 9 Apr 2019 13:11:57 -0700 Subject: [PATCH 06/20] updates after Ville's comments --- docs/registry/README.md | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index 2be01107ad1..f100ffd3848 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -282,22 +282,12 @@ spec: ### Broker Ingress Policies We briefly mentioned configuring an ingress policy in the Broker so that it can auto-register EventTypes. -In order to support that, we propose adding an `ingressPolicy` field to the Broker's spec CRD. +In order to support that, we propose adding an `ingressPolicy` field to the Broker's spec CRD. -Below are three CR examples with Brokers configured using different policies. +Below are two CR examples with Brokers configured using different policies. Note that the fields of +`ingressPolicy` might change (e.g., `ingressPolicy` might just end up being a string). We are omitting Broker's fields irrelevant to this discussion. -Note that such configuration is done by the `Cluster Configurator`. - -- Allow Any - -By not specifying an ingress policy, the default will be to accept any event. - -```yaml -apiVersion: eventing.knative.dev/v1alpha1 -kind: Broker -metadata: - name: broker-allow-any -``` +Also note that such configuration is done by the `Cluster Configurator`. - Allow Registered @@ -328,9 +318,12 @@ spec: autoAdd: true ``` -Note that more policies should probably need to be configured, e.g., allow auto-registration of EventTypes received +By not specifying an ingress policy, we thought that the default behavior would be to accept any event, which is similar to `Auto Add` +but without registration of events. This needs further discussion. + +Note that more policies should probably need to be configured in the future, e.g., allow auto-registration of EventTypes received from within the cluster (e.g., from a response of a Service running in the cluster) as opposed to external services, and so on. -We just enumerate three simple cases here. +We just enumerate two simple cases here. ## FAQ @@ -359,8 +352,8 @@ the destination Sink? - Does a user need to know which type of environment they're in before they should know if they should look at the Registry? In other words, is a Registry always going to be there and if not under what conditions will it be? - If there are Sources pointing to Brokers, then there should be a Registry. `Event Consumers` will always be able to - `kubectl get eventtypes -n `. + A Registry will always be there, i.e., `Event Consumers` will always be able to `kubectl get eventtypes -n `. + In case no CR Sources are pointing to Brokers, then the Registry will be empty. - Once an Event Source is created, how is a new one created with different auth in an env where the user is really just meant to deal with Triggers? This may not be a Registry specific question but if one of the goals of the Registry is to make it From 0a4b49c792a14e83eb5b04922530ac040b8f3eb9 Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 9 Apr 2019 14:42:22 -0700 Subject: [PATCH 07/20] addressing some of Doug's comments. --- docs/registry/README.md | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index f100ffd3848..90f22e1613d 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -12,6 +12,7 @@ Design an **initial** version of the **Registry** for the **MVP** that can suppo the different event types that can be consumed from the eventing mesh. For details on the different user stories that this proposal touches, please refer to the [User stories and personas for Knative eventing](https://docs.google.com/document/d/15uhyqQvaomxRX2u8s0i6CNhA86BQTNztkdsLUnPmvv4/edit?usp=sharing) document. +Note that this proposal targets the cases were the Broker/Trigger model is used. #### Out of scope @@ -56,22 +57,24 @@ spec: broker: default ``` -- The `name` of the EventType is advisory, non-authoritative. Given that Cloud Event types can +- The `name` of the EventType is advisory, non-authoritative. Given that CloudEvents types can contain characters that may not comply with Kubernetes naming conventions, we will (slightly) modify those names to make them K8s-compliant, whenever we need to generate them. -- `type` is authoritative. This refers to the Cloud Event type as it enters into the eventing mesh. +- `type` is authoritative. This refers to the CloudEvent type as it enters into the eventing mesh. -- `source`: an identifier of where we receive the event from. This might not necessarily be the Cloud Event source +- `source`: an identifier of where we receive the event from. This might not necessarily be the CloudEvent source attribute. -If we have control over the entity emitting the Cloud Event, as is the case of many of our receive adaptors, -then we propose to add a Cloud Event custom extension (e.g., from) with this information, to ease the creation of filters +If we have control over the entity emitting the CloudEvent, as is the case of many of our receive adaptors, +then we propose to add a CloudEvent custom extension (e.g., from) with this information, to ease the creation of filters on Triggers later on. -As the Cloud Event source attribute is somewhat useless (e.g., github pull requests are populated with `https://github.com///pull/`), -there is no way of doing exact matching of Cloud Event sources on Triggers. Thus, we propose adding this custom extension -to Cloud Events whenever we can. If the extension is not present, then we fallback to the Cloud Event source. -Note that when we start supporting more advanced filtering mechanisms on Triggers, we might not need this. +As the CloudEvent source attribute is somewhat useless (e.g., github pull requests are populated with `https://github.com///pull/`), +there is no way of doing exact matching of CloudEvent sources on Triggers. Thus, we propose adding this custom extension +to CloudEvents whenever we can. If the extension is not present, then we fallback to the CloudEvent source. +Note that when we start supporting more advanced filtering mechanisms on Triggers, we might not need this. Further, with the +addition of `subject`, the meaning of the CloudEvent source might change, and we might be better off then. This needs further +discussion. - `schema` is a URI with the EventType schema. It may be a JSON schema, a protobuf schema, etc. It is optional. @@ -117,7 +120,8 @@ spec: By applying the above file, two EventTypes will be registered, with types `dev.knative.source.github.push` and `dev.knative.source.github.pull_request`, source `my-other-user/my-other-repo`, for the `default` Broker in the `default` - namespace, and with owner `github-source-sample`. + namespace, and with owner `github-source-sample`. This should be done by the Event Source controller, in this case, + the GitHubSource controller. Note that the `Cluster Configurator` is the person in charge of taking care of authentication-related matters. E.g., if a new `Event Consumer` wants to listen for events from a different GitHub repo, the `Cluster Configurator` will take care of the necessary secrets generation, @@ -184,7 +188,7 @@ their EventTypes. The `Cluster Configurator` configures the Broker ingress policy to allow auto-registration of EventTypes. Upon arrival of a non-registered EventType to the Broker ingress, the Broker will then create that type. -Note that the creation of the EventType is done asynchronously, i.e., the Cloud Event is accepted and sent +Note that the creation of the EventType is done asynchronously, i.e., the CloudEvent is accepted and sent to the appropriate Trigger(s) in parallel of the EventType creation. If the creation fails, on a subsequent arrival there will be a new creation attempt. @@ -337,10 +341,12 @@ Here is a list of frequently asked questions that may help clarify the scope of - If I have a simple use case where I'm just setting up an Event Source and my KnService is its Sink (i.e., no Triggers involved), is there a need/use for the Registry? - In this case, we believe there is no need for the Registry. As you can see in the EventType CRD, there is a mandatory - `broker` field. If you are not sending events to a Broker, then there is no need to use a Registry. - Implementation-wise, we can check whether the Source's sink kind is `Broker`, and if so, then register its EventTypes. - + As stated before, this Registry proposal for the MVP helps creating Triggers, i.e., when you use the Broker/Trigger + model. As you can see in the EventType CRD, there is a mandatory `broker` field. If you are not sending events to a Broker, + then EventTypes won't be added to the Registry (at least in this proposal). + Implementation-wise, we can check whether the Source's sink kind is `Broker`, and if so, then register its EventTypes. + + - Is the Registry meant to be used in a single-user environment where the same person is setting up both the Event Source and the destination Sink? From 6155ebb1f662e7d8930d27d3446e47103bde5c85 Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 9 Apr 2019 15:34:40 -0700 Subject: [PATCH 08/20] typos --- docs/registry/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index 90f22e1613d..e663ccb86a5 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -192,8 +192,8 @@ Note that the creation of the EventType is done asynchronously, i.e., the CloudE to the appropriate Trigger(s) in parallel of the EventType creation. If the creation fails, on a subsequent arrival there will be a new creation attempt. -Although sub-optimal (as `Event Consumers` find out about EventTypes upon first arrival), we believe this mechanism is needed -for Sources where the `Cluster Configurator` does not know in advance the EventTypes they can produced (e.g., a ContainerSource). +Although sub-optimal (as `Event Consumers` find out about EventTypes upon first arrival), we believe this mechanism (or something similar) +is needed for Sources where the `Cluster Configurator` does not know in advance the EventTypes they can produce (e.g., a ContainerSource). Example: @@ -375,7 +375,7 @@ Event Sources so that the user doesn't need to query the CRDs to get the list. W The Registry is a list of EventTypes. Having said that, the `Event Consumer` could also (if it has the proper RBAC permissions) list Event Sources (e.g., `kubectl get crds -l eventing.knative.dev/source=true`), but that list is not part of what we call Registry here. The idea behind the fields in the EventType CRD is to have all the necessary information there in - order to create a Triggers, thus, in most cases, the `Event Consumer` shouldn't have to list Sources. + order to create Triggers, thus, in most cases, the `Event Consumer` shouldn't have to list Sources. - I wonder if the Event Source populating the Registry should happen when the Event Source is loaded into the system, meaning when the Event Source's CRD is installed (not when an instance of the CRD is created). @@ -385,7 +385,7 @@ Which namespace the EventType should be created on? Pointing to which Broker? Implementation-wise, one potential solution is to have a controller for source CRDs, whenever one is installed, search for all the namespaces with eventing enabled (`kubectl get namespaces -l knative-eventing-injection=enabled`), and adding all the possible EventTypes from that CRD to each of the Brokers in those namespaces. A downside of this is that the Registry information is not "accurate", in the sense that it only has info about EventTypes -that may eventually flow in the system. But actually, they will only be able to flow when a CR is created. +that may potentially flow in the system. But actually, they will only be able to flow when a CR is created. - ... From 90dfa070e8481b8d395d3b3bb2100f5c0a13a01b Mon Sep 17 00:00:00 2001 From: Nacho Cano Date: Thu, 18 Apr 2019 12:00:07 -0700 Subject: [PATCH 09/20] removing the custom extension from, as we will be able to use Ce-source. --- docs/registry/README.md | 49 +++++++++++++++++------------------------ 1 file changed, 20 insertions(+), 29 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index e663ccb86a5..c90b240651a 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -52,7 +52,7 @@ metadata: namespace: default spec: type: repo:push - source: my-user/my-repo + source: /my-user/my-repo schema: /my-schema broker: default ``` @@ -63,19 +63,12 @@ modify those names to make them K8s-compliant, whenever we need to generate them - `type` is authoritative. This refers to the CloudEvent type as it enters into the eventing mesh. -- `source`: an identifier of where we receive the event from. This might not necessarily be the CloudEvent source -attribute. - -If we have control over the entity emitting the CloudEvent, as is the case of many of our receive adaptors, -then we propose to add a CloudEvent custom extension (e.g., from) with this information, to ease the creation of filters -on Triggers later on. -As the CloudEvent source attribute is somewhat useless (e.g., github pull requests are populated with `https://github.com///pull/`), -there is no way of doing exact matching of CloudEvent sources on Triggers. Thus, we propose adding this custom extension -to CloudEvents whenever we can. If the extension is not present, then we fallback to the CloudEvent source. -Note that when we start supporting more advanced filtering mechanisms on Triggers, we might not need this. Further, with the -addition of `subject`, the meaning of the CloudEvent source might change, and we might be better off then. This needs further -discussion. +- `source`: is a valid URI. Refers to the CloudEvent source as it enters into the eventing mesh. +Given that `subject` was approved in the new CloudEvents spec, the CloudEvents `source` became more suitable for +our purposes. Before, it contained dynamically generated information (e.g., `https://github.com///pull/`) +that we didn't know beforehand. Now, with the introduction of `subject`, `source` was 're-purposed'. In the previous example +`source` would be `https://github.com///pull` and `subject` just contain the ``. - `schema` is a URI with the EventType schema. It may be a JSON schema, a protobuf schema, etc. It is optional. @@ -119,7 +112,7 @@ spec: ``` By applying the above file, two EventTypes will be registered, with types `dev.knative.source.github.push` and -`dev.knative.source.github.pull_request`, source `my-other-user/my-other-repo`, for the `default` Broker in the `default` +`dev.knative.source.github.pull_request`, source `/my-other-user/my-other-repo`, for the `default` Broker in the `default` namespace, and with owner `github-source-sample`. This should be done by the Event Source controller, in this case, the GitHubSource controller. @@ -138,7 +131,7 @@ metadata: owner: # Owned by github-source-sample spec: type: dev.knative.source.github.push - source: my-other-user/my-other-repo + source: /my-other-user/my-other-repo broker: default --- apiVersion: eventing.knative.dev/v1alpha1 @@ -149,14 +142,15 @@ metadata: owner: # Owned by github-source-sample spec: type: dev.knative.source.github.pull_request - source: my-other-user/my-other-repo + source: /my-other-user/my-other-repo broker: default ``` -Two things to notice: +Three things to notice: - We generate the names by stripping invalid characters from the original type (e.g., `_`) -- The `spec.type` adds the prefix `dev.knative.source.github.` This is a **separate discussion** on whether we should +- We add the prefix `dev.knative.source.github.` to `spec.type`. This is a **separate discussion** on whether we should change the (GitHub) types or not. +- We add the prefix `/` to `spec.source` to make it a valid URI. `1.2.` Manual Registration @@ -172,11 +166,11 @@ metadata: namespace: default spec: type: repo:fork - source: my-other-user/my-other-repo + source: /my-other-user/my-other-repo broker: dev ``` -This would register the EventType named `repofork` with type `repo:fork`, source `my-other-user/my-other-repo` +This would register the EventType named `repofork` with type `repo:fork`, source `/my-other-user/my-other-repo` in the `dev` Broker of the `default` namespace. As under the hood, `kubeclt apply` just makes a REST call to the API server with the appropriate RBAC permissions, @@ -211,9 +205,6 @@ spec: Now if someone emits a non-registered event (e.g., `dev.knative.foo.bar`) into the Broker `auto-add-demo`, an EventType will be created upon the event arrival. -Note that the Broker's address is well-known, it will always be -`-broker..svc.`. In this case case, it is -`auto-add-demo-broker.default.svc.cluster.local`. We can send the event manually. While SSHed into a `Pod` with the Istio sidecar, run: @@ -253,12 +244,12 @@ Example: `$ kubectl get eventtypes -n default` ``` -NAME TYPE SOURCE SCHEMA BROKER READY REASON -dev.knative.foo.bar-55wcn dev.knative.foo.bar dev.knative.example auto-add-demo True -repofork repo:fork my-other-user/my-other-repo dev False BrokerIsNotReady -repopush repo:push my-other-user/my-other-repo /my-schema default True -dev.knative.source.github.push-34cnb dev.knative.source.github.push my-user/my-repo default True -dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request my-user/my-repo default True +NAME TYPE SOURCE SCHEMA BROKER READY REASON +dev.knative.foo.bar-55wcn dev.knative.foo.bar dev.knative.example auto-add-demo True +repofork repo:fork /my-other-user/my-other-repo dev False BrokerIsNotReady +repopush repo:push /my-user/my-repo /my-schema default True +dev.knative.source.github.push-34cnb dev.knative.source.github.push /my-other-user/my-other-repo default True +dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request /my-other-user/my-other-repo default True ``` `3.` The `Event Consumer` creates a Trigger to listen to an EventType in the Registry. From 6b852aca3b54e71035308ba3f5ac61b17c25cd78 Mon Sep 17 00:00:00 2001 From: Nacho Cano Date: Thu, 18 Apr 2019 12:03:53 -0700 Subject: [PATCH 10/20] typo --- docs/registry/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index c90b240651a..07daecbadbf 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -68,7 +68,7 @@ modify those names to make them K8s-compliant, whenever we need to generate them Given that `subject` was approved in the new CloudEvents spec, the CloudEvents `source` became more suitable for our purposes. Before, it contained dynamically generated information (e.g., `https://github.com///pull/`) that we didn't know beforehand. Now, with the introduction of `subject`, `source` was 're-purposed'. In the previous example -`source` would be `https://github.com///pull` and `subject` just contain the ``. +`source` would be `https://github.com///pull` and `subject` would just contain the ``. - `schema` is a URI with the EventType schema. It may be a JSON schema, a protobuf schema, etc. It is optional. From 78ee9c8e85246f28f5365548acc4d799f6deccb2 Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 23 Apr 2019 17:09:25 -0700 Subject: [PATCH 11/20] some updates --- docs/registry/README.md | 364 ++++++++++++++++++---------------------- 1 file changed, 163 insertions(+), 201 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index 07daecbadbf..aa44d94b162 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -3,7 +3,7 @@ ## Problem As an `Event Consumer` I want to be able to discover the different event types that I can consume -from the different Brokers, without resorting to any OOB mechanism. +from the different Brokers. This is also known as the `discoverability` use case, and is the main focus of this proposal. ## Objective @@ -12,7 +12,7 @@ Design an **initial** version of the **Registry** for the **MVP** that can suppo the different event types that can be consumed from the eventing mesh. For details on the different user stories that this proposal touches, please refer to the [User stories and personas for Knative eventing](https://docs.google.com/document/d/15uhyqQvaomxRX2u8s0i6CNhA86BQTNztkdsLUnPmvv4/edit?usp=sharing) document. -Note that this proposal targets the cases were the Broker/Trigger model is used. +Note that this proposal targets the cases where the Broker/Trigger model is used. #### Out of scope @@ -24,6 +24,8 @@ GitHub after our cluster has been configured (i.e., our GitHub CRD Source instal have been created, we will need to create new webhooks (and update the GitHub CRD Source) if we want to listen for those new events. Until doing so, those new events shouldn't be listed in the Registry. Such task will again be in charge of the `Cluster Configurator`. +- Auto-Registration of events at the Broker-level. This doesn't seem needed for an MVP, but we include a brief description +in this document as well. ## Requirements @@ -34,7 +36,7 @@ Our design revolves around the following core requirements: the eventing mesh in order to support the `discoverability` use case. If an event type is not ready for consumption, we should explicitly indicate so (e.g., if the Broker is not ready). -3. The event types stored in the Registry should contain all the required information +3. The event types stored in the Registry should contain (all) the required information for a consumer to create a Trigger without resorting to some other OOB mechanism. @@ -48,12 +50,13 @@ We propose introducing a namespaced-EventType CRD. Here is an example of how a C apiVersion: eventing.knative.dev/v1alpha1 kind: EventType metadata: - name: repopush + name: pullrequest namespace: default spec: - type: repo:push - source: /my-user/my-repo - schema: /my-schema + type: pull_request + source: github.com + schema: //github.com/schemas/pull_request + description: "GitHub pull request" broker: default ``` @@ -61,228 +64,182 @@ spec: contain characters that may not comply with Kubernetes naming conventions, we will (slightly) modify those names to make them K8s-compliant, whenever we need to generate them. -- `type` is authoritative. This refers to the CloudEvent type as it enters into the eventing mesh. +- `type`: is authoritative. This refers to the CloudEvent type as it enters into the eventing mesh. - `source`: is a valid URI. Refers to the CloudEvent source as it enters into the eventing mesh. -Given that `subject` was approved in the new CloudEvents spec, the CloudEvents `source` became more suitable for -our purposes. Before, it contained dynamically generated information (e.g., `https://github.com///pull/`) -that we didn't know beforehand. Now, with the introduction of `subject`, `source` was 're-purposed'. In the previous example -`source` would be `https://github.com///pull` and `subject` would just contain the ``. - - `schema` is a URI with the EventType schema. It may be a JSON schema, a protobuf schema, etc. It is optional. +- `description` is a string describing what the EventType is about. It is optional. + - `broker` refers to the Broker that can provide the EventType. ### Typical Flow -`1.` A `Cluster Configurator` configures the cluster in a way that allows the population of EventTypes in the Registry. -We foresee the following three ways of populating the Registry so far: - -`1.1` Event Source CR installation +1. A `Cluster Configurator` configures the cluster in a way that allows the population of EventTypes in the Registry. +We foresee the following two ways of populating the Registry for the MVP. A third one (Broker Auto-Registration) will be + explained in the following section but is most likely out of the scope of the Registry MVP. -Upon installation of an Event Source CR by a `Cluster Configurator`, the Source will register its EventTypes. + 1.1. Event Source CR installation -Example: - -```yaml -apiVersion: sources.eventing.knative.dev/v1alpha1 -kind: GitHubSource -metadata: - name: github-source-sample - namespace: default -spec: - eventTypes: - - push - - pull_request - ownerAndRepository: my-other-user/my-other-repo - accessToken: - secretKeyRef: - name: github-secret - key: accessToken - secretToken: - secretKeyRef: - name: github-secret - key: secretToken - sink: - apiVersion: eventing.knative.dev/v1alpha1 - kind: Broker - name: default + Upon installation of an Event Source CR by a `Cluster Configurator`, the Source will register its EventTypes. -``` - -By applying the above file, two EventTypes will be registered, with types `dev.knative.source.github.push` and -`dev.knative.source.github.pull_request`, source `/my-other-user/my-other-repo`, for the `default` Broker in the `default` - namespace, and with owner `github-source-sample`. This should be done by the Event Source controller, in this case, - the GitHubSource controller. + Example: + + ```yaml + apiVersion: sources.eventing.knative.dev/v1alpha1 + kind: GitHubSource + metadata: + name: github-source-sample + namespace: default + spec: + eventTypes: + - push + - pull_request + ownerAndRepository: my-other-user/my-other-repo + accessToken: + secretKeyRef: + name: github-secret + key: accessToken + secretToken: + secretKeyRef: + name: github-secret + key: secretToken + sink: + apiVersion: eventing.knative.dev/v1alpha1 + kind: Broker + name: default + ``` -Note that the `Cluster Configurator` is the person in charge of taking care of authentication-related matters. E.g., if a new `Event Consumer` -wants to listen for events from a different GitHub repo, the `Cluster Configurator` will take care of the necessary secrets generation, -and new Source instantiation. + By applying the above file, two EventTypes will be registered, with types `dev.knative.source.github.push` and + `dev.knative.source.github.pull_request`, source `github.com`, for the `default` Broker in the `default` + namespace, and with owner `github-source-sample`. This should be done by the Event Source controller, in this case, + the GitHubSource controller. Although not shown here, the controller could add some `description` to the EventTypes, e.g., + from which owner and repo the events are, etc. + + Note that the `Cluster Configurator` is the person in charge of taking care of authentication-related matters. E.g., if a new `Event Consumer` + wants to listen for events from a different GitHub repo, the `Cluster Configurator` will take care of the necessary secrets generation, + and new Source instantiation. + + In YAML, the above EventTypes would look something like these: + + ```yaml + apiVersion: eventing.knative.dev/v1alpha1 + kind: EventType + metadata: + generateName: dev.knative.source.github.push- + namespace: default + owner: # Owned by github-source-sample + spec: + type: dev.knative.source.github.push + source: github.com + broker: default + --- + apiVersion: eventing.knative.dev/v1alpha1 + kind: EventType + metadata: + generateName: dev.knative.source.github.pullrequest- + namespace: default + owner: # Owned by github-source-sample + spec: + type: dev.knative.source.github.pull_request + source: github.com + broker: default + ``` + + Two things to notice: + - We generate the names by stripping invalid characters from the original type (e.g., `_`) + - We add the prefix `dev.knative.source.github.` to `spec.type`. This is a **separate discussion** on whether we should + change the (GitHub) web hook types or not. -In YAML, the above EventTypes would look something like these: - -```yaml -apiVersion: eventing.knative.dev/v1alpha1 -kind: EventType -metadata: - generateName: dev.knative.source.github.push- - namespace: default - owner: # Owned by github-source-sample -spec: - type: dev.knative.source.github.push - source: /my-other-user/my-other-repo - broker: default ---- -apiVersion: eventing.knative.dev/v1alpha1 -kind: EventType -metadata: - generateName: dev.knative.source.github.pullrequest- - namespace: default - owner: # Owned by github-source-sample -spec: - type: dev.knative.source.github.pull_request - source: /my-other-user/my-other-repo - broker: default -``` - -Three things to notice: -- We generate the names by stripping invalid characters from the original type (e.g., `_`) -- We add the prefix `dev.knative.source.github.` to `spec.type`. This is a **separate discussion** on whether we should -change the (GitHub) types or not. -- We add the prefix `/` to `spec.source` to make it a valid URI. + 1.2. Manual Registration -`1.2.` Manual Registration - -The `Cluster Configurator` manually `kubectl applies` an EventType CR. + The `Cluster Configurator` manually `kubectl applies` an EventType CR. + + Example: + + ```yaml + apiVersion: eventing.knative.dev/v1alpha1 + kind: EventType + metadata: + name: repofork + namespace: default + spec: + type: repo:fork + source: bitbucket.org + broker: dev + description: "BitBucket fork" + ``` + + This would register the EventType named `repofork` with type `repo:fork`, source `bitbucket.org` in the `dev` + Broker of the `default` namespace. + + As under the hood, `kubeclt apply` just makes a REST call to the API server with the appropriate RBAC permissions, + the `Cluster Configurator` can give EventType `create` permissions to trusted parties, so that they can register + their EventTypes. -Example: +1. An `Event Consumer` checks the Registry to see what EventTypes it can consume from the mesh. -```yaml -apiVersion: eventing.knative.dev/v1alpha1 -kind: EventType -metadata: - name: repofork - namespace: default -spec: - type: repo:fork - source: /my-other-user/my-other-repo - broker: dev -``` + Example: + + `$ kubectl get eventtypes -n default` + + ``` + NAME TYPE SOURCE SCHEMA BROKER DESCRIPTION READY REASON + repofork repo:fork bitbucket.org dev BitBucket fork False BrokerIsNotReady + pullrequest pull_request github.com //github.com/schemas/pull_request default GitHub pull request True + dev.knative.source.github.push-34cnb dev.knative.source.github.push github.com default True + dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request github.com default True + ``` -This would register the EventType named `repofork` with type `repo:fork`, source `/my-other-user/my-other-repo` -in the `dev` Broker of the `default` namespace. +1. The `Event Consumer` creates a Trigger to listen to an EventType in the Registry. -As under the hood, `kubeclt apply` just makes a REST call to the API server with the appropriate RBAC permissions, -the `Cluster Configurator` can give EventType `create` permissions to trusted parties, so that they can register -their EventTypes. + Example: + + ```yaml + apiVersion: eventing.knative.dev/v1alpha1 + kind: Trigger + metadata: + name: my-service-trigger + namespace: default + spec: + filter: + sourceAndType: + type: dev.knative.source.github.push + source: github.com + subscriber: + ref: + apiVersion: serving.knative.dev/v1alpha1 + kind: Service + name: my-service + ``` -`1.3` Broker Auto Registration Policy +### Broker Ingress Policies -The `Cluster Configurator` configures the Broker ingress policy to allow auto-registration of EventTypes. -Upon arrival of a non-registered EventType to the Broker ingress, the Broker will then create that type. -Note that the creation of the EventType is done asynchronously, i.e., the CloudEvent is accepted and sent -to the appropriate Trigger(s) in parallel of the EventType creation. If the creation fails, on a subsequent arrival -there will be a new creation attempt. +Although this section is more related to the Broker rather than the Registry itself, we include it here to +showcase the interplay between the two. -Although sub-optimal (as `Event Consumers` find out about EventTypes upon first arrival), we believe this mechanism (or something similar) -is needed for Sources where the `Cluster Configurator` does not know in advance the EventTypes they can produce (e.g., a ContainerSource). +Herein, we propose configuring a policy in the Broker so that it can decide what EventTypes to accept into the mesh. +In order to support that, we propose adding an `ingressPolicy` field to the Broker's spec CRD. + +Below are two CR examples with Brokers configured using different policies. Note that the fields of +`ingressPolicy` might change (e.g., `ingressPolicy` might just end up being a string). +Also note that such configuration is done by the `Cluster Configurator`. -Example: +- Allow Any -Set up a Broker with `autoAdd` enabled. +By setting the ingress policy to allow any, the Broker will accept every event into the mesh. ```yaml apiVersion: eventing.knative.dev/v1alpha1 kind: Broker metadata: - name: auto-add-demo + name: broker-allow-any spec: ingressPolicy: - autoAdd: true -``` - -Now if someone emits a non-registered event (e.g., `dev.knative.foo.bar`) -into the Broker `auto-add-demo`, an EventType will be created upon the event arrival. - -We can send the event manually. While SSHed into a `Pod` with the Istio sidecar, run: - -```shell -curl -v "http://auto-add-demo-broker.default.svc.cluster.local/" \ - -X POST \ - -H "X-B3-Flags: 1" \ - -H "CE-CloudEventsVersion: 0.1" \ - -H "CE-EventType: dev.knative.foo.bar" \ - -H "CE-EventTime: 2018-04-05T03:56:24Z" \ - -H "CE-EventID: 45a8b444-3213-4758-be3f-540bf93f85ff" \ - -H "CE-Source: dev.knative.example" \ - -H 'Content-Type: application/json' \ - -d '{ "much": "wow" }' + allowAny: true ``` - -The Broker `auto-add-demo` will then create the EventType with type `dev.knative.foo.bar` in the Registry. -In YAML, the EventType would look something like these: - -```yaml -apiVersion: eventing.knative.dev/v1alpha1 -kind: EventType -metadata: - generateName: dev.knative.foo.bar- - namespace: default - owner: # Owned by auto-add-demo? -spec: - type: dev.knative.foo.bar - source: dev.knative.example - broker: auto-add-demo -``` - -`2.` An `Event Consumer` checks the Registry to see what EventTypes it can consume from the mesh. - -Example: - -`$ kubectl get eventtypes -n default` - -``` -NAME TYPE SOURCE SCHEMA BROKER READY REASON -dev.knative.foo.bar-55wcn dev.knative.foo.bar dev.knative.example auto-add-demo True -repofork repo:fork /my-other-user/my-other-repo dev False BrokerIsNotReady -repopush repo:push /my-user/my-repo /my-schema default True -dev.knative.source.github.push-34cnb dev.knative.source.github.push /my-other-user/my-other-repo default True -dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request /my-other-user/my-other-repo default True -``` - -`3.` The `Event Consumer` creates a Trigger to listen to an EventType in the Registry. - -Example: - -```yaml -apiVersion: eventing.knative.dev/v1alpha1 -kind: Trigger -metadata: - name: my-service-trigger - namespace: default -spec: - filter: - sourceAndType: - type: dev.knative.foo.bar - source: dev.knative.example - subscriber: - ref: - apiVersion: serving.knative.dev/v1alpha1 - kind: Service - name: my-service -``` - -### Broker Ingress Policies - -We briefly mentioned configuring an ingress policy in the Broker so that it can auto-register EventTypes. -In order to support that, we propose adding an `ingressPolicy` field to the Broker's spec CRD. - -Below are two CR examples with Brokers configured using different policies. Note that the fields of -`ingressPolicy` might change (e.g., `ingressPolicy` might just end up being a string). -We are omitting Broker's fields irrelevant to this discussion. -Also note that such configuration is done by the `Cluster Configurator`. - Allow Registered @@ -298,6 +255,11 @@ spec: allowAny: false ``` +By not specifying an ingress policy, we thought that the default behavior would be to allow any event, but this needs further discussion. + +Finally, we envision another policy that would allow to auto-register EventTypes. This is currently not being considered +for the MVP. + - Auto Add By setting the ingress policy to auto add, the Broker will accept any event and will add its EventType @@ -313,12 +275,12 @@ spec: autoAdd: true ``` -By not specifying an ingress policy, we thought that the default behavior would be to accept any event, which is similar to `Auto Add` -but without registration of events. This needs further discussion. +Note that the creation of the EventType can done asynchronously, i.e., the CloudEvent is accepted and sent +to the appropriate Trigger(s) in parallel of the EventType creation. If the creation fails, on a subsequent arrival +there might be a new creation attempt. -Note that more policies should probably need to be configured in the future, e.g., allow auto-registration of EventTypes received -from within the cluster (e.g., from a response of a Service running in the cluster) as opposed to external services, and so on. -We just enumerate two simple cases here. +Although sub-optimal (as `Event Consumers` find out about EventTypes upon first arrival), we believe this mechanism (or something similar) +might be needed for Sources where the `Cluster Configurator` does not know in advance the EventTypes they can produce (e.g., a ContainerSource). ## FAQ From dd69ba60f4e2391e80fc80e2c3e7bea117b95e1f Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 23 Apr 2019 17:13:01 -0700 Subject: [PATCH 12/20] adding Grant's example --- docs/registry/README.md | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index aa44d94b162..1cb2dfec49f 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -333,12 +333,34 @@ Event Sources so that the user doesn't need to query the CRDs to get the list. W - I wonder if the Event Source populating the Registry should happen when the Event Source is loaded into the system, meaning when the Event Source's CRD is installed (not when an instance of the CRD is created). -The problem with that is that you don't have a namespace (nor Broker, user/repo, etc.) at that point. -Which namespace the EventType should be created on? Pointing to which Broker? -Implementation-wise, one potential solution is to have a controller for source CRDs, whenever one is installed, search for all the namespaces with -eventing enabled (`kubectl get namespaces -l knative-eventing-injection=enabled`), and adding all the possible EventTypes from that CRD to each of -the Brokers in those namespaces. A downside of this is that the Registry information is not "accurate", in the sense that it only has info about EventTypes -that may potentially flow in the system. But actually, they will only be able to flow when a CR is created. - -- ... + The problem with that is that you don't have a namespace (nor Broker, user/repo, etc.) at that point. + Which namespace the EventType should be created on? Pointing to which Broker? + Implementation-wise, one potential solution is to have a controller for source CRDs, whenever one is installed, search for all the namespaces with + eventing enabled (`kubectl get namespaces -l knative-eventing-injection=enabled`), and adding all the possible EventTypes from that CRD to each of + the Brokers in those namespaces. A downside of this is that the Registry information is not "accurate", in the sense that it only has info about EventTypes + that may potentially flow in the system. But actually, they will only be able to flow when a CR is created. + +- How can I filter events by the CloudEvents `subject` field? + + The EventType CRD will not include a `subject` field because we expect the cardinality of `subject` to be quite high. + However, a user that would like to filter by a known subject value can do this in the Trigger with the + Advanced Filtering proposed in [#1047](https://github.com/knative/eventing/pull/1047). + + Example: + + ```yaml + apiVersion: eventing.knative.dev/v1alpha1 + kind: Trigger + metadata: + name: only-knative + spec: + filter: + cel: + expression: ce.subject.match("/knative/*") + subscriber: + ref: + apiVersion: serving.knative.dev/v1alpha1 + kind: Service + name: knative-events-processor +``` From 61c46dbed890fb06dd864524c6aa6de6338d02c0 Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 23 Apr 2019 17:28:15 -0700 Subject: [PATCH 13/20] cosmetics --- docs/registry/README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index 1cb2dfec49f..dfb35aa7c43 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -74,6 +74,9 @@ modify those names to make them K8s-compliant, whenever we need to generate them - `broker` refers to the Broker that can provide the EventType. +In order to *uniquely* identify an EventType, we would need to look at the tuple `(type, source, schema, broker)`, +as there might be EventTypes with the same `type` but different `sources`, or pointing to different `brokers`, and so on. + ### Typical Flow 1. A `Cluster Configurator` configures the cluster in a way that allows the population of EventTypes in the Registry. @@ -215,7 +218,7 @@ We foresee the following two ways of populating the Registry for the MVP. A thir ``` -### Broker Ingress Policies +## Broker Ingress Policies Although this section is more related to the Broker rather than the Registry itself, we include it here to showcase the interplay between the two. @@ -362,5 +365,4 @@ meaning when the Event Source's CRD is installed (not when an instance of the CR apiVersion: serving.knative.dev/v1alpha1 kind: Service name: knative-events-processor -``` - + ``` From ce49a62b8fedd275ea25d73e60d09d2bdc3c1596 Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 23 Apr 2019 17:29:19 -0700 Subject: [PATCH 14/20] cosmetic --- docs/registry/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index dfb35aa7c43..87a5a20a104 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -137,7 +137,7 @@ We foresee the following two ways of populating the Registry for the MVP. A thir type: dev.knative.source.github.push source: github.com broker: default - --- + --- apiVersion: eventing.knative.dev/v1alpha1 kind: EventType metadata: From b3a07fbc413f7938442e44a6354076d0e0029fe4 Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 23 Apr 2019 17:58:24 -0700 Subject: [PATCH 15/20] updating types --- docs/registry/README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index 87a5a20a104..dc3bf9018dc 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -50,10 +50,10 @@ We propose introducing a namespaced-EventType CRD. Here is an example of how a C apiVersion: eventing.knative.dev/v1alpha1 kind: EventType metadata: - name: pullrequest + name: com.github.pullrequest namespace: default spec: - type: pull_request + type: com.github.pull_request source: github.com schema: //github.com/schemas/pull_request description: "GitHub pull request" @@ -68,9 +68,9 @@ modify those names to make them K8s-compliant, whenever we need to generate them - `source`: is a valid URI. Refers to the CloudEvent source as it enters into the eventing mesh. -- `schema` is a URI with the EventType schema. It may be a JSON schema, a protobuf schema, etc. It is optional. +- `schema`: is a URI with the EventType schema. It may be a JSON schema, a protobuf schema, etc. It is optional. -- `description` is a string describing what the EventType is about. It is optional. +- `description`: is a string describing what the EventType is about. It is optional. - `broker` refers to the Broker that can provide the EventType. @@ -165,17 +165,17 @@ We foresee the following two ways of populating the Registry for the MVP. A thir apiVersion: eventing.knative.dev/v1alpha1 kind: EventType metadata: - name: repofork + name: org.bitbucket.repofork namespace: default spec: - type: repo:fork + type: org.bitbucket.repo:fork source: bitbucket.org broker: dev description: "BitBucket fork" ``` - This would register the EventType named `repofork` with type `repo:fork`, source `bitbucket.org` in the `dev` - Broker of the `default` namespace. + This would register the EventType named `org.bitbucket.repofork` with type `org.bitbucket.repo:fork`, + source `bitbucket.org` in the `dev` Broker of the `default` namespace. As under the hood, `kubeclt apply` just makes a REST call to the API server with the appropriate RBAC permissions, the `Cluster Configurator` can give EventType `create` permissions to trusted parties, so that they can register @@ -188,11 +188,11 @@ We foresee the following two ways of populating the Registry for the MVP. A thir `$ kubectl get eventtypes -n default` ``` - NAME TYPE SOURCE SCHEMA BROKER DESCRIPTION READY REASON - repofork repo:fork bitbucket.org dev BitBucket fork False BrokerIsNotReady - pullrequest pull_request github.com //github.com/schemas/pull_request default GitHub pull request True - dev.knative.source.github.push-34cnb dev.knative.source.github.push github.com default True - dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request github.com default True + NAME TYPE SOURCE SCHEMA BROKER DESCRIPTION READY REASON + org.bitbucket.repofork org.bitbucket.repo:fork bitbucket.org dev BitBucket fork False BrokerIsNotReady + com.github.pullrequest com.github.pull_request github.com //github.com/schemas/pull_request default GitHub pull request True + dev.knative.source.github.push-34cnb dev.knative.source.github.push github.com default True + dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request github.com default True ``` 1. The `Event Consumer` creates a Trigger to listen to an EventType in the Registry. From 4beac7deb439386d3cfc81ba037f57dd1ec30e3f Mon Sep 17 00:00:00 2001 From: nachocano Date: Tue, 23 Apr 2019 18:00:31 -0700 Subject: [PATCH 16/20] typo --- docs/registry/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index dc3bf9018dc..cc8e2ffcbb3 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -177,7 +177,7 @@ We foresee the following two ways of populating the Registry for the MVP. A thir This would register the EventType named `org.bitbucket.repofork` with type `org.bitbucket.repo:fork`, source `bitbucket.org` in the `dev` Broker of the `default` namespace. - As under the hood, `kubeclt apply` just makes a REST call to the API server with the appropriate RBAC permissions, + As under the hood, `kubectl apply` just makes a REST call to the API server with the appropriate RBAC permissions, the `Cluster Configurator` can give EventType `create` permissions to trusted parties, so that they can register their EventTypes. From a94b623d96095adc2e4617d595c32a3a1a8de6a7 Mon Sep 17 00:00:00 2001 From: nachocano Date: Fri, 26 Apr 2019 09:22:45 -0700 Subject: [PATCH 17/20] removing broker ingress policies --- docs/registry/README.md | 86 +++++------------------------------------ 1 file changed, 9 insertions(+), 77 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index cc8e2ffcbb3..e9f42f00da4 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -1,4 +1,4 @@ -# Registry Proposal +# Registry Design ## Problem @@ -24,8 +24,6 @@ GitHub after our cluster has been configured (i.e., our GitHub CRD Source instal have been created, we will need to create new webhooks (and update the GitHub CRD Source) if we want to listen for those new events. Until doing so, those new events shouldn't be listed in the Registry. Such task will again be in charge of the `Cluster Configurator`. -- Auto-Registration of events at the Broker-level. This doesn't seem needed for an MVP, but we include a brief description -in this document as well. ## Requirements @@ -40,11 +38,12 @@ is not ready). for a consumer to create a Trigger without resorting to some other OOB mechanism. -## Proposed Ideas +## Design Ideas ### EventType CRD -We propose introducing a namespaced-EventType CRD. Here is an example of how a CR would look like: +We propose introducing a namespaced-EventType Custom Resource Definition (CRD). +Here is an example of how a **Custom Object** (CO) would look like: ```yaml apiVersion: eventing.knative.dev/v1alpha1 @@ -80,8 +79,7 @@ as there might be EventTypes with the same `type` but different `sources`, or po ### Typical Flow 1. A `Cluster Configurator` configures the cluster in a way that allows the population of EventTypes in the Registry. -We foresee the following two ways of populating the Registry for the MVP. A third one (Broker Auto-Registration) will be - explained in the following section but is most likely out of the scope of the Registry MVP. +We foresee the following two ways of populating the Registry for the MVP: 1.1. Event Source CR installation @@ -151,7 +149,10 @@ We foresee the following two ways of populating the Registry for the MVP. A thir ``` Two things to notice: - - We generate the names by stripping invalid characters from the original type (e.g., `_`) + - We generate the names by stripping invalid characters from the original type (e.g., `_`). By generating names we aim to + avoid naming collisions. This is a **separate discussion** on whether we should generate them when code (in this case, + the GitHubSource controller) creates the EventTypes or not. + - We add the prefix `dev.knative.source.github.` to `spec.type`. This is a **separate discussion** on whether we should change the (GitHub) web hook types or not. @@ -217,74 +218,6 @@ We foresee the following two ways of populating the Registry for the MVP. A thir name: my-service ``` - -## Broker Ingress Policies - -Although this section is more related to the Broker rather than the Registry itself, we include it here to -showcase the interplay between the two. - -Herein, we propose configuring a policy in the Broker so that it can decide what EventTypes to accept into the mesh. -In order to support that, we propose adding an `ingressPolicy` field to the Broker's spec CRD. - -Below are two CR examples with Brokers configured using different policies. Note that the fields of -`ingressPolicy` might change (e.g., `ingressPolicy` might just end up being a string). -Also note that such configuration is done by the `Cluster Configurator`. - -- Allow Any - -By setting the ingress policy to allow any, the Broker will accept every event into the mesh. - -```yaml -apiVersion: eventing.knative.dev/v1alpha1 -kind: Broker -metadata: - name: broker-allow-any -spec: - ingressPolicy: - allowAny: true -``` - -- Allow Registered - -By setting the ingress policy to not allow any, the Broker will accept only events with EventTypes in the Registry. - -```yaml -apiVersion: eventing.knative.dev/v1alpha1 -kind: Broker -metadata: - name: broker-allow-registered -spec: - ingressPolicy: - allowAny: false -``` - -By not specifying an ingress policy, we thought that the default behavior would be to allow any event, but this needs further discussion. - -Finally, we envision another policy that would allow to auto-register EventTypes. This is currently not being considered -for the MVP. - -- Auto Add - -By setting the ingress policy to auto add, the Broker will accept any event and will add its EventType -to the Registry (in case it is not present). - -```yaml -apiVersion: eventing.knative.dev/v1alpha1 -kind: Broker -metadata: - name: broker-auto-add -spec: - ingressPolicy: - autoAdd: true -``` - -Note that the creation of the EventType can done asynchronously, i.e., the CloudEvent is accepted and sent -to the appropriate Trigger(s) in parallel of the EventType creation. If the creation fails, on a subsequent arrival -there might be a new creation attempt. - -Although sub-optimal (as `Event Consumers` find out about EventTypes upon first arrival), we believe this mechanism (or something similar) -might be needed for Sources where the `Cluster Configurator` does not know in advance the EventTypes they can produce (e.g., a ContainerSource). - ## FAQ Here is a list of frequently asked questions that may help clarify the scope of the Registry. @@ -302,7 +235,6 @@ is there a need/use for the Registry? then EventTypes won't be added to the Registry (at least in this proposal). Implementation-wise, we can check whether the Source's sink kind is `Broker`, and if so, then register its EventTypes. - - Is the Registry meant to be used in a single-user environment where the same person is setting up both the Event Source and the destination Sink? From 46a6b8704b9ec0cc94bbecd87cf97ca4a541c03f Mon Sep 17 00:00:00 2001 From: nachocano Date: Fri, 26 Apr 2019 09:34:07 -0700 Subject: [PATCH 18/20] changing CR to CO --- docs/registry/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index e9f42f00da4..588503c97a8 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -81,9 +81,9 @@ as there might be EventTypes with the same `type` but different `sources`, or po 1. A `Cluster Configurator` configures the cluster in a way that allows the population of EventTypes in the Registry. We foresee the following two ways of populating the Registry for the MVP: - 1.1. Event Source CR installation + 1.1. Event Source CO instantiation - Upon installation of an Event Source CR by a `Cluster Configurator`, the Source will register its EventTypes. + Upon instantiation of an Event Source CO by a `Cluster Configurator`, the Source will register its EventTypes. Example: @@ -247,7 +247,7 @@ the destination Sink? In other words, is a Registry always going to be there and if not under what conditions will it be? A Registry will always be there, i.e., `Event Consumers` will always be able to `kubectl get eventtypes -n `. - In case no CR Sources are pointing to Brokers, then the Registry will be empty. + In case no CO Sources are pointing to Brokers, then the Registry will be empty. - Once an Event Source is created, how is a new one created with different auth in an env where the user is really just meant to deal with Triggers? This may not be a Registry specific question but if one of the goals of the Registry is to make it @@ -273,7 +273,7 @@ meaning when the Event Source's CRD is installed (not when an instance of the CR Implementation-wise, one potential solution is to have a controller for source CRDs, whenever one is installed, search for all the namespaces with eventing enabled (`kubectl get namespaces -l knative-eventing-injection=enabled`), and adding all the possible EventTypes from that CRD to each of the Brokers in those namespaces. A downside of this is that the Registry information is not "accurate", in the sense that it only has info about EventTypes - that may potentially flow in the system. But actually, they will only be able to flow when a CR is created. + that may potentially flow in the system. But actually, they will only be able to flow when a CO is created. - How can I filter events by the CloudEvents `subject` field? From 83b8820ba8f207b4579d64aa85524754d91eacd4 Mon Sep 17 00:00:00 2001 From: nachocano Date: Fri, 3 May 2019 11:52:54 -0700 Subject: [PATCH 19/20] updating source values --- docs/registry/README.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index 588503c97a8..0252d5f4ae6 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -53,7 +53,7 @@ metadata: namespace: default spec: type: com.github.pull_request - source: github.com + source: //github.com/user/repo schema: //github.com/schemas/pull_request description: "GitHub pull request" broker: default @@ -113,7 +113,7 @@ We foresee the following two ways of populating the Registry for the MVP: ``` By applying the above file, two EventTypes will be registered, with types `dev.knative.source.github.push` and - `dev.knative.source.github.pull_request`, source `github.com`, for the `default` Broker in the `default` + `dev.knative.source.github.pull_request`, source `//github.com/my-other-user/my-other-repo`, for the `default` Broker in the `default` namespace, and with owner `github-source-sample`. This should be done by the Event Source controller, in this case, the GitHubSource controller. Although not shown here, the controller could add some `description` to the EventTypes, e.g., from which owner and repo the events are, etc. @@ -133,7 +133,7 @@ We foresee the following two ways of populating the Registry for the MVP: owner: # Owned by github-source-sample spec: type: dev.knative.source.github.push - source: github.com + source: //github.com/my-other-user/my-other-repo broker: default --- apiVersion: eventing.knative.dev/v1alpha1 @@ -144,7 +144,7 @@ We foresee the following two ways of populating the Registry for the MVP: owner: # Owned by github-source-sample spec: type: dev.knative.source.github.pull_request - source: github.com + source: //github.com/my-other-user/my-other-repo broker: default ``` @@ -170,13 +170,13 @@ We foresee the following two ways of populating the Registry for the MVP: namespace: default spec: type: org.bitbucket.repo:fork - source: bitbucket.org + source: //bitbucket.org/my-other-user/my-other-repo broker: dev description: "BitBucket fork" ``` This would register the EventType named `org.bitbucket.repofork` with type `org.bitbucket.repo:fork`, - source `bitbucket.org` in the `dev` Broker of the `default` namespace. + source `//bitbucket.org/my-other-user/my-other-repo` in the `dev` Broker of the `default` namespace. As under the hood, `kubectl apply` just makes a REST call to the API server with the appropriate RBAC permissions, the `Cluster Configurator` can give EventType `create` permissions to trusted parties, so that they can register @@ -189,11 +189,11 @@ We foresee the following two ways of populating the Registry for the MVP: `$ kubectl get eventtypes -n default` ``` - NAME TYPE SOURCE SCHEMA BROKER DESCRIPTION READY REASON - org.bitbucket.repofork org.bitbucket.repo:fork bitbucket.org dev BitBucket fork False BrokerIsNotReady - com.github.pullrequest com.github.pull_request github.com //github.com/schemas/pull_request default GitHub pull request True - dev.knative.source.github.push-34cnb dev.knative.source.github.push github.com default True - dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request github.com default True + NAME TYPE SOURCE SCHEMA BROKER DESCRIPTION READY REASON + org.bitbucket.repofork org.bitbucket.repo:fork //bitbucket.org/my-other-user/my-other-repo dev BitBucket fork False BrokerIsNotReady + com.github.pullrequest com.github.pull_request //github.com/user/repo //github.com/schemas/pull_request default GitHub pull request True + dev.knative.source.github.push-34cnb dev.knative.source.github.push //github.com/my-other-user/my-other-repo default True + dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request //github.com/my-other-user/my-other-repo default True ``` 1. The `Event Consumer` creates a Trigger to listen to an EventType in the Registry. @@ -210,7 +210,7 @@ We foresee the following two ways of populating the Registry for the MVP: filter: sourceAndType: type: dev.knative.source.github.push - source: github.com + source: //github.com/my-other-user/my-other-repo subscriber: ref: apiVersion: serving.knative.dev/v1alpha1 From 94a1891fa874dfc5cc4ca674bcb60b409b412b31 Mon Sep 17 00:00:00 2001 From: nachocano Date: Fri, 3 May 2019 15:30:33 -0700 Subject: [PATCH 20/20] adding the protocol to match the examples in CloudEvents spec --- docs/registry/README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/registry/README.md b/docs/registry/README.md index 0252d5f4ae6..17405c586ad 100644 --- a/docs/registry/README.md +++ b/docs/registry/README.md @@ -53,8 +53,8 @@ metadata: namespace: default spec: type: com.github.pull_request - source: //github.com/user/repo - schema: //github.com/schemas/pull_request + source: https://github.com/user/repo + schema: https://github.com/schemas/pull_request description: "GitHub pull request" broker: default ``` @@ -113,7 +113,7 @@ We foresee the following two ways of populating the Registry for the MVP: ``` By applying the above file, two EventTypes will be registered, with types `dev.knative.source.github.push` and - `dev.knative.source.github.pull_request`, source `//github.com/my-other-user/my-other-repo`, for the `default` Broker in the `default` + `dev.knative.source.github.pull_request`, source `https://github.com/my-other-user/my-other-repo`, for the `default` Broker in the `default` namespace, and with owner `github-source-sample`. This should be done by the Event Source controller, in this case, the GitHubSource controller. Although not shown here, the controller could add some `description` to the EventTypes, e.g., from which owner and repo the events are, etc. @@ -133,7 +133,7 @@ We foresee the following two ways of populating the Registry for the MVP: owner: # Owned by github-source-sample spec: type: dev.knative.source.github.push - source: //github.com/my-other-user/my-other-repo + source: https://github.com/my-other-user/my-other-repo broker: default --- apiVersion: eventing.knative.dev/v1alpha1 @@ -144,7 +144,7 @@ We foresee the following two ways of populating the Registry for the MVP: owner: # Owned by github-source-sample spec: type: dev.knative.source.github.pull_request - source: //github.com/my-other-user/my-other-repo + source: https://github.com/my-other-user/my-other-repo broker: default ``` @@ -170,13 +170,13 @@ We foresee the following two ways of populating the Registry for the MVP: namespace: default spec: type: org.bitbucket.repo:fork - source: //bitbucket.org/my-other-user/my-other-repo + source: https://bitbucket.org/my-other-user/my-other-repo broker: dev description: "BitBucket fork" ``` This would register the EventType named `org.bitbucket.repofork` with type `org.bitbucket.repo:fork`, - source `//bitbucket.org/my-other-user/my-other-repo` in the `dev` Broker of the `default` namespace. + source `https://bitbucket.org/my-other-user/my-other-repo` in the `dev` Broker of the `default` namespace. As under the hood, `kubectl apply` just makes a REST call to the API server with the appropriate RBAC permissions, the `Cluster Configurator` can give EventType `create` permissions to trusted parties, so that they can register @@ -189,11 +189,11 @@ We foresee the following two ways of populating the Registry for the MVP: `$ kubectl get eventtypes -n default` ``` - NAME TYPE SOURCE SCHEMA BROKER DESCRIPTION READY REASON - org.bitbucket.repofork org.bitbucket.repo:fork //bitbucket.org/my-other-user/my-other-repo dev BitBucket fork False BrokerIsNotReady - com.github.pullrequest com.github.pull_request //github.com/user/repo //github.com/schemas/pull_request default GitHub pull request True - dev.knative.source.github.push-34cnb dev.knative.source.github.push //github.com/my-other-user/my-other-repo default True - dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request //github.com/my-other-user/my-other-repo default True + NAME TYPE SOURCE SCHEMA BROKER DESCRIPTION READY REASON + org.bitbucket.repofork org.bitbucket.repo:fork https://bitbucket.org/my-other-user/my-other-repo dev BitBucket fork False BrokerIsNotReady + com.github.pullrequest com.github.pull_request https://github.com/user/repo https://github.com/schemas/pull_request default GitHub pull request True + dev.knative.source.github.push-34cnb dev.knative.source.github.push https://github.com/my-other-user/my-other-repo default True + dev.knative.source.github.pullrequest-86jhv dev.knative.source.github.pull_request https://github.com/my-other-user/my-other-repo default True ``` 1. The `Event Consumer` creates a Trigger to listen to an EventType in the Registry. @@ -210,7 +210,7 @@ We foresee the following two ways of populating the Registry for the MVP: filter: sourceAndType: type: dev.knative.source.github.push - source: //github.com/my-other-user/my-other-repo + source: https://github.com/my-other-user/my-other-repo subscriber: ref: apiVersion: serving.knative.dev/v1alpha1