From 0bc589eb47ac1a8bf33ea0d20e925ea7b5f1eb74 Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Wed, 22 Aug 2018 11:49:11 -0700 Subject: [PATCH 01/14] Add protobuf format This PR is meant for the spec as it is currently written, where the extensions is an explicit bag. The focus of this PR is on the protobuf format itself, not on the extensions proposal. Signed-off-by: Spencer Fang --- protobuf-format.md | 313 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 313 insertions(+) create mode 100644 protobuf-format.md diff --git a/protobuf-format.md b/protobuf-format.md new file mode 100644 index 000000000..37b1b3f8e --- /dev/null +++ b/protobuf-format.md @@ -0,0 +1,313 @@ +# Protobuf Event Format for CloudEvents - Version 0.1 + +## Abstract + +The Protocol Buffers (protobuf) Format for CloudEvents defines how events are to be expressed in protobuf version 3. + +## Status of this document + +This document is a working draft + +## 1. Introduction + +CloudEvents is a standardized and transport-neutral definition of the structure and metadata description of events. This specification defines how the elements defined in the CloudEvents specification are to be represented in the Protocol buffers version 3. + +Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data. A message is defined once using the protobuf interface description language (IDL), and the protobuf compiler generates the language specific libraries for serializing to and deserializing from the binary representation. The protobuf version 3 library can also convert a protobuf message into its [standard JSON form](PROTO_JSON). + +Some built in message types in protobuf 3 (`google.protobuf.Value`, `google.protobuf.Any`, etc) have special runtime support from the protobuf library. These built in messages are given special treatment when transcoding to and from JSON. Tools in the protobuf ecosystem also give special treatment to these messages. For example, the [Common Expression Language tool (CEL)](CEL) automatically converts a `google.protobuf.Any` into its contained message when performing data queries. + + +### 1.1. Conformance +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC2119. + +## 2. Protobuf format + +Users of protobuf MUST use the official message definition when expressing a CloudEvent as a protobuf message. Users SHOULD NOT change the language specific options in order to have consistent code generation output. Users MUST NOT change the protobuf package name. Users MUST NOT use any protobuf options that change the protobuf wire compatibility such as message_set_wire_format. + +```proto +syntax = "proto3"; + +package io.cloudevents.v0; + +import "google/protobuf/any.proto"; +import "google/protobuf/struct.proto"; +import "google/protobuf/timestamp.proto"; + +option go_package = "cloudevents.io/protobuf/"; +option java_package = "io.cloudevents"; +option java_multiple_files = true; + +message CloudEvent { + string event_type = 1; + string event_type_version = 2; + string cloud_events_version = 3; + string source = 4; + string event_id = 5; + google.protobuf.Timestamp timestamp = 6; + string schema_url = 7; + string content_type = 8; + // `oneof payload` itself is not a real field, it only enforces the `oneof` constraint + oneof payload { + google.protobuf.Value data = 9; + google.protobuf.BytesValue bytes_data = 10; + google.protobuf.Any proto_data = 11; + } + google.protobuf.Struct extensions = 12; +} +``` + +In general, the CloudEvents attribute names are converted into protobuf fields in lower_snake_case, and the the CloudEvents type system is mapped into protobuf types as follows: + + +| CloudEvents | Protobuf +|--------------|------------------------------------------------------------- +| String | string +| Binary | google.protobuf.BytesValue +| URI | string +| Timestamp | google.protobuf.Timestamp +| Map | google.protobuf.Struct +| Object | google.protobuf.Value +| Integer | int32 + +### 2.1 Note on cloud_events_version: + +The protobuf `package` keyword defines a package name that includes the CloudEvents major version number. The package name and message name together form a fully qualified name for a protobuf message. Messages with the same name but different package names are independent messages with no relation to one another. The `cloud_events_version` field’s major version number MUST match the major version of the protobuf package. + +### 2.2 Special handling of `data` attribute: + +#### 2.2.1 Background knowledge on special built in protobuf types + +`google.protobuf.Value` ([more info](PROTO_JSON)) is used to express a JSON structure. Its fields are composed of special wrapper value types such as `google.protobuf.BytesValue`. The protobuf runtime library provides JSON conversion support for these special built in message types. + +`google.protobuf.Any` ([more info](PROTO_ANY)) is used in protobuf to express a field whose type is not known to the enclosing message. It contains the binary representation of the embedded message and a URL string that identifies the type of the embedded message. + + +#### 2.2.2 Handling instructions + +If the `contentType` is `application/json` or any media type with the structured `+json` suffix, the implementation MUST store the JSON payload in the `data` field. + +If the payload is binary, the implementation MUST store the bytes in the `bytes_data` field. + +If the payload is a protobuf, the implementation MUST store the payload in the `proto_data` field and the `contentType` MUST be `application/protobuf`. + +### 2.3 Extensions: + +`google.protobuf.Struct` ([more info](PROTO_STRUCT)) represents an arbitrary JSON structure. The keys of the `Struct` are the names of the extensions. The values of the `google.protobuf.Struct` are `com.protobuf.Value` ([more info](PROTO_VALUE)) fields whose contents correspond to the value of the extension mapped into JSON using the [JSON mapping rules](CE_JSON_ENCODING). All extensions attributes MUST be put into this bag because they are not a part of the CloudEvents spec. + +Well known extensions can not be given a top level field because they have no official standing, therefore the protobuf definition can not commit to a strongly typed representation of the value. The protobuf binding must be able to map an extension name to different value types, e.g. a "exampleExtension" with CE type "Object" in one message and another "exampleExtension" with CE type "Integer" in another message. + +When a new CloudEvent spec version is released, promoted extensions move from the extensions bag to a strongly typed top level field. + +For example, consider an extension for CloudEvents 1.0: + +```java +CloudEvent.newBuilder() + .setCloudEventsVersion("1.0") + .setExtensions( + Struct.newBuilder() + .putFields( + "comExampleMyextension", + Value.newBuilder().setStringValue("myvalue").build())) + .build(); +``` + +In the next CloudEvents minor release, the extension is promoted to an official field. The protobuf message definition is updated for CloudEvents 1.1: + +```java +CloudEvent.newBuilder() + .setCloudEventsVersion("1.1") + .setComExampleMyextension("myvalue") + // During the upgrade process producers SHOULD continue setting exts bag to maintain backwards compat + .setExtensions( + Struct.newBuilder() + .putFields( + "comExampleMyextension", + Value.newBuilder().setStringValue("myvalue").build())) + .build(); +``` + +#### 2.3.1 CloudEvents upgrade process + +If a field moves from the extensions bag to a top level field, then the producers and consumers of a CloudEvents system SHOULD coordinate in the upgrade process: + +1. Initially, the producers and consumers are using CloudEvents 1.0 and the extension is expressed in the “extensions” bag. +1. CloudEvents 1.1 is released. +1. The producers write the extension to both the extensions bag and the type safe top level field. The messages will have "cloudEventsVersion" set to "1.1", but the extension is still readable by 1.0 consumers. +1. All consumers upgrade to 1.1 and stop reading from the extensions bag and switch to reading from only the type safe top level field. +1. The producers stop writing to the extensions bag, and only write to the type safe top level field. + + +### 2.4 Relation to CloudEvents JSON format: + +All proto3 messages have a standard JSON form. The standard JSON of this protobuf format is not compatible from the official [CloudEvents JSON encoding](CE_JSON_ENCODING) at the time of writing. A service that supports the standard protobuf JSON encoding as well as the official CloudEvents JSON encoding will end up supporting two different JSON encodings. + +Below are a few examples of the proto3 JSON. + +A CloudEvent whose payload is a JSON may be constructed as follows: +```java +CloudEvent ce1 = CloudEvent.newBuilder() + .setEventType("com.example.usercreated") + .setEventTypeVersion("1.0") + .setCloudEventsVersion("0.1") + .setSource("producer1") + .setEventId("100") + .setTimestamp(ts) + .setContentType("application/json") + .setSchemaUrl("https://com.example.schema/usercreated") + .setData( + Value.newBuilder().setStructValue(Struct.newBuilder() + .putFields( + "username", + Value.newBuilder().setStringValue("theusername").build()) + .putFields( + "email", + Value.newBuilder().setStringValue("user@example.com").build()) + .build())) + .setExtensions( + Struct.newBuilder() + .putFields( + "comExampleExtension1", + Value.newBuilder().setStringValue("value1").build()) + .putFields( + "comExampleExtension2", + Value.newBuilder().setStringValue("value2").build())) + .build(); +``` + +It has the following proto3 standard JSON representation: +```json +{ + "eventType": "com.example.usercreated", + "eventTypeVersion": "1.0", + "cloudEventsVersion": "0.1", + "source": "producer1", + "eventId": "100", + "timestamp": "2018-08-22T15:47:00.951Z", + "schemaUrl": "https://com.example.schema/usercreated", + "contentType": "application/json", + "data": { + "username": "theusername", + "email": "user@example.com" + }, + "extensions": { + "comExampleExtension1": "value1", + "comExampleExtension2": "value2" + } +} +``` + +A CloudEvent whose data payload is bytes may be constructed as follows: +```java + Timestamp ts = Timestamp.newBuilder().setSeconds(millis / 1000) + .setNanos((int) ((millis % 1000) * 1000000)).build(); + CloudEvent ce1 = CloudEvent.newBuilder() + .setEventType("com.example.octetstream") + .setEventTypeVersion("1.0") + .setCloudEventsVersion("0.1") + .setSource("producer1") + .setEventId("100") + .setTimestamp(ts) + .setContentType("application/octet-stream") + .setBytesData( + BytesValue.newBuilder().setValue(ByteString.copyFrom(new byte[] {1, 2, 3, 4}))) + .setExtensions( + Struct.newBuilder() + .putFields( + "comExampleExtension1", + Value.newBuilder().setStringValue("value1").build()) + .putFields( + "comExampleExtension2", + Value.newBuilder().setStringValue("value2").build())) + .build(); +``` + +It has the following proto3 standard JSON representation. +```json +{ + "eventType": "com.example.octetstreamservice", + "eventTypeVersion": "1.0", + "cloudEventsVersion": "0.1", + "source": "producer1", + "eventId": "100", + "timestamp": "2018-08-22T16:04:45.017Z", + "contentType": "application/protobuf", + "bytesData": "AQIDBA==", + "extensions": { + "comExampleExtension1": "value1", + "comExampleExtension2": "value2" + } +} +``` + +A CloudEvent whose data payload is a proto message can be constructed as follows: + +The proto payload’s message definition: +```proto +package com.example; + +message CustomMessage { + string contents = 1; +} +``` + +It has the following proto3 standard JSON representation. Note that the binary data is put in the `protoData` field rather than `data`: + +```java +CloudEvent.newBuilder() + .setEventType("com.example.readablemessagesservice") + .setEventTypeVersion("1.0") + .setCloudEventsVersion("0.1") + .setSource("producer1") + .setEventId("100") + .setTimestamp(ts) + .setContentType("application/protobuf") + .setProtoData( + Any.pack(CustomMessage.newBuilder().setContents("helloworld").build())) + .setExtensions( + Struct.newBuilder() + .putFields( + "comExampleExtension1", + Value.newBuilder().setStringValue("value1").build()) + .putFields( + "comExampleExtension2", + Value.newBuilder().setStringValue("value2").build())) + .build(); +``` + +It has the following proto3 standard JSON representation. +```json +{ + "eventType": "com.example.readablemessages", + "eventTypeVersion": "1.0", + "cloudEventsVersion": "0.1", + "source": "producer1", + "eventId": "100", + "timestamp": "2018-08-22T16:15:07.386Z", + "contentType": "application/protobuf", + "protoData": { + "@type": "type.googleapis.com/com.example.CustomMessage", + "contents": "helloworld" + }, + "extensions": { + "comExampleExtension1": "value1", + "comExampleExtension2": "value2" + } +} +``` + +## 3. References: + +* [google.protobuf.Any][PROTO_ANY] +* [google.protobuf.Struct][PROTO_STRUCT] +* [google.protobuf.Value][PROTO_VALUE] +* [Protobuf and JSON][PROTO_JSON] +* [Common Expression Language][CEL] + + +[PROTO_ANY]: https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#any +[PROTO_STRUCT]: https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#struct +[PROTO_VALUE]: https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Value +[PROTO_JSON]: https://developers.google.com/protocol-buffers/docs/proto3#json +[CEL]: https://github.com/google/cel-spec/blob/master/doc/langdef.md +[CE_JSON_ENCODING]: ./json-format.md +[CE_SPEC]: ./spec.md From b3fd58cd7e0e77911c647f326aa05dc936e01e52 Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Wed, 22 Aug 2018 12:04:59 -0700 Subject: [PATCH 02/14] Avoid redundant BytesWrapper by using bytes directly. Signed-off-by: Spencer Fang --- protobuf-format.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/protobuf-format.md b/protobuf-format.md index 37b1b3f8e..eefd10ba0 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -49,7 +49,7 @@ message CloudEvent { // `oneof payload` itself is not a real field, it only enforces the `oneof` constraint oneof payload { google.protobuf.Value data = 9; - google.protobuf.BytesValue bytes_data = 10; + bytes bytes_data = 10; google.protobuf.Any proto_data = 11; } google.protobuf.Struct extensions = 12; @@ -62,7 +62,7 @@ In general, the CloudEvents attribute names are converted into protobuf fields i | CloudEvents | Protobuf |--------------|------------------------------------------------------------- | String | string -| Binary | google.protobuf.BytesValue +| Binary | bytes | URI | string | Timestamp | google.protobuf.Timestamp | Map | google.protobuf.Struct @@ -208,8 +208,7 @@ A CloudEvent whose data payload is bytes may be constructed as follows: .setEventId("100") .setTimestamp(ts) .setContentType("application/octet-stream") - .setBytesData( - BytesValue.newBuilder().setValue(ByteString.copyFrom(new byte[] {1, 2, 3, 4}))) + .setBytesData(ByteString.copyFrom(new byte[] {1, 2, 3, 4})) .setExtensions( Struct.newBuilder() .putFields( From 7e1880d6eb29c149181a09696a52ae6f137f5cee Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Wed, 22 Aug 2018 12:21:26 -0700 Subject: [PATCH 03/14] Fix markdown links Signed-off-by: Spencer Fang --- protobuf-format.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/protobuf-format.md b/protobuf-format.md index eefd10ba0..c7af7e2f4 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -12,9 +12,9 @@ This document is a working draft CloudEvents is a standardized and transport-neutral definition of the structure and metadata description of events. This specification defines how the elements defined in the CloudEvents specification are to be represented in the Protocol buffers version 3. -Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data. A message is defined once using the protobuf interface description language (IDL), and the protobuf compiler generates the language specific libraries for serializing to and deserializing from the binary representation. The protobuf version 3 library can also convert a protobuf message into its [standard JSON form](PROTO_JSON). +Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data. A message is defined once using the protobuf interface description language (IDL), and the protobuf compiler generates the language specific libraries for serializing to and deserializing from the binary representation. The protobuf version 3 library can also convert a protobuf message into its [standard JSON form][PROTO_JSON]. -Some built in message types in protobuf 3 (`google.protobuf.Value`, `google.protobuf.Any`, etc) have special runtime support from the protobuf library. These built in messages are given special treatment when transcoding to and from JSON. Tools in the protobuf ecosystem also give special treatment to these messages. For example, the [Common Expression Language tool (CEL)](CEL) automatically converts a `google.protobuf.Any` into its contained message when performing data queries. +Some built in message types in protobuf 3 (`google.protobuf.Value`, `google.protobuf.Any`, etc) have special runtime support from the protobuf library. These built in messages are given special treatment when transcoding to and from JSON. Tools in the protobuf ecosystem also give special treatment to these messages. For example, the [Common Expression Language tool (CEL)][CEL] automatically converts a `google.protobuf.Any` into its contained message when performing data queries. ### 1.1. Conformance @@ -77,9 +77,9 @@ The protobuf `package` keyword defines a package name that includes the CloudEve #### 2.2.1 Background knowledge on special built in protobuf types -`google.protobuf.Value` ([more info](PROTO_JSON)) is used to express a JSON structure. Its fields are composed of special wrapper value types such as `google.protobuf.BytesValue`. The protobuf runtime library provides JSON conversion support for these special built in message types. +`google.protobuf.Value` ([more info][PROTO_JSON]) is used to express a JSON structure. Its fields are composed of special wrapper value types such as `google.protobuf.BytesValue`. The protobuf runtime library provides JSON conversion support for these special built in message types. -`google.protobuf.Any` ([more info](PROTO_ANY)) is used in protobuf to express a field whose type is not known to the enclosing message. It contains the binary representation of the embedded message and a URL string that identifies the type of the embedded message. +`google.protobuf.Any` ([more info][PROTO_ANY]) is used in protobuf to express a field whose type is not known to the enclosing message. It contains the binary representation of the embedded message and a URL string that identifies the type of the embedded message. #### 2.2.2 Handling instructions @@ -92,7 +92,7 @@ If the payload is a protobuf, the implementation MUST store the payload in the ` ### 2.3 Extensions: -`google.protobuf.Struct` ([more info](PROTO_STRUCT)) represents an arbitrary JSON structure. The keys of the `Struct` are the names of the extensions. The values of the `google.protobuf.Struct` are `com.protobuf.Value` ([more info](PROTO_VALUE)) fields whose contents correspond to the value of the extension mapped into JSON using the [JSON mapping rules](CE_JSON_ENCODING). All extensions attributes MUST be put into this bag because they are not a part of the CloudEvents spec. +`google.protobuf.Struct` ([more info][PROTO_STRUCT]) represents an arbitrary JSON structure. The keys of the `Struct` are the names of the extensions. The values of the `google.protobuf.Struct` are `com.protobuf.Value` ([more info][PROTO_VALUE]) fields whose contents correspond to the value of the extension mapped into JSON using the [JSON mapping rules][CE_JSON_ENCODING]. All extensions attributes MUST be put into this bag because they are not a part of the CloudEvents spec. Well known extensions can not be given a top level field because they have no official standing, therefore the protobuf definition can not commit to a strongly typed representation of the value. The protobuf binding must be able to map an extension name to different value types, e.g. a "exampleExtension" with CE type "Object" in one message and another "exampleExtension" with CE type "Integer" in another message. @@ -139,7 +139,7 @@ If a field moves from the extensions bag to a top level field, then the producer ### 2.4 Relation to CloudEvents JSON format: -All proto3 messages have a standard JSON form. The standard JSON of this protobuf format is not compatible from the official [CloudEvents JSON encoding](CE_JSON_ENCODING) at the time of writing. A service that supports the standard protobuf JSON encoding as well as the official CloudEvents JSON encoding will end up supporting two different JSON encodings. +All proto3 messages have a standard JSON form. The standard JSON of this protobuf format is not compatible from the official [CloudEvents JSON encoding][CE_JSON_ENCODING] at the time of writing. A service that supports the standard protobuf JSON encoding as well as the official CloudEvents JSON encoding will end up supporting two different JSON encodings. Below are a few examples of the proto3 JSON. From 23fc68a54382e2441b9ee4f502475d841fad5d0b Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Wed, 22 Aug 2018 16:10:33 -0700 Subject: [PATCH 04/14] Reformat to be 80 columns. Signed-off-by: Spencer Fang --- protobuf-format.md | 297 ++++++++++++++++++++++++++++----------------- 1 file changed, 186 insertions(+), 111 deletions(-) diff --git a/protobuf-format.md b/protobuf-format.md index c7af7e2f4..438718d5c 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -2,7 +2,8 @@ ## Abstract -The Protocol Buffers (protobuf) Format for CloudEvents defines how events are to be expressed in protobuf version 3. +The Protocol Buffers (protobuf) Format for CloudEvents defines how +events are to be expressed in protobuf version 3. ## Status of this document @@ -10,19 +11,42 @@ This document is a working draft ## 1. Introduction -CloudEvents is a standardized and transport-neutral definition of the structure and metadata description of events. This specification defines how the elements defined in the CloudEvents specification are to be represented in the Protocol buffers version 3. - -Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data. A message is defined once using the protobuf interface description language (IDL), and the protobuf compiler generates the language specific libraries for serializing to and deserializing from the binary representation. The protobuf version 3 library can also convert a protobuf message into its [standard JSON form][PROTO_JSON]. - -Some built in message types in protobuf 3 (`google.protobuf.Value`, `google.protobuf.Any`, etc) have special runtime support from the protobuf library. These built in messages are given special treatment when transcoding to and from JSON. Tools in the protobuf ecosystem also give special treatment to these messages. For example, the [Common Expression Language tool (CEL)][CEL] automatically converts a `google.protobuf.Any` into its contained message when performing data queries. +CloudEvents is a standardized and transport-neutral definition of the +structure and metadata description of events. This specification +defines how the elements defined in the CloudEvents specification are +to be represented in the Protocol buffers version 3. + +Protocol buffers are a language-neutral, platform-neutral extensible +mechanism for serializing structured data. A message is defined once +using the protobuf interface description language (IDL), and the +protobuf compiler generates the language specific libraries for +serializing to and deserializing from the binary representation. The +protobuf version 3 library can also convert a protobuf message into +its [standard JSON form][PROTO_JSON]. + +Some built in message types in protobuf 3 (`google.protobuf.Value`, +`google.protobuf.Any`, etc) have special runtime support from the +protobuf library. These built in messages are given special treatment +when transcoding to and from JSON. Tools in the protobuf ecosystem +also give special treatment to these messages. For example, the +[Common Expression Language tool (CEL)][CEL] automatically converts a +`google.protobuf.Any` into its contained message when performing data +queries. ### 1.1. Conformance -The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC2119. +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", +"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this +document are to be interpreted as described in RFC2119. ## 2. Protobuf format -Users of protobuf MUST use the official message definition when expressing a CloudEvent as a protobuf message. Users SHOULD NOT change the language specific options in order to have consistent code generation output. Users MUST NOT change the protobuf package name. Users MUST NOT use any protobuf options that change the protobuf wire compatibility such as message_set_wire_format. +Users of protobuf MUST use the official message definition when +expressing a CloudEvent as a protobuf message. Users SHOULD NOT change +the language specific options in order to have consistent code +generation output. Users MUST NOT change the protobuf package +name. Users MUST NOT use any protobuf options that change the protobuf +wire compatibility such as message_set_wire_format. ```proto syntax = "proto3"; @@ -46,7 +70,8 @@ message CloudEvent { google.protobuf.Timestamp timestamp = 6; string schema_url = 7; string content_type = 8; - // `oneof payload` itself is not a real field, it only enforces the `oneof` constraint + // `oneof payload` itself is not a real field, it only enforces the + // `oneof` constraint oneof payload { google.protobuf.Value data = 9; bytes bytes_data = 10; @@ -56,7 +81,9 @@ message CloudEvent { } ``` -In general, the CloudEvents attribute names are converted into protobuf fields in lower_snake_case, and the the CloudEvents type system is mapped into protobuf types as follows: +In general, the CloudEvents attribute names are converted into +protobuf fields in lower_snake_case, and the the CloudEvents type +system is mapped into protobuf types as follows: | CloudEvents | Protobuf @@ -71,107 +98,155 @@ In general, the CloudEvents attribute names are converted into protobuf fields i ### 2.1 Note on cloud_events_version: -The protobuf `package` keyword defines a package name that includes the CloudEvents major version number. The package name and message name together form a fully qualified name for a protobuf message. Messages with the same name but different package names are independent messages with no relation to one another. The `cloud_events_version` field’s major version number MUST match the major version of the protobuf package. +The protobuf `package` keyword defines a package name that includes +the CloudEvents major version number. The package name and message +name together form a fully qualified name for a protobuf +message. Messages with the same name but different package names are +independent messages with no relation to one another. The +`cloud_events_version` field’s major version number MUST match the +major version of the protobuf package. ### 2.2 Special handling of `data` attribute: #### 2.2.1 Background knowledge on special built in protobuf types -`google.protobuf.Value` ([more info][PROTO_JSON]) is used to express a JSON structure. Its fields are composed of special wrapper value types such as `google.protobuf.BytesValue`. The protobuf runtime library provides JSON conversion support for these special built in message types. +`google.protobuf.Value` ([more info][PROTO_JSON]) is used to express a +JSON structure. Its fields are composed of special wrapper value types +such as `google.protobuf.BytesValue`. The protobuf runtime library +provides JSON conversion support for these special built in message +types. -`google.protobuf.Any` ([more info][PROTO_ANY]) is used in protobuf to express a field whose type is not known to the enclosing message. It contains the binary representation of the embedded message and a URL string that identifies the type of the embedded message. +`google.protobuf.Any` ([more info][PROTO_ANY]) is used in protobuf to +express a field whose type is not known to the enclosing message. It +contains the binary representation of the embedded message and a URL +string that identifies the type of the embedded message. #### 2.2.2 Handling instructions -If the `contentType` is `application/json` or any media type with the structured `+json` suffix, the implementation MUST store the JSON payload in the `data` field. +If the `contentType` is `application/json` or any media type with the +structured `+json` suffix, the implementation MUST store the JSON +payload in the `data` field. -If the payload is binary, the implementation MUST store the bytes in the `bytes_data` field. +If the payload is binary, the implementation MUST store the bytes in +the `bytes_data` field. -If the payload is a protobuf, the implementation MUST store the payload in the `proto_data` field and the `contentType` MUST be `application/protobuf`. +If the payload is a protobuf, the implementation MUST store the +payload in the `proto_data` field and the `contentType` MUST be +`application/protobuf`. ### 2.3 Extensions: -`google.protobuf.Struct` ([more info][PROTO_STRUCT]) represents an arbitrary JSON structure. The keys of the `Struct` are the names of the extensions. The values of the `google.protobuf.Struct` are `com.protobuf.Value` ([more info][PROTO_VALUE]) fields whose contents correspond to the value of the extension mapped into JSON using the [JSON mapping rules][CE_JSON_ENCODING]. All extensions attributes MUST be put into this bag because they are not a part of the CloudEvents spec. - -Well known extensions can not be given a top level field because they have no official standing, therefore the protobuf definition can not commit to a strongly typed representation of the value. The protobuf binding must be able to map an extension name to different value types, e.g. a "exampleExtension" with CE type "Object" in one message and another "exampleExtension" with CE type "Integer" in another message. - -When a new CloudEvent spec version is released, promoted extensions move from the extensions bag to a strongly typed top level field. +`google.protobuf.Struct` ([more info][PROTO_STRUCT]) represents an +arbitrary JSON structure. The keys of the `Struct` are the names of +the extensions. The values of the `google.protobuf.Struct` are +`com.protobuf.Value` ([more info][PROTO_VALUE]) fields whose contents +correspond to the value of the extension mapped into JSON using the +[JSON mapping rules][CE_JSON_ENCODING]. All extensions attributes MUST +be put into this bag because they are not a part of the CloudEvents +spec. + +Well known extensions can not be given a top level field because they +have no official standing, therefore the protobuf definition can not +commit to a strongly typed representation of the value. The protobuf +binding must be able to map an extension name to different value +types, e.g. a "exampleExtension" with CE type "Object" in one message +and another "exampleExtension" with CE type "Integer" in another +message. + +When a new CloudEvent spec version is released, promoted extensions +move from the extensions bag to a strongly typed top level field. For example, consider an extension for CloudEvents 1.0: ```java CloudEvent.newBuilder() - .setCloudEventsVersion("1.0") - .setExtensions( - Struct.newBuilder() - .putFields( - "comExampleMyextension", - Value.newBuilder().setStringValue("myvalue").build())) - .build(); + .setCloudEventsVersion("1.0") + .setExtensions( + Struct.newBuilder() + .putFields( + "comExampleMyextension", + Value.newBuilder().setStringValue("myvalue").build())) + .build(); ``` -In the next CloudEvents minor release, the extension is promoted to an official field. The protobuf message definition is updated for CloudEvents 1.1: +In the next CloudEvents minor release, the extension is promoted to an +official field. The protobuf message definition is updated for +CloudEvents 1.1: ```java CloudEvent.newBuilder() - .setCloudEventsVersion("1.1") - .setComExampleMyextension("myvalue") - // During the upgrade process producers SHOULD continue setting exts bag to maintain backwards compat - .setExtensions( - Struct.newBuilder() - .putFields( - "comExampleMyextension", - Value.newBuilder().setStringValue("myvalue").build())) - .build(); + .setCloudEventsVersion("1.1") + .setComExampleMyextension("myvalue") + // During the upgrade process producers SHOULD continue setting + // exts bag to maintain backwards compat + .setExtensions( + Struct.newBuilder() + .putFields( + "comExampleMyextension", + Value.newBuilder().setStringValue("myvalue").build())) + .build(); ``` #### 2.3.1 CloudEvents upgrade process -If a field moves from the extensions bag to a top level field, then the producers and consumers of a CloudEvents system SHOULD coordinate in the upgrade process: +If a field moves from the extensions bag to a top level field, then +the producers and consumers of a CloudEvents system SHOULD coordinate +in the upgrade process: -1. Initially, the producers and consumers are using CloudEvents 1.0 and the extension is expressed in the “extensions” bag. +1. Initially, the producers and consumers are using CloudEvents 1.0 + and the extension is expressed in the “extensions” bag. 1. CloudEvents 1.1 is released. -1. The producers write the extension to both the extensions bag and the type safe top level field. The messages will have "cloudEventsVersion" set to "1.1", but the extension is still readable by 1.0 consumers. -1. All consumers upgrade to 1.1 and stop reading from the extensions bag and switch to reading from only the type safe top level field. -1. The producers stop writing to the extensions bag, and only write to the type safe top level field. +1. The producers write the extension to both the extensions bag and + the type safe top level field. The messages will have + "cloudEventsVersion" set to "1.1", but the extension is still + readable by 1.0 consumers. +1. All consumers upgrade to 1.1 and stop reading from the extensions + bag and switch to reading from only the type safe top level field. +1. The producers stop writing to the extensions bag, and only write to + the type safe top level field. ### 2.4 Relation to CloudEvents JSON format: -All proto3 messages have a standard JSON form. The standard JSON of this protobuf format is not compatible from the official [CloudEvents JSON encoding][CE_JSON_ENCODING] at the time of writing. A service that supports the standard protobuf JSON encoding as well as the official CloudEvents JSON encoding will end up supporting two different JSON encodings. +All proto3 messages have a standard JSON form. The standard JSON of +this protobuf format is not compatible from the official [CloudEvents +JSON encoding][CE_JSON_ENCODING] at the time of writing. A service +that supports the standard protobuf JSON encoding as well as the +official CloudEvents JSON encoding will end up supporting two +different JSON encodings. Below are a few examples of the proto3 JSON. A CloudEvent whose payload is a JSON may be constructed as follows: ```java -CloudEvent ce1 = CloudEvent.newBuilder() - .setEventType("com.example.usercreated") - .setEventTypeVersion("1.0") - .setCloudEventsVersion("0.1") - .setSource("producer1") - .setEventId("100") - .setTimestamp(ts) - .setContentType("application/json") - .setSchemaUrl("https://com.example.schema/usercreated") - .setData( - Value.newBuilder().setStructValue(Struct.newBuilder() - .putFields( - "username", - Value.newBuilder().setStringValue("theusername").build()) - .putFields( - "email", - Value.newBuilder().setStringValue("user@example.com").build()) - .build())) - .setExtensions( - Struct.newBuilder() - .putFields( - "comExampleExtension1", - Value.newBuilder().setStringValue("value1").build()) - .putFields( - "comExampleExtension2", - Value.newBuilder().setStringValue("value2").build())) - .build(); +CloudEvent.newBuilder() + .setEventType("com.example.usercreated") + .setEventTypeVersion("1.0") + .setCloudEventsVersion("0.1") + .setSource("producer1") + .setEventId("100") + .setTimestamp(ts) + .setContentType("application/json") + .setSchemaUrl("https://com.example.schema/usercreated") + .setData( + Value.newBuilder().setStructValue(Struct.newBuilder() + .putFields( + "username", + Value.newBuilder().setStringValue("theusername").build()) + .putFields( + "email", + Value.newBuilder().setStringValue("user@example.com").build()) + .build())) + .setExtensions( + Struct.newBuilder() + .putFields( + "comExampleExtension1", + Value.newBuilder().setStringValue("value1").build()) + .putFields( + "comExampleExtension2", + Value.newBuilder().setStringValue("value2").build())) + .build(); ``` It has the following proto3 standard JSON representation: @@ -198,26 +273,24 @@ It has the following proto3 standard JSON representation: A CloudEvent whose data payload is bytes may be constructed as follows: ```java - Timestamp ts = Timestamp.newBuilder().setSeconds(millis / 1000) - .setNanos((int) ((millis % 1000) * 1000000)).build(); - CloudEvent ce1 = CloudEvent.newBuilder() - .setEventType("com.example.octetstream") - .setEventTypeVersion("1.0") - .setCloudEventsVersion("0.1") - .setSource("producer1") - .setEventId("100") - .setTimestamp(ts) - .setContentType("application/octet-stream") - .setBytesData(ByteString.copyFrom(new byte[] {1, 2, 3, 4})) - .setExtensions( - Struct.newBuilder() - .putFields( - "comExampleExtension1", - Value.newBuilder().setStringValue("value1").build()) - .putFields( - "comExampleExtension2", - Value.newBuilder().setStringValue("value2").build())) - .build(); +CloudEvent.newBuilder() + .setEventType("com.example.octetstream") + .setEventTypeVersion("1.0") + .setCloudEventsVersion("0.1") + .setSource("producer1") + .setEventId("100") + .setTimestamp(ts) + .setContentType("application/octet-stream") + .setBytesData(ByteString.copyFrom(new byte[] {1, 2, 3, 4})) + .setExtensions( + Struct.newBuilder() + .putFields( + "comExampleExtension1", + Value.newBuilder().setStringValue("value1").build()) + .putFields( + "comExampleExtension2", + Value.newBuilder().setStringValue("value2").build())) + .build(); ``` It has the following proto3 standard JSON representation. @@ -238,7 +311,8 @@ It has the following proto3 standard JSON representation. } ``` -A CloudEvent whose data payload is a proto message can be constructed as follows: +A CloudEvent whose data payload is a proto message can be constructed +as follows: The proto payload’s message definition: ```proto @@ -249,28 +323,29 @@ message CustomMessage { } ``` -It has the following proto3 standard JSON representation. Note that the binary data is put in the `protoData` field rather than `data`: +It has the following proto3 standard JSON representation. Note that +the binary data is put in the `protoData` field rather than `data`: ```java CloudEvent.newBuilder() - .setEventType("com.example.readablemessagesservice") - .setEventTypeVersion("1.0") - .setCloudEventsVersion("0.1") - .setSource("producer1") - .setEventId("100") - .setTimestamp(ts) - .setContentType("application/protobuf") - .setProtoData( - Any.pack(CustomMessage.newBuilder().setContents("helloworld").build())) - .setExtensions( - Struct.newBuilder() - .putFields( - "comExampleExtension1", - Value.newBuilder().setStringValue("value1").build()) - .putFields( - "comExampleExtension2", - Value.newBuilder().setStringValue("value2").build())) - .build(); + .setEventType("com.example.readablemessagesservice") + .setEventTypeVersion("1.0") + .setCloudEventsVersion("0.1") + .setSource("producer1") + .setEventId("100") + .setTimestamp(ts) + .setContentType("application/protobuf") + .setProtoData( + Any.pack(CustomMessage.newBuilder().setContents("helloworld").build())) + .setExtensions( + Struct.newBuilder() + .putFields( + "comExampleExtension1", + Value.newBuilder().setStringValue("value1").build()) + .putFields( + "comExampleExtension2", + Value.newBuilder().setStringValue("value2").build())) + .build(); ``` It has the following proto3 standard JSON representation. From b7e39227ae4e358f30d50a2c0f12087260ac394c Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Thu, 23 Aug 2018 10:33:49 -0700 Subject: [PATCH 05/14] Only allow JSON data payloads Efficient representation of binary bytes or proto payloads is not a must have. The "data" field can represent bytes or serialized protos, just in a less efficient matter. I removed usages of these features to make the standard JSON look the same as the CE JSON. Signed-off-by: Spencer Fang --- protobuf-format.md | 119 +++++++++++---------------------------------- 1 file changed, 29 insertions(+), 90 deletions(-) diff --git a/protobuf-format.md b/protobuf-format.md index 438718d5c..09655157e 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -70,14 +70,8 @@ message CloudEvent { google.protobuf.Timestamp timestamp = 6; string schema_url = 7; string content_type = 8; - // `oneof payload` itself is not a real field, it only enforces the - // `oneof` constraint - oneof payload { - google.protobuf.Value data = 9; - bytes bytes_data = 10; - google.protobuf.Any proto_data = 11; - } - google.protobuf.Struct extensions = 12; + google.protobuf.Value data = 9; + google.protobuf.Struct extensions = 11; } ``` @@ -94,7 +88,7 @@ system is mapped into protobuf types as follows: | Timestamp | google.protobuf.Timestamp | Map | google.protobuf.Struct | Object | google.protobuf.Value -| Integer | int32 +| Integer | int32 ### 2.1 Note on cloud_events_version: @@ -165,7 +159,7 @@ CloudEvent.newBuilder() .setExtensions( Struct.newBuilder() .putFields( - "comExampleMyextension", + "vendorExtension", Value.newBuilder().setStringValue("myvalue").build())) .build(); ``` @@ -209,14 +203,13 @@ in the upgrade process: ### 2.4 Relation to CloudEvents JSON format: -All proto3 messages have a standard JSON form. The standard JSON of -this protobuf format is not compatible from the official [CloudEvents -JSON encoding][CE_JSON_ENCODING] at the time of writing. A service -that supports the standard protobuf JSON encoding as well as the -official CloudEvents JSON encoding will end up supporting two -different JSON encodings. +All proto3 messages have a standard JSON form. + +Note: at the time of writing, extensions are in a separate extensions +bag. The standard JSON is the same as the CloudEvents JSON. -Below are a few examples of the proto3 JSON. +Below are a few examples of the proto message and its JSON +representation. A CloudEvent whose payload is a JSON may be constructed as follows: ```java @@ -241,10 +234,10 @@ CloudEvent.newBuilder() .setExtensions( Struct.newBuilder() .putFields( - "comExampleExtension1", + "vendorExtension1", Value.newBuilder().setStringValue("value1").build()) .putFields( - "comExampleExtension2", + "vendorExtension2", Value.newBuilder().setStringValue("value2").build())) .build(); ``` @@ -265,8 +258,8 @@ It has the following proto3 standard JSON representation: "email": "user@example.com" }, "extensions": { - "comExampleExtension1": "value1", - "comExampleExtension2": "value2" + "vendorExtension1": "value1", + "vendorExtension2": "value2" } } ``` @@ -274,76 +267,24 @@ It has the following proto3 standard JSON representation: A CloudEvent whose data payload is bytes may be constructed as follows: ```java CloudEvent.newBuilder() - .setEventType("com.example.octetstream") + .setEventType("com.example.somebytes") .setEventTypeVersion("1.0") .setCloudEventsVersion("0.1") .setSource("producer1") .setEventId("100") .setTimestamp(ts) - .setContentType("application/octet-stream") - .setBytesData(ByteString.copyFrom(new byte[] {1, 2, 3, 4})) - .setExtensions( - Struct.newBuilder() - .putFields( - "comExampleExtension1", - Value.newBuilder().setStringValue("value1").build()) - .putFields( - "comExampleExtension2", - Value.newBuilder().setStringValue("value2").build())) - .build(); -``` - -It has the following proto3 standard JSON representation. -```json -{ - "eventType": "com.example.octetstreamservice", - "eventTypeVersion": "1.0", - "cloudEventsVersion": "0.1", - "source": "producer1", - "eventId": "100", - "timestamp": "2018-08-22T16:04:45.017Z", - "contentType": "application/protobuf", - "bytesData": "AQIDBA==", - "extensions": { - "comExampleExtension1": "value1", - "comExampleExtension2": "value2" - } -} -``` - -A CloudEvent whose data payload is a proto message can be constructed -as follows: - -The proto payload’s message definition: -```proto -package com.example; - -message CustomMessage { - string contents = 1; -} -``` - -It has the following proto3 standard JSON representation. Note that -the binary data is put in the `protoData` field rather than `data`: - -```java -CloudEvent.newBuilder() - .setEventType("com.example.readablemessagesservice") - .setEventTypeVersion("1.0") - .setCloudEventsVersion("0.1") - .setSource("producer1") - .setEventId("100") - .setTimestamp(ts) - .setContentType("application/protobuf") - .setProtoData( - Any.pack(CustomMessage.newBuilder().setContents("helloworld").build())) + .setContentType("application/json") + .setSchemaUrl("https://com.example.schema/usercreated") + .setData( + Value.newBuilder() + .setStringValue(Base64.encode(new byte[] {1, 2, 3, 4}))) .setExtensions( Struct.newBuilder() .putFields( - "comExampleExtension1", + "vendorExtension1", Value.newBuilder().setStringValue("value1").build()) .putFields( - "comExampleExtension2", + "vendorExtension2", Value.newBuilder().setStringValue("value2").build())) .build(); ``` @@ -351,20 +292,18 @@ CloudEvent.newBuilder() It has the following proto3 standard JSON representation. ```json { - "eventType": "com.example.readablemessages", + "eventType": "com.example.somebytes", "eventTypeVersion": "1.0", "cloudEventsVersion": "0.1", "source": "producer1", "eventId": "100", - "timestamp": "2018-08-22T16:15:07.386Z", - "contentType": "application/protobuf", - "protoData": { - "@type": "type.googleapis.com/com.example.CustomMessage", - "contents": "helloworld" - }, + "timestamp": "2018-08-23T17:24:34.226Z", + "schemaUrl": "https://com.example.schema/usercreated", + "contentType": "application/json", + "data": "AQIDBA==", "extensions": { - "comExampleExtension1": "value1", - "comExampleExtension2": "value2" + "vendorExtension1": "value1", + "vendorExtension2": "value2" } } ``` From 138fb99a0f8f138079270a6b4cf281ba959f1e4c Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Fri, 24 Aug 2018 09:31:39 -0700 Subject: [PATCH 06/14] Clarify attributing casing styles protobuf lower_snake_case attrs are turned into lowerCamelCase in the proto standard JSON form. But it is legal to have top level attribute fields that are not lowerCamelCase. Signed-off-by: Spencer Fang --- protobuf-format.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/protobuf-format.md b/protobuf-format.md index 09655157e..0ef4dbc51 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -90,6 +90,15 @@ system is mapped into protobuf types as follows: | Object | google.protobuf.Value | Integer | int32 +By default, lower_case_names in the protobuf IDL field names are +represented as lowerCamelCase in the protobuf standard JSON format. If +the field has a different styled casing, the `json_name` protobuf +option MUST be used: + +``` +string arbitrary_casing_attr = 12 [json_name = "aRbItRaRy_casing_ATTR_"]; +``` + ### 2.1 Note on cloud_events_version: The protobuf `package` keyword defines a package name that includes From 2505d1be1118187cba6f3ebba014a655aafd3c80 Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Mon, 8 Oct 2018 17:47:31 -0700 Subject: [PATCH 07/14] Update protobuf spec to use new design that favors forwards compatibility over type safety. Signed-off-by: Spencer Fang --- protobuf-format.md | 349 +++++++-------------------------------------- 1 file changed, 50 insertions(+), 299 deletions(-) diff --git a/protobuf-format.md b/protobuf-format.md index 0ef4dbc51..36ae1c91b 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -2,36 +2,31 @@ ## Abstract -The Protocol Buffers (protobuf) Format for CloudEvents defines how -events are to be expressed in protobuf version 3. +The Protocol Buffers Format for CloudEvents (CE) defines +the encoding of events in the protocol buffers binary format. ## Status of this document -This document is a working draft +This document is a working draft. ## 1. Introduction CloudEvents is a standardized and transport-neutral definition of the structure and metadata description of events. This specification -defines how the elements defined in the CloudEvents specification are -to be represented in the Protocol buffers version 3. +defines how the [Context +Attributes](spec.md#context-attributes)defined in the CloudEvents in +the protocol buffers binary encoding MUST BE encoded. Transcoding to +and from other formats (e.g. JSON) is out of the scope of this +proposal. Protocol buffers are a language-neutral, platform-neutral extensible -mechanism for serializing structured data. A message is defined once -using the protobuf interface description language (IDL), and the -protobuf compiler generates the language specific libraries for -serializing to and deserializing from the binary representation. The -protobuf version 3 library can also convert a protobuf message into -its [standard JSON form][PROTO_JSON]. - -Some built in message types in protobuf 3 (`google.protobuf.Value`, -`google.protobuf.Any`, etc) have special runtime support from the -protobuf library. These built in messages are given special treatment -when transcoding to and from JSON. Tools in the protobuf ecosystem -also give special treatment to these messages. For example, the -[Common Expression Language tool (CEL)][CEL] automatically converts a -`google.protobuf.Any` into its contained message when performing data -queries. +mechanism for serializing structured data. The [Google reference +implementation of protobuf](PROTO) includes support for an interface +descriptor language (IDL), and this document makes use of language +level 3 IDL from Protocol Buffers v3.5.0. CloudEvents systems using +protocol buffers are not required to use the IDL or any particular +implementation of Protocol buffers as long as they produce messages +which match the binary encoding defined by the IDL. ### 1.1. Conformance @@ -39,297 +34,53 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC2119. -## 2. Protobuf format - -Users of protobuf MUST use the official message definition when -expressing a CloudEvent as a protobuf message. Users SHOULD NOT change -the language specific options in order to have consistent code -generation output. Users MUST NOT change the protobuf package -name. Users MUST NOT use any protobuf options that change the protobuf -wire compatibility such as message_set_wire_format. - -```proto -syntax = "proto3"; - -package io.cloudevents.v0; - -import "google/protobuf/any.proto"; -import "google/protobuf/struct.proto"; -import "google/protobuf/timestamp.proto"; - -option go_package = "cloudevents.io/protobuf/"; -option java_package = "io.cloudevents"; -option java_multiple_files = true; - -message CloudEvent { - string event_type = 1; - string event_type_version = 2; - string cloud_events_version = 3; - string source = 4; - string event_id = 5; - google.protobuf.Timestamp timestamp = 6; - string schema_url = 7; - string content_type = 8; - google.protobuf.Value data = 9; - google.protobuf.Struct extensions = 11; -} -``` - -In general, the CloudEvents attribute names are converted into -protobuf fields in lower_snake_case, and the the CloudEvents type -system is mapped into protobuf types as follows: - - -| CloudEvents | Protobuf -|--------------|------------------------------------------------------------- -| String | string -| Binary | bytes -| URI | string -| Timestamp | google.protobuf.Timestamp -| Map | google.protobuf.Struct -| Object | google.protobuf.Value -| Integer | int32 - -By default, lower_case_names in the protobuf IDL field names are -represented as lowerCamelCase in the protobuf standard JSON format. If -the field has a different styled casing, the `json_name` protobuf -option MUST be used: - -``` -string arbitrary_casing_attr = 12 [json_name = "aRbItRaRy_casing_ATTR_"]; -``` - -### 2.1 Note on cloud_events_version: - -The protobuf `package` keyword defines a package name that includes -the CloudEvents major version number. The package name and message -name together form a fully qualified name for a protobuf -message. Messages with the same name but different package names are -independent messages with no relation to one another. The -`cloud_events_version` field’s major version number MUST match the -major version of the protobuf package. - -### 2.2 Special handling of `data` attribute: - -#### 2.2.1 Background knowledge on special built in protobuf types - -`google.protobuf.Value` ([more info][PROTO_JSON]) is used to express a -JSON structure. Its fields are composed of special wrapper value types -such as `google.protobuf.BytesValue`. The protobuf runtime library -provides JSON conversion support for these special built in message -types. - -`google.protobuf.Any` ([more info][PROTO_ANY]) is used in protobuf to -express a field whose type is not known to the enclosing message. It -contains the binary representation of the embedded message and a URL -string that identifies the type of the embedded message. - +## 2. Protocol Buffers format -#### 2.2.2 Handling instructions +Protocol buffers provide a binary data serialization format which is +substantially more compact and efficient to parse when compared to XML +or JSON, along with a variety of language-specific libraries to +perform automatic serialization and deserialization. The [protocol +buffers specification defines a well-known encoding +format](https://developers.google.com/protocol-buffers/docs/encoding) +which is the basis of this specification. Specifications below may be +written using the protocol buffers project IDL for readability, but +the ultimate basis of this specification is the protocol buffers +binary encoding. -If the `contentType` is `application/json` or any media type with the -structured `+json` suffix, the implementation MUST store the JSON -payload in the `data` field. -If the payload is binary, the implementation MUST store the bytes in -the `bytes_data` field. +### 2.1 Definition -If the payload is a protobuf, the implementation MUST store the -payload in the `proto_data` field and the `contentType` MUST be -`application/protobuf`. +Users of protocol buffers MUST use a message whose binary encoding is +identical to the one described by the CloudEventMap message: -### 2.3 Extensions: - -`google.protobuf.Struct` ([more info][PROTO_STRUCT]) represents an -arbitrary JSON structure. The keys of the `Struct` are the names of -the extensions. The values of the `google.protobuf.Struct` are -`com.protobuf.Value` ([more info][PROTO_VALUE]) fields whose contents -correspond to the value of the extension mapped into JSON using the -[JSON mapping rules][CE_JSON_ENCODING]. All extensions attributes MUST -be put into this bag because they are not a part of the CloudEvents -spec. - -Well known extensions can not be given a top level field because they -have no official standing, therefore the protobuf definition can not -commit to a strongly typed representation of the value. The protobuf -binding must be able to map an extension name to different value -types, e.g. a "exampleExtension" with CE type "Object" in one message -and another "exampleExtension" with CE type "Integer" in another -message. - -When a new CloudEvent spec version is released, promoted extensions -move from the extensions bag to a strongly typed top level field. - -For example, consider an extension for CloudEvents 1.0: - -```java -CloudEvent.newBuilder() - .setCloudEventsVersion("1.0") - .setExtensions( - Struct.newBuilder() - .putFields( - "vendorExtension", - Value.newBuilder().setStringValue("myvalue").build())) - .build(); -``` - -In the next CloudEvents minor release, the extension is promoted to an -official field. The protobuf message definition is updated for -CloudEvents 1.1: - -```java -CloudEvent.newBuilder() - .setCloudEventsVersion("1.1") - .setComExampleMyextension("myvalue") - // During the upgrade process producers SHOULD continue setting - // exts bag to maintain backwards compat - .setExtensions( - Struct.newBuilder() - .putFields( - "comExampleMyextension", - Value.newBuilder().setStringValue("myvalue").build())) - .build(); -``` - -#### 2.3.1 CloudEvents upgrade process - -If a field moves from the extensions bag to a top level field, then -the producers and consumers of a CloudEvents system SHOULD coordinate -in the upgrade process: - -1. Initially, the producers and consumers are using CloudEvents 1.0 - and the extension is expressed in the “extensions” bag. -1. CloudEvents 1.1 is released. -1. The producers write the extension to both the extensions bag and - the type safe top level field. The messages will have - "cloudEventsVersion" set to "1.1", but the extension is still - readable by 1.0 consumers. -1. All consumers upgrade to 1.1 and stop reading from the extensions - bag and switch to reading from only the type safe top level field. -1. The producers stop writing to the extensions bag, and only write to - the type safe top level field. - - -### 2.4 Relation to CloudEvents JSON format: - -All proto3 messages have a standard JSON form. - -Note: at the time of writing, extensions are in a separate extensions -bag. The standard JSON is the same as the CloudEvents JSON. - -Below are a few examples of the proto message and its JSON -representation. +```proto +syntax = "proto3"; -A CloudEvent whose payload is a JSON may be constructed as follows: -```java -CloudEvent.newBuilder() - .setEventType("com.example.usercreated") - .setEventTypeVersion("1.0") - .setCloudEventsVersion("0.1") - .setSource("producer1") - .setEventId("100") - .setTimestamp(ts) - .setContentType("application/json") - .setSchemaUrl("https://com.example.schema/usercreated") - .setData( - Value.newBuilder().setStructValue(Struct.newBuilder() - .putFields( - "username", - Value.newBuilder().setStringValue("theusername").build()) - .putFields( - "email", - Value.newBuilder().setStringValue("user@example.com").build()) - .build())) - .setExtensions( - Struct.newBuilder() - .putFields( - "vendorExtension1", - Value.newBuilder().setStringValue("value1").build()) - .putFields( - "vendorExtension2", - Value.newBuilder().setStringValue("value2").build())) - .build(); -``` +package io.cloudevents; -It has the following proto3 standard JSON representation: -```json -{ - "eventType": "com.example.usercreated", - "eventTypeVersion": "1.0", - "cloudEventsVersion": "0.1", - "source": "producer1", - "eventId": "100", - "timestamp": "2018-08-22T15:47:00.951Z", - "schemaUrl": "https://com.example.schema/usercreated", - "contentType": "application/json", - "data": { - "username": "theusername", - "email": "user@example.com" - }, - "extensions": { - "vendorExtension1": "value1", - "vendorExtension2": "value2" - } +// allows a map to appear inside `oneof` +message CloudEventMap { + map value = 1; } -``` - -A CloudEvent whose data payload is bytes may be constructed as follows: -```java -CloudEvent.newBuilder() - .setEventType("com.example.somebytes") - .setEventTypeVersion("1.0") - .setCloudEventsVersion("0.1") - .setSource("producer1") - .setEventId("100") - .setTimestamp(ts) - .setContentType("application/json") - .setSchemaUrl("https://com.example.schema/usercreated") - .setData( - Value.newBuilder() - .setStringValue(Base64.encode(new byte[] {1, 2, 3, 4}))) - .setExtensions( - Struct.newBuilder() - .putFields( - "vendorExtension1", - Value.newBuilder().setStringValue("value1").build()) - .putFields( - "vendorExtension2", - Value.newBuilder().setStringValue("value2").build())) - .build(); -``` -It has the following proto3 standard JSON representation. -```json -{ - "eventType": "com.example.somebytes", - "eventTypeVersion": "1.0", - "cloudEventsVersion": "0.1", - "source": "producer1", - "eventId": "100", - "timestamp": "2018-08-23T17:24:34.226Z", - "schemaUrl": "https://com.example.schema/usercreated", - "contentType": "application/json", - "data": "AQIDBA==", - "extensions": { - "vendorExtension1": "value1", - "vendorExtension2": "value2" +message CloudEventAny { + oneof value { + string string_value = 1; + bytes binary_value = 2; + uint32 int_value = 3; + CloudEventMap map_value = 4; } } ``` -## 3. References: - -* [google.protobuf.Any][PROTO_ANY] -* [google.protobuf.Struct][PROTO_STRUCT] -* [google.protobuf.Value][PROTO_VALUE] -* [Protobuf and JSON][PROTO_JSON] -* [Common Expression Language][CEL] +The CloudEvents type system is mapped into the fields of CloudEventAny as follows: -[PROTO_ANY]: https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#any -[PROTO_STRUCT]: https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#struct -[PROTO_VALUE]: https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Value -[PROTO_JSON]: https://developers.google.com/protocol-buffers/docs/proto3#json -[CEL]: https://github.com/google/cel-spec/blob/master/doc/langdef.md -[CE_JSON_ENCODING]: ./json-format.md -[CE_SPEC]: ./spec.md +| CloudEvents | CloudEventAny field +|--------------|------------------------------------------------------------- +| String | string_value +| Binary | binary_value +| URI | string_value (string expression conforming to URI-reference as defined in RFC 3986 §4.1) +| Timestamp | string_value (string expression as defined in RFC 3339.) +| Map | map_value +| Integer | int_value From e7ad0806220d99d854b7320b2a9a13b7e964b03e Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Fri, 12 Oct 2018 15:20:43 -0700 Subject: [PATCH 08/14] Fix URL to pass linter Signed-off-by: Spencer Fang --- protobuf-format.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/protobuf-format.md b/protobuf-format.md index 36ae1c91b..96110175c 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -21,12 +21,14 @@ proposal. Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data. The [Google reference -implementation of protobuf](PROTO) includes support for an interface -descriptor language (IDL), and this document makes use of language -level 3 IDL from Protocol Buffers v3.5.0. CloudEvents systems using -protocol buffers are not required to use the IDL or any particular -implementation of Protocol buffers as long as they produce messages -which match the binary encoding defined by the IDL. +implementation of +protobuf](https://github.com/protocolbuffers/protobuf) includes +support for an interface descriptor language (IDL), and this document +makes use of language level 3 IDL from Protocol Buffers +v3.5.0. CloudEvents systems using protocol buffers are not required to +use the IDL or any particular implementation of Protocol buffers as +long as they produce messages which match the binary encoding defined +by the IDL. ### 1.1. Conformance From 212f40c6554c534c8440f28c6149d7b43685490b Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Fri, 12 Oct 2018 15:28:03 -0700 Subject: [PATCH 09/14] Fix whitespace Signed-off-by: Spencer Fang --- protobuf-format.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/protobuf-format.md b/protobuf-format.md index 96110175c..47e2aeb4c 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -14,7 +14,7 @@ This document is a working draft. CloudEvents is a standardized and transport-neutral definition of the structure and metadata description of events. This specification defines how the [Context -Attributes](spec.md#context-attributes)defined in the CloudEvents in +Attributes](spec.md#context-attributes) defined in the CloudEvents in the protocol buffers binary encoding MUST BE encoded. Transcoding to and from other formats (e.g. JSON) is out of the scope of this proposal. From 83c3b820527a1d72b310940fccd85ea5bf051562 Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Thu, 25 Oct 2018 13:29:58 -0700 Subject: [PATCH 10/14] Address PR comments, add example. Signed-off-by: Spencer Fang --- protobuf-format.md | 99 +++++++++++++++++++++++++++++++++------------- 1 file changed, 72 insertions(+), 27 deletions(-) diff --git a/protobuf-format.md b/protobuf-format.md index 47e2aeb4c..2349c7b4f 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -1,9 +1,9 @@ -# Protobuf Event Format for CloudEvents - Version 0.1 +# Protocol Buffers Event Format for CloudEvents - Version 0.1 ## Abstract -The Protocol Buffers Format for CloudEvents (CE) defines -the encoding of events in the protocol buffers binary format. +The Protocol Buffers Format for CloudEvents (CE) defines the encoding +of CloudEvents in the Protocol Buffers binary format. ## Status of this document @@ -11,48 +11,46 @@ This document is a working draft. ## 1. Introduction -CloudEvents is a standardized and transport-neutral definition of the -structure and metadata description of events. This specification -defines how the [Context -Attributes](spec.md#context-attributes) defined in the CloudEvents in -the protocol buffers binary encoding MUST BE encoded. Transcoding to -and from other formats (e.g. JSON) is out of the scope of this -proposal. +This specification defines how the [Context +Attributes](spec.md#context-attributes) Attributes defined in the +CloudEvents specification MUST be encoded in the protocol buffer +binary format. Transcoding to and from other formats (e.g. JSON) is +out of the scope of this document. -Protocol buffers are a language-neutral, platform-neutral extensible +Protocol Buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data. The [Google reference -implementation of -protobuf](https://github.com/protocolbuffers/protobuf) includes -support for an interface descriptor language (IDL), and this document -makes use of language level 3 IDL from Protocol Buffers -v3.5.0. CloudEvents systems using protocol buffers are not required to -use the IDL or any particular implementation of Protocol buffers as -long as they produce messages which match the binary encoding defined -by the IDL. +implementation of Protocol +Buffers](https://github.com/protocolbuffers/protobuf) includes support +for an interface descriptor language (IDL), and this document makes +use of language level 3 IDL from Protocol Buffers v3.5.0. CloudEvents +systems using Protocol Buffers are not required to use the IDL or any +particular implementation of Protocol Buffers as long as they produce +messages which match the binary encoding defined by the IDL. ### 1.1. Conformance The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this -document are to be interpreted as described in RFC2119. +document are to be interpreted as described in +[RFC2119](https://tools.ietf.org/html/rfc2119). ## 2. Protocol Buffers format -Protocol buffers provide a binary data serialization format which is +Protocol Buffers provide a binary data serialization format which is substantially more compact and efficient to parse when compared to XML or JSON, along with a variety of language-specific libraries to -perform automatic serialization and deserialization. The [protocol -buffers specification defines a well-known encoding +perform automatic serialization and deserialization. The [Protocol +Buffers specification defines a well-known encoding format](https://developers.google.com/protocol-buffers/docs/encoding) which is the basis of this specification. Specifications below may be -written using the protocol buffers project IDL for readability, but -the ultimate basis of this specification is the protocol buffers +written using the Protocol Buffers project IDL for readability, but +the ultimate basis of this specification is the Protocol Buffers binary encoding. ### 2.1 Definition -Users of protocol buffers MUST use a message whose binary encoding is +Users of Protocol Buffers MUST use a message whose binary encoding is identical to the one described by the CloudEventMap message: ```proto @@ -75,7 +73,8 @@ message CloudEventAny { } ``` -The CloudEvents type system is mapped into the fields of CloudEventAny as follows: +The CloudEvents type system MUST be mapped into the fields of +`CloudEventAny` as follows: | CloudEvents | CloudEventAny field @@ -86,3 +85,49 @@ The CloudEvents type system is mapped into the fields of CloudEventAny as follow | Timestamp | string_value (string expression as defined in RFC 3339.) | Map | map_value | Integer | int_value +| Any | Not applicable. Any is the enclosing CloudEventAny message itself + + +## Examples + +Below is an example of how to create a CloudEvent Protocol Buffer +message using the Java Google Protocol Buffers library: + +```java +import com.google.common.base.Charsets; +import com.google.protobuf.ByteString; + + +CloudEventMap event = CloudEventMap.newBuilder() + .putValue( + "eventType", + CloudEventAny.newBuilder() + .setStringValue("com.example.emitter.event") + .build()) + .putValue( + "cloudEventsVersion", + CloudEventAny.newBuilder() + .setStringValue("0.1") + .build()) + .putValue( + "eventTime", + CloudEventAny.newBuilder() + .setStringValue("2018-10-25T00:00:00+00:00") + .build()) + .putValue( + "source", + CloudEventAny.newBuilder() + .setStringValue("com.example.source.host1") + .build()) + .putValue( + "comExampleCustomextension", + CloudEventAny.newBuilder() + .setStringValue("some value for the extension") + .build()) + .putValue( + "data", + CloudEventAny.newBuilder() + .setBinaryValue(ByteString.copyFrom("a binary string", Charsets.UTF_8)) + .build()) + .build(); +``` From b018b8627f09d40158caf181990e4f19a1e14a41 Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Thu, 25 Oct 2018 13:32:56 -0700 Subject: [PATCH 11/14] Turn RFC references into links. Signed-off-by: Spencer Fang --- protobuf-format.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/protobuf-format.md b/protobuf-format.md index 2349c7b4f..ca44c31f1 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -81,8 +81,8 @@ The CloudEvents type system MUST be mapped into the fields of |--------------|------------------------------------------------------------- | String | string_value | Binary | binary_value -| URI | string_value (string expression conforming to URI-reference as defined in RFC 3986 §4.1) -| Timestamp | string_value (string expression as defined in RFC 3339.) +| URI | string_value (string expression conforming to URI-reference as defined in [RFC 3986 §4.1](https://tools.ietf.org/html/rfc3986#section-4.1)) +| Timestamp | string_value (string expression as defined in [RFC 3339](https://tools.ietf.org/html/rfc3339)) | Map | map_value | Integer | int_value | Any | Not applicable. Any is the enclosing CloudEventAny message itself From 8f2d651e9b517e52088720d6823a0e6cdf4a8864 Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Fri, 2 Nov 2018 16:13:11 -0700 Subject: [PATCH 12/14] PR comments - s/Attributes Atttributes/Attributes/ - s/required/mandated/ - Add protobuf-format.md to Makefile - add cloudevent.proto Signed-off-by: Spencer Fang --- Makefile | 2 +- cloudevent.proto | 17 +++++++++++++++++ protobuf-format.md | 22 +++++++++++++++------- 3 files changed, 33 insertions(+), 8 deletions(-) create mode 100644 cloudevent.proto diff --git a/Makefile b/Makefile index b8d183f73..e391a42ba 100644 --- a/Makefile +++ b/Makefile @@ -7,6 +7,6 @@ verify: @echo Running the spec phrase checker: @tools/verify-specs.sh -v spec.md documented-extensions.md json-format.md \ http-transport-binding.md http-webhook.md mqtt-transport-binding.md \ - nats-transport-binding.md + nats-transport-binding.md protobuf-format.md @echo Running the doc phrase checker: @tools/verify-docs.sh -v . diff --git a/cloudevent.proto b/cloudevent.proto new file mode 100644 index 000000000..7e406c702 --- /dev/null +++ b/cloudevent.proto @@ -0,0 +1,17 @@ +syntax = "proto3"; + +package io.cloudevents; + +// allows a map to appear inside `oneof` +message CloudEventMap { + map value = 1; +} + +message CloudEventAny { + oneof value { + string string_value = 1; + bytes binary_value = 2; + uint32 int_value = 3; + CloudEventMap map_value = 4; + } +} diff --git a/protobuf-format.md b/protobuf-format.md index ca44c31f1..505c3c0eb 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -12,10 +12,10 @@ This document is a working draft. ## 1. Introduction This specification defines how the [Context -Attributes](spec.md#context-attributes) Attributes defined in the -CloudEvents specification MUST be encoded in the protocol buffer -binary format. Transcoding to and from other formats (e.g. JSON) is -out of the scope of this document. +Attributes](spec.md#context-attributes) defined in the CloudEvents +specification MUST be encoded in the protocol buffer binary +format. Transcoding to and from other formats (e.g. JSON) is out of +the scope of this document. Protocol Buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data. The [Google reference @@ -23,7 +23,7 @@ implementation of Protocol Buffers](https://github.com/protocolbuffers/protobuf) includes support for an interface descriptor language (IDL), and this document makes use of language level 3 IDL from Protocol Buffers v3.5.0. CloudEvents -systems using Protocol Buffers are not required to use the IDL or any +systems using Protocol Buffers are not mandated to use the IDL or any particular implementation of Protocol Buffers as long as they produce messages which match the binary encoding defined by the IDL. @@ -51,7 +51,8 @@ binary encoding. ### 2.1 Definition Users of Protocol Buffers MUST use a message whose binary encoding is -identical to the one described by the CloudEventMap message: +identical to the one described by the [CloudEventMap +message](./cloudevent.proto): ```proto syntax = "proto3"; @@ -87,8 +88,15 @@ The CloudEvents type system MUST be mapped into the fields of | Integer | int_value | Any | Not applicable. Any is the enclosing CloudEventAny message itself +## 2.2 Relation to HTTP -## Examples +The [HTTP transport binding](./http-transport-binding.md) defines a +*structued* and *binary* mode. + +For the Protocol Buffer CloudEvent format expressed + + +## 3. Examples Below is an example of how to create a CloudEvent Protocol Buffer message using the Java Google Protocol Buffers library: From 1068d9a7690de5692d52f2645afd269b209d6575 Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Fri, 2 Nov 2018 16:44:31 -0700 Subject: [PATCH 13/14] remove HTTP modes section Signed-off-by: Spencer Fang --- protobuf-format.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/protobuf-format.md b/protobuf-format.md index 505c3c0eb..acdfe4faa 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -88,13 +88,6 @@ The CloudEvents type system MUST be mapped into the fields of | Integer | int_value | Any | Not applicable. Any is the enclosing CloudEventAny message itself -## 2.2 Relation to HTTP - -The [HTTP transport binding](./http-transport-binding.md) defines a -*structued* and *binary* mode. - -For the Protocol Buffer CloudEvent format expressed - ## 3. Examples From 93428cbec51990a5ca66b9ed9b289a6ddaa3a70c Mon Sep 17 00:00:00 2001 From: Spencer Fang Date: Mon, 5 Nov 2018 10:30:28 -0800 Subject: [PATCH 14/14] Reword confusing use of speficiation / below Signed-off-by: Spencer Fang --- protobuf-format.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/protobuf-format.md b/protobuf-format.md index acdfe4faa..cc0d65d7e 100644 --- a/protobuf-format.md +++ b/protobuf-format.md @@ -42,8 +42,8 @@ or JSON, along with a variety of language-specific libraries to perform automatic serialization and deserialization. The [Protocol Buffers specification defines a well-known encoding format](https://developers.google.com/protocol-buffers/docs/encoding) -which is the basis of this specification. Specifications below may be -written using the Protocol Buffers project IDL for readability, but +which is the basis of this specification. This specification is +described using the Protocol Buffers project IDL for readability, but the ultimate basis of this specification is the Protocol Buffers binary encoding.