-
Notifications
You must be signed in to change notification settings - Fork 612
RocketMQ transport binding #352
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
ba0463e
74a3713
d9a9882
b5b1d9f
1d345f3
1f2c9ea
4dee749
7546f2e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,234 @@ | ||
| # RocketMQ Transport Binding for CloudEvents | ||
|
|
||
| ## Abstract | ||
|
|
||
| The [RocketMQ][RocketMQ] Transport Binding for CloudEvents defines how events are mapped | ||
| to RocketMQ messages. | ||
|
|
||
| ## Status of this document | ||
|
|
||
| This document is a working draft. | ||
|
|
||
| ## Table of Contents | ||
| 1. [Introduction](#1-introduction) | ||
| - 1.1. [Conformance](#11-conformance) | ||
| - 1.2. [Relation to RocketMQ](#12-relation-to-rocketmq) | ||
| - 1.3. [Content Modes](#13-content-modes) | ||
| - 1.4. [Event Formats](#14-event-formats)) | ||
| - 1.5. [Security](#15-security) | ||
| 2. [Use of CloudEvents Attributes](#2-use-of-cloudevents-attributes) | ||
| - 2.1. [contenttype Attribute](#21-contenttype-attribute) | ||
| - 2.2. [data Attribute](#22-data-attribute) | ||
| 3. [RocketMQ Message Mapping](#3-rocketmq-message-mapping) | ||
| - 3.1. [Binary Content Mode](#31-binary-content-mode) | ||
| - 3.2. [Structured Content Mode](#32-structured-content-mode) | ||
| 4. [References](#4-references) | ||
|
|
||
| ## 1. Introduction | ||
| [CloudEvents][CE] is a standardized and transport-neutral definition of the | ||
| structure and metadata description of events. This specification defines how | ||
| the elements defined in the CloudEvents specification are to be used in the | ||
| RocketMQ Message protocol as client produced and consumed messages. | ||
|
|
||
| ### 1.1. Conformance | ||
|
|
||
| The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", | ||
| "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be | ||
| interpreted as described in RFC2119. | ||
|
|
||
| ### 1.2. Relation to RocketMQ | ||
|
|
||
| This specification does not prescribe rules constraining transfer or settlement | ||
| of event messages with RocketMQ; it solely defines how CloudEvents are expressed | ||
| in the RocketMQ message transport protocol as client messages that are produced and consumed. | ||
|
|
||
| ### 1.3. Content Modes | ||
|
|
||
| The specification defines two content modes for transferring events: | ||
| *structured* and *binary*. | ||
|
|
||
| The RocketMQ protocol have already supported custom message headers, | ||
| necessary for *binary* mode. | ||
|
|
||
| Event metadata attributes and event data are placed into the RocketMQ message | ||
| payload using an [event format](#14-event-formats). | ||
|
|
||
| ### 1.4. Event Formats | ||
|
|
||
| Event formats, used with the *stuctured* content mode, define how an event is | ||
| expressed in a particular data format. All implementations of this | ||
| specification MUST support the [JSON event format][JSON-format]. | ||
|
|
||
| ### 1.5. Security | ||
| This specification does not introduce any new security features for RocketMQ, | ||
| or mandate specific existing features to be used. | ||
|
|
||
| ## 2. Use of CloudEvents Attributes | ||
|
|
||
| This specification does not further define any of the [CloudEvents][CE] event | ||
| attributes. | ||
|
|
||
| ### 2.1. contenttype Attribute | ||
|
|
||
| The `contenttype` attribute is assumed to contain a media-type expression | ||
| compliant with [RFC2046][RFC2046]. | ||
|
|
||
| ### 2.2. data Attribute | ||
| The `data` attribute is assumed to contain opaque application data that is | ||
| encoded as declared by the `contenttype` attribute. | ||
|
|
||
| An application is free to hold the information in any in-memory representation | ||
| of its choosing, but as the value is transposed into RocketMQ as defined in this | ||
| specification, core RocketMQ provides data available as a sequence of bytes. | ||
|
|
||
| For instance, if the declared `contenttype` is | ||
| `application/json;charset=utf-8`, the expectation is that the `data` attribute | ||
| value is made available as [UTF-8][RFC3629] encoded JSON text. | ||
|
|
||
| ## 3. RocketMQ Message Mapping | ||
| The receiver of the event can distinguish between the two content modes by inspecting | ||
| the `CE_contentType` property of the RocketMQ message. If the value is prefixed with the | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Again,binary mode is used in rocketmq message header.Structured content mode is used for RocketMQ message body.So,we put the cloudevent property into the message body to determine the message contentType.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. From looking at: https://github.com/cloudevents/spec/pull/352/files#diff-b1ac390f402e79182f0aa7a46737eb90R198 something doesn't feel right. On line 198 you've pulled out one of the CE properties (contenttype) as special and placed it in the "userproperties" section but I don't think that's correct. "contenttype" is meant to describe the format of "data" not of the "payload" which is how you've used it. So, let me ask again... is "CE_contenttype" the cloudevents property or one defined by RocketMQ? I believe it should be defined by RocketMQ and NOT the CloudEvents one since at that point in the processing we don't even know we have a cloudEvent at all, so prefixing a property with "CE_" feels premature. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @duglin We need a 'CE_contenttype' field to ensure the receiver of the event can distinguish between the two content modes by inspecting the 'CE_contenttype' property of the RocketMQ message. and this is a property can be set according to the data format of CloudEvents. |
||
| CloudEvents media type `application/cloudevents`, indicating the use of a known event format, | ||
| the receiver uses structured mode, otherwise it defaults to binary mode. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Again, let's drop the notion of binary since we don't support it. Then we can just mandate, with a MUST, what |
||
|
|
||
| If a receiver finds a CloudEvents media type as per the above rule, but with an event format | ||
| that it cannot handle, for instance `application/cloudevents+avro`, it MAY still treat the event | ||
| as binary and forward it to another party as-is . | ||
|
|
||
| ### 3.1. Binary Content Mode | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd drop this entire section since we're not supporting it.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Binary mode is used in rocketmq message header.So,I write the "Binary Content Mode" Here. |
||
|
|
||
| The [binary content mode](#31-binary-content-mode) accommodates any shape of event data, | ||
| and allows for efficient transfer and without transcoding effort. | ||
|
|
||
| #### 3.1.1. Content Type | ||
|
|
||
| For the binary mode, the header `CE_contenttype property` MUST be mapped directly to the CloudEvents | ||
| contentType attribute. | ||
|
|
||
| #### 3.1.2. Event Data Encoding | ||
|
|
||
| The data attribute byte-sequence MUST be used as the value of the RocketMQ message. | ||
|
|
||
| #### 3.1.3. Metadata Headers | ||
|
|
||
| All CloudEvents attributes and CloudEvent Attributes Extensions with exception of | ||
| data MUST be individually mapped to and from the Header fields in the RocketMQ message. | ||
|
|
||
| ##### 3.1.3.1 Property Names | ||
|
|
||
| [CloudEvents][CE] attributes are prefixed with `"CE_"` for use in the message section. | ||
|
|
||
| Examples: | ||
|
|
||
| * `time` maps to `CE_time` | ||
| * `id` maps to `CE_id` | ||
| * `specversion` maps to `CE_specversion` | ||
|
|
||
| ##### 3.1.3.2 Property Values | ||
|
|
||
| The value for each RocketMQ Message header is constructed from the respective header's RocketMQ | ||
| representation, compliant with the RocketMQ message format specification. | ||
|
|
||
| #### 3.1.4 Example | ||
|
|
||
| This example shows the binary mode mapping of an event into the RocketMQ message. | ||
| All other CloudEvents attributes are mapped to RocketMQ message property fields with prefix `CE_`. | ||
|
|
||
| Mind that `CE_` here does refer to the event data content carried in the payload. | ||
|
|
||
| ``` text | ||
| ------------------ Message ------------------- | ||
|
|
||
| Topic: mytopic | ||
|
|
||
|
|
||
| -------------- user properties --------------- | ||
|
|
||
| CE_contenttype: application/avro | ||
| CE_specversion: "0.1" | ||
| CE_type: "com.example.someevent" | ||
| CE_time: "2018-11-23T03:56:24Z" | ||
| CE_id: "1234-1234-1234" | ||
| CE_source: "/mycontext/subcontext" | ||
| .... further attributes ... | ||
|
|
||
| ------------------- value -------------------- | ||
|
|
||
| ... application data ... | ||
|
|
||
| ----------------------------------------------- | ||
| ``` | ||
|
|
||
| ### 3.2. Structured Content Mode | ||
|
|
||
| The [structured content mode](#32-structured-content-mode) keeps event metadata and | ||
| data together in the payload, allowing simple forwarding of the same event across | ||
| multiple routing hops, and across multiple transports. | ||
|
|
||
| #### 3.2.1. RocketMQ Content-Type | ||
|
|
||
| The [RocketMQ][RocketMQ] `CE_contenttype` property field MUST be set to the media type | ||
| of an event format. | ||
|
|
||
| Example for the JSON format: | ||
|
|
||
| ``` | ||
| CE_contenttype: application/cloudevents+json; charset=UTF-8 | ||
|
|
||
| ``` | ||
|
|
||
| #### 3.2.2. Event Data Encoding | ||
| The chosen event format defines how all attributes, including the payload, are represented. | ||
| And in RocketMQ Message Header, it describes what is the type of transport event. | ||
|
|
||
| The event metadata and data MAY then be rendered in accordance with the event format | ||
| specification and the resulting data becomes the payload. | ||
|
|
||
| #### 3.2.3. Metadata Headers | ||
|
|
||
| Implementations MAY include the same RocketMQ headers as defined for the binary mode. | ||
|
|
||
| #### 3.2.4. Example | ||
| This example shows a JSON event format encoded structured data event: | ||
|
|
||
| ``` text | ||
| ------------------ Message --------------------------- | ||
|
|
||
| Topic: mytopic | ||
|
|
||
| ------------------ user properties ------------------- | ||
|
|
||
| CE_contenttype: application/cloudevents+json; charset=UTF-8 | ||
|
|
||
| ------------------ value ----------------------------- | ||
|
|
||
| { | ||
| "cloudEventsVersion" : "0.1", | ||
| "eventType" : "com.example.someevent", | ||
|
|
||
| ... further attributes omitted ... | ||
|
|
||
| "data" : { | ||
| ... application data ... | ||
| } | ||
| } | ||
|
|
||
| ------------------------------------------------------ | ||
| ``` | ||
|
|
||
| ## 4. References | ||
|
|
||
| - [RocketMQ][RocketMQ] The RocketMQ Messaging System | ||
| - [RFC2046][RFC2046] Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types | ||
| - [RFC2119][RFC2119] Key words for use in RFCs to Indicate Requirement Levels | ||
| - [RFC3629][RFC3629] UTF-8, a transformation format of ISO 10646 | ||
| - [RFC7159][RFC7159] The JavaScript Object Notation (JSON) Data Interchange Format | ||
|
|
||
| [CE]: ./spec.md | ||
| [JSON-format]: ./json-format.md | ||
| [RocketMQ]: http://rocketmq.apache.org/ | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you make this a direct pointer to the messaging format and not the project? I can't seem to find the format itself in the docs.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hi,duglin,you can see our messaging format in github.The link is https://github.com/apache/rocketmq/blob/master/common/src/main/java/org/apache/rocketmq/common/message/Message.java.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Later,we will put the detailed messaging format on the direct pointer Link which is on our RocketMQ website. |
||
| [JSON-Value]: https://tools.ietf.org/html/rfc7159#section-3 | ||
| [RFC2046]: https://tools.ietf.org/html/rfc2046 | ||
| [RFC2119]: https://tools.ietf.org/html/rfc2119 | ||
| [RFC3629]: https://tools.ietf.org/html/rfc3629 | ||
| [RFC7159]: https://tools.ietf.org/html/rfc7159 | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This feels a bit odd to say we're going to define 2 modes but then say only one of them is supported. I'd prefer to not confuse people by only talking about the one we do support.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay,I will rewrite this paragraph.