Kafka transport binding

Makefile

@@ -7,6 +7,6 @@ verify:
 	@echo Running the spec phrase checker:
 	@tools/verify-specs.sh -v spec.md documented-extensions.md json-format.md \
 		http-transport-binding.md http-webhook.md mqtt-transport-binding.md \
-		nats-transport-binding.md
+		nats-transport-binding.md kafka-transport-binding.md
 	@echo Running the doc phrase checker:
 	@tools/verify-docs.sh -v .

README.md

@@ -34,6 +34,7 @@ The following specifications are available:
 | **NATS Transport Binding** | - | [master](https://github.com/cloudevents/spec/blob/master/nats-transport-binding.md) |
 | **AMQP Event Format** | - | [master](https://github.com/cloudevents/spec/blob/master/amqp-format.md) |
 | **AMQP Transport Binding** | - | [master](https://github.com/cloudevents/spec/blob/master/amqp-transport-binding.md) |
+| **Kafka Transport Binding** | - | [master](https://github.com/cloudevents/spec/blob/master/kafka-transport-binding.md) |
 
 There is also the
 [CloudEvents documented extension attributes](https://github.com/cloudevents/spec/blob/master/documented-extensions.md)

kafka-transport-binding.md

@@ -0,0 +1,277 @@
+# Kafka Transport Binding for CloudEvents
+
+## Abstract
+
+The [Kafka][Kafka] Transport Binding for CloudEvents defines how events are
+mapped to [Kafka messages][Kafka-Message-Format].
+
+## Status of this document
+
+This document is a working draft.
+
+## Table of Contents
+
+1. [Introduction](#1-introduction)
+- 1.1. [Conformance](#11-conformance)
+- 1.2. [Relation to Kafka](#12-relation-to-kafka)
+- 1.3. [Content Modes](#13-content-modes)
+- 1.4. [Event Formats](#14-event-formats)
+- 1.5. [Security](#15-security)
+2. [Use of CloudEvents Attributes](#2-use-of-cloudevents-attributes)
+- 2.1. [contentType Attribute](#21-contenttype-attribute)
+- 2.2. [data Attribute](#22-data-attribute)
+3. [kafka Message Mapping](#3-kafka-message-mapping)
+- 3.2. [Binary Content Mode](#31-binary-content-mode)
+- 3.1. [Structured Content Mode](#32-structured-content-mode)
+4. [References](#4-references)
+
+## 1. Introduction
+
+[CloudEvents][CE] is a standardized and transport-neutral definition of the
+structure and metadata description of events. This specification defines how
+the elements defined in the CloudEvents specification are to be used in the
+Kafka protocol as [Kafka messages][Kafka-Message-Format] (aka Kafka records).
+
+### 1.1. Conformance
+
+The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
+"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be
+interpreted as described in [RFC2119][RFC2119].
+
+### 1.2. Relation to Kafka
+
+This specification does not prescribe rules constraining transfer or settlement
+of event messages with Kafka; it solely defines how CloudEvents are expressed
+in the Kafka protocol as [Kafka messages][Kafka-Message-Format].
+
+### 1.3. Content Modes
+
+The specification defines two content modes for transferring events:
+*structured* and *binary*.
+
+The *binary* mode *only* applies to Kafka 0.11.0.0 and above, because of Kafka
+0.10.x.x and below are lack of support for message level headers.
+
+In the *structured* content mode, event metadata attributes and event data are
+placed into the Kafka message value section
+using an [event format](#14-event-formats).
+
+In the *binary* content mode, the value of the event `data` attribute is placed
+into the Kafka message's value section as-is, with
+the `kafka_contentType` header value declaring its media type; all other event
+attributes are mapped to the Kafka message's
+[header section][Kafka-Message-Header].
+
+### 1.4. Event Formats
+
+Event formats, used with the *stuctured* content mode, define how an event is
+expressed in a particular data format. All implementations of this
+specification MUST support the [JSON event format][JSON-format].
+
+### 1.5. Security
+
+This specification does not introduce any new security features for Kafka, or
+mandate specific existing features to be used.
+
+## 2. Use of CloudEvents Attributes
+
+This specification does not further define any of the [CloudEvents][CE] event
+attributes.
+
+### 2.1. contentType Attribute
+
+The `contentType` attribute is assumed to contain a media-type expression
+compliant with [RFC2046][RFC2046].
+
+### 2.2. data Attribute
+
+The `data` attribute is assumed to contain opaque application data that is
+encoded as declared by the `contentType` attribute.
+
+An application is free to hold the information in any in-memory representation
+of its choosing, but as the value is transposed into Kafka as defined in this
+specification, core Kafka provides data available as a sequence of bytes.
+
+For instance, if the declared `contentType` is
+`application/json;charset=utf-8`, the expectation is that the `data` attribute
+value is made available as [UTF-8][RFC3629] encoded JSON text.
+
+## 3. Kafka Message Mapping
+
+With Kafka 0.11.0.0 and above, the content mode is chosen by the sender of the 
+event. Protocol usage patterns that might allow solicitation of events using a
+particular content mode might be defined by an application, but are not defined
+ here.
+
+The receiver of the event can distinguish between the two content modes by
+inspecting the `kafka_contentType` property of the Kafka message. If the value
+of the `kafka_contentType` property is prefixed with the CloudEvents media type
+`application/cloudevents`, indicating the use of a known [event
+format](#14-event-formats), the receiver uses *structured* mode, otherwise it
+defaults to *binary* mode.
+
+If a receiver finds a CloudEvents media type as per the above rule, but with an
+event format that it cannot handle, for instance
+`application/cloudevents+avro`, it MAY still treat the event as binary and
+forward it to another party as-is.
+
+### 3.1. Binary Content Mode
+
+The *binary* content mode accommodates any shape of event data, and allows for
+efficient transfer and without transcoding effort.
+
+#### 3.1.1. Kafka Content Type
+
+For the *binary* mode, the `kafka_contentType` property MUST be mapped directly
+to the CloudEvents `contentType` attribute.
+
+#### 3.1.2. Event Data Encoding
+
+The [`data` attribute](#22-data-attribute) byte-sequence MUST be used as the
+value of the Kafka message.
+
+#### 3.1.3. Metadata Headers
+
+All [CloudEvents][CE] attributes with exception of `data` MUST be individually
+mapped to and from the Header fields in the Kafka message.
+
+##### 3.1.3.1 User Property Names
+
+Cloud Event attributes are prefixed with "cloudEvents_" for use in the
+[message-headers][Kafka-Message-Header] section with exception of `contentType`
+MUST be `kafka_contentType`
+
+Examples:
+
+    * `eventTime` maps to `cloudEvents_eventTime`
+    * `eventID` maps to `cloudEvents_eventID`
+    * `cloudEventsVersion` maps to `cloudEvents_cloudEventsVersion`
+
+##### 3.1.3.2 User Property Values
+
+The value for each Kafka header is constructed from the respective
+header's Kafka representation, compliant with the [Kafka message
+format][Kafka-Message-Format] specification.
+
+
+#### 3.1.4 Examples
+
+This example shows the *binary* mode mapping of an event into the
+Kafka message. The CloudEvents `contentType` attribute is mapped to
+the Kafka `kafka_contentType` header; all other CloudEvents attributes
+are mapped to Kafka Header fields with prefix `cloudEvents_`.
+
+Mind that `cloudEvents_` here does refer to the event `data`
+content carried in the payload.
+
+``` text
+------------------ Message -------------------
+
+Topic Name: mytopic
+
+------------------- key ----------------------
+
+Key: mykey
+
+------------------ headers -------------------
+
+kafka_contentType: application/avro
+cloudEvents_cloudEventsVersion: "0.1"
+cloudEvents_eventType: "com.example.someevent"
+cloudEvents_eventTime: "2018-04-05T03:56:24Z"
+cloudEvents_eventID: "1234-1234-1234"
+cloudEvents_source: "/mycontext/subcontext"
+       .... further attributes ...
+
+------------------- value --------------------
+
+            ... application data ...
+
+-----------------------------------------------
+```
+
+### 3.2. Structured Content Mode
+
+The *structured* content mode keeps event metadata and data together in the
+payload, allowing simple forwarding of the same event across multiple routing
+hops, and across multiple transports.
+
+#### 3.2.1. Kafka Content-Type
+
+The [Kafka `kafka_contentType`] property field is set to the media type
+of an [event format](#14-event-formats).
+
+Example for the [JSON format][JSON-format]:
+
+``` text
+content-type: application/cloudevents+json; charset=UTF-8
+```
+
+#### 3.2.2. Event Data Encoding
+
+The chosen [event format](#14-event-formats) defines how all attributes,
+including the `data` attribute, are represented.
+
+The event metadata and data is then rendered in accordance with the event
+format specification and the resulting data becomes the Kafka application 
+[data][data] section.
+
+#### 3.2.3. Metadata Headers
+
+Implementations MAY include the same Kafka headers as defined for the
+[binary mode](#313-metadata-headers).
+
+#### 3.2.4 Examples
+
+This example shows a JSON event format encoded event:
+
+``` text
+------------------ Message -------------------
+
+Topic Name: mytopic
+
+------------------- key ----------------------
+
+Key: mykey
+
+------------------ headers -------------------
+
+kafka_contentType: application/cloudevents+json; charset=UTF-8
+
+------------------- value --------------------
+
+{
+    "cloudEventsVersion" : "0.1",
+    "eventType" : "com.example.someevent",
+
+    ... further attributes omitted ...
+
+    "data" : {
+        ... application data ...
+    }
+}
+
+-----------------------------------------------
+```
+
+## 4. References
+
+- [Kafka][Kafka] The distributed stream platform
+- [Kafka-Message-Format][Kafka-Message-Format] The Kafka format message
+- [RFC2046][RFC2046] Multipurpose Internet Mail Extensions (MIME) Part Two: 
+  Media Types
+- [RFC2119][RFC2119] Key words for use in RFCs to Indicate Requirement Levels
+- [RFC3629][RFC3629] UTF-8, a transformation format of ISO 10646
+- [RFC7159][RFC7159] The JavaScript Object Notation (JSON) Data Interchange
+  Format
+
+[CE]: ./spec.md
+[JSON-format]: ./json-format.md
+[Kafka]: https://kafka.apache.org
+[Kafka-Message-Format]: https://kafka.apache.org/documentation/#messageformat
+[Kafka-Message-Header]: https://kafka.apache.org/documentation/#recordheader
+[JSON-Value]: https://tools.ietf.org/html/rfc7159#section-3
+[RFC2046]: https://tools.ietf.org/html/rfc2046
+[RFC2119]: https://tools.ietf.org/html/rfc2119
+[RFC3629]: https://tools.ietf.org/html/rfc3629
+[RFC7159]: https://tools.ietf.org/html/rfc7159