Skip to content

Encoding & Transport Independent Interoperability Guarantee (JSON-based Size Measurement) #364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cneijenhuis wants to merge 6 commits into from
Closed

Encoding & Transport Independent Interoperability Guarantee (JSON-based Size Measurement) #364

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7c5d7e5
Size limits or guarantees
cneijenhuis Jan 9, 2019
f40f128
Settle on size guarantee; remove attribute limit
cneijenhuis Jan 23, 2019
a877004
Better headline
cneijenhuis Jan 24, 2019
3ae6f27
Remove message, replace with event
cneijenhuis Jan 24, 2019
a58c45d
Clarify relationship of event and message with an extra paragraph and…
cneijenhuis Jan 28, 2019
5913344
Lower size to 64KB, replace MUST with SHOULD
cneijenhuis Feb 7, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ This document is a working draft.
- [Type System](#type-system)
- [Context Attributes](#context-attributes)
- [Data Attribute](#data-attribute)
- [Minimum Supported Event Size](#minimum-supported-event-size)
- [Example](#example)

## Overview
Expand Down Expand Up @@ -289,6 +290,30 @@ encapsulated within the `data` attribute.
* Constraints:
* OPTIONAL

# Minimum Supported Event Size

In order to increase interoperability, all CloudEvent consumers SHOULD accept
events up to a size of 64KB, measured by serializing the CloudEvent as
[JSON](./json-format.md) (minified, i.e. without white-space).
Copy link
Copy Markdown
Contributor

@clemensv clemensv Feb 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't find the whole constraint after 64KB necessary, at all.

"All CloudEvent consumers SHOULD accept events up to an overall wire-frame (HTTP message, AMQP message, MQTT packet, etc.) size of 64Kbyte."

The publisher picks a protocol and encoding whatever comes out of that ought to fit into 64KByte. If the publisher picks MQTT and Protobuf fills that to 64Kbyte, and an intermediary wants to transcode into HTTP and JSON, that might result in a larger payload, but that's the intermediary's risk to deal with. And it might indeed not if you use a transfer encoding with compression. Minification might be strategy to make JSON a little smaller, but I don't think that requires mention here.

Copy link
Copy Markdown
Collaborator

@duglin duglin Feb 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cneijenhuis any thoughts on this?

Copy link
Copy Markdown
Contributor Author

@cneijenhuis cneijenhuis Feb 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but that's the intermediary's risk to deal with.

@clemensv That subverts the whole PR. It is actually the exact opposite of the goal that I had with opening this PR.

As a producer I want to create a CloudEvent that I know is supported in ANY protocol and encoding. And as an intermediary, I want to know whether I can forward a CloudEvent in any other transcoding.

The goal of CloudEvents is "to provide interoperability across services, platforms and systems." Therefore, in my opinion, we need to find a guarantee that is respected across services, platforms and systems.

Taking your example, if Foo publishes a CloudEvent with MQTT and Protobuf with <64KB, Bar transcodes it into HTTP and JSON and sends it to EventGrid (and the result is >=64KB), then EventGrid will not accept it. Therefore, Foo and EventGrid are not interoperable. Yet both would be, according to your proposal, valid CloudEvents implementations.

According to my proposal, the size of the CloudEvent in MQTT and Protobuf is irrelevant. There is a standardized way to measure a CloudEvent, and that always returns the same result, independently of service, platform or system. Therefore, if Foo publishes a CloudEvent that - according to this global measurement - SHOULD be accepted, Foo knows that it is interoperable with Bar, and Bar is interoperable with EventGrid.

Again, I'm open to discuss how the measurement is done (focussing solely on the size of the CloudEvent as encoded in JSON wasn't my initial proposal, but it is simple and straight forward, and that was valued in the discussions we had). But I believe the overarching goal of measuring the CloudEvent must be to provide for interoperability, and therefore we must measure the (abstract) event, and not the (concrete) message.


CloudEvent consumers MAY reject events above this size and MAY reject messages
that are not minified (e.g. contain unnecessary white-space).
Copy link
Copy Markdown
Contributor

@clemensv clemensv Feb 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the MAY is implied by the above and the "minified" clause adds nothing of value if the event is under the set threshold. Why should I reject an event that is within 64KByte, but isn't minified?

It is RECOMMENDED for CloudEvent producers to only create events within this
size, unless they can be sure all consumers support larger sizes.

The same event serialized with a different format will likely result in a
different size. However, to aid interoperability, only the minified
[JSON](./json-format.md) is used to measure the size. As an example, an
[AMQP](./amqp-format.md) message of 70KB MUST be accepted if the contained
event serialized as JSON is below 64KB. However, an [AMQP](./amqp-format.md)
message of 60KB MAY be rejected if the contained event serialized as JSON
exceeds 64KB. Practically, an [AMQP](./amqp-format.md) consumer that wants to
protect theirself from large messages MAY simply choose to reject messagess
after a higher limit (e.g. 256KB) that comfortably fits any valid event size.
However, a middleware that wants to guarantee that the event can be forwarded
in any format SHOULD measure the size of the event independently of the format
it was received in.
Copy link
Copy Markdown
Contributor

@clemensv clemensv Feb 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can deal with transcoding differences here, at all. The receiver gets an event and that SHOULD be within 64KByte wire-frame size (headers and payload and transport protocol overhead). When the event gets transcoded, it's up to the router/re-publisher to deal with how to keep the transcoded event under the threshold. It's assured that the unmodified event would fit, so that's a choice. I would drop that whole paragraph.


# Example

The following example shows a CloudEvent serialized as JSON:
Expand Down