HTTP Mapping Strawman

[clemensv]

Just to capture a straw-man for HTTP mappings for discussion purposes - not about the mapping per-se (that might be too early at this point) but to illustrate concrete usage of the properties in the core spec.

With the properties that are in the spec as of today, I'm going to show two candidate mappings for HTTP. One carries an opaque binary payload as-is with the data field being mapped to the HTTP payload, the other carries an event consisting of JSON data. I believe that both scenarios have legitimate use cases.

Upside of the binary mapping is that binary payloads can be routed w/o size penalty. Downside is that you'll have to juggle HTTP headers, which is painful in many environments. Upside up the JSON format is that you have it all in one object, but non-JSON data effectively needs to be treated as binary and base64 encoded.

JSON mapping with JSON data has priority for Microsoft.

Binary: Cloud events properties are mapped to HTTP headers. Content-Type is mapped to the native header.

POST {external} HTTP/1.1
CloudEvents-Namespace: azure.microsoft.com/storage
CloudEvents-EventType: Microsoft.Storage.BlobCreated
CloudEvents-EventTypeVersion: 2018-01-20
CloudEvents-Version: 0.2
CloudEvents-Topic: subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/xstoretestaccount
CloudEvents-Subject: /blobServices/default/containers/oc2d2817345i200097container/blobs/oc2d2817345i20002296blob
CloudEvents-EventId: 831e1650-001e-001b-66ab-eeb76e069631
CloudEvents-EventTime: 2017-06-26T18:41:00.9584103Z
**Content-Type**: application/x-my-opaque-binary
Content-Length: 1234

{Data}

---

JSON: Here, all the cloud events properties are mapped inline at the outer scope of a JSON object. The "data" object contains event-specific data, here an Azure blob store event. I think this will be the more common form.

POST {external} HTTP/1.1
Content-Type: application/json
Content-Length: nnnn

{
    "namespace" : "azure.com"
 /oc2d2817345i20002296blob",
    "cloud-events-version": "0.2"
    "event-type": "Microsoft.Storage.BlobCreated",
    "event-type-version": "1",
    "event-time": "2017-06-26T18:41:00.9584103Z",
    "event-id": "831e1650-001e-001b-66ab-eeb76e069631",
   "topic": "subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/xstoretestaccount",
    "subject": "/blobServices/default/containers/oc2d2817345i200097container/blobs
    "schema-url": "https://raw.githubusercontent.com/Azure/azure-rest-api-specs/master/specification/eventgrid/data-plane/Microsoft.Storage/stable/2018-01-01/Storage.json",
    "data": { 
      "api": "PutBlockList",
      "clientRequestId": "6d79dbfb-0e37-4fc4-981f-442c9ca65760",
      "requestId": "831e1650-001e-001b-66ab-eeb76e000000",
      "eTag": "0x8D4BCC2E4835CD0",
      "contentType": "application/octet-stream",
      "contentLength": 524288,
      "blobType": "BlockBlob",
      "url": "https://oc2d2817345i60006.blob.core.windows.net/oc2d2817345i200097container/oc2d2817345i20002296blob",
      "sequencer": "00000000000004420000000000028963",
      "storageDiagnostics": {
        "batchId": "b68529f3-68cd-4744-baa4-3c0498ec19f0"
      }
    }
  }
}

[yaronha]

@clemensv the JSON way creates significant overhead in many cases, e.g. when forwarding events you have to fully de-serialize & re-serialize the event vs peeking at the headers, also as you noted when using non JSON data like protobuf, Avro, XML, Image, .. you are forced to yet another expensive ser/des, in our API gateways which map cloud APIs to our native APIs the CPU is spending 80% of the time on JSON ser/des, i would rather have a greener approach with less energy waist :)

I also think we can minimize the header info or push some of the attributes to the event data portion, e.g. combine Namespace & Event-type, have a single source field for the ARN with further source details as part of the event message.

[clemensv]

The JSON way is also routable without touching it, whereas the HTTP header variation is not. I can receive the JSON event and stick it into an AMQP queue and process it from there without loss of fidelity. We have a ton of scenarios where event flows are more than one hop. There are pros and cons to both, and that's why I would allow for both mappings. The JSON mapping will be common across several transports for that reason; I can make that work for MQTT3 (has no metadata on the frame) and AMQP and HTTP and do hope-wise routing across those.

Azure Event Grid's primary cloudevents mapping will almost surely be the JSON flavor, because we accept, for instance, from HTTP and then route via AMQP into Event Hubs, to name one path.

I agree with the goal of consolidating the metadata as good as we can. "namespace" spans across all app-defined constructs, like the conventions for event-type and source and source-id (which I propose to rename to subject in #95), so I'm a bit skeptical of pulling that in with the event-type.

[yaronha]

@clemensv AMQP is a great example, it has Headers just like HTTP, you can use those for the context. we need to have the minimal possible context (not repeating the Event data), and place it in transport specific mechanism.

JSON limits cloud events to low performance use cases, we are working (with Azure as well, see: https://github.com/nuclio/demos/tree/master/fleet-alarm-detection-azure) on real-time stream processing and use machine optimized Ser/Des between functions (Capnproto/Flatbuf & Protobuf), in some cases pass images, this is how we achieve 400K events/sec/function. i dont want to be forced to do JSON ser/des for the encoded messages it will drive the performance down by an order of magnitude.

in future i can see how we do encrypted data payloads, yet another reason to separate the headers from data.

[duglin]

I'd like to see this issue as part of our v0.1 milestone - I think having the HTTP mapping will be needed for any kind of interop testing we'll do.

[duglin]

@clemensv wanna do a PR for this one?

[duglin]

@yaronha perhaps you could write-up what you'd like the HTTP mapping should look like too, then we could compare

[yaronha]

@duglin seems like @clemensv already described it under binary mapping (the first one), so do we need a separate PR? we can also list both options as valid (BIN & JSON), the only challenge with the JSON one is that you first need to de-serialize it for you to know its a valid one.

[duglin]

@yaronha oh ok. Sorry, I thought you were proposing an alternate solution. Yes, I do think we'll need a PR to turn it (or them) into a formal proposal for people to review.

[clemensv]

I'll write it up this week.

[duglin]

thanks!!

[duglin]

Agreed to add this to v0.1 milestone on 4/5 call