What is a CloudEvent property vs Event Data?

[duglin]

I'd like to gather our thoughts around a very important topic for our spec... what are the CloudEvents properties for? Or, what is CloudEvents itself for?

This has come up in relation to people starting to use CloudEvents and in particular when they are adding their own extensions to the serialization. We have a whole separate discussion going on around how to serialize extensions, but this issue is more about what type of data should be put into extensions at all.

While on the one hand we can't stop any one from putting whatever they want into an extensions, we should at least provide some guidance as to how we, as spec authors, intended them to be used so that people are more likely to use the spec the way we intended, they're then (hopefully) more likely to get the interop we're hoping for, and are then less likely to break as we make changes to the spec in the future.

I think the high-order question I'm trying to ask is: when does data go into data and when does it go into a CloudEvents extension? And I think a related question is... what is the role of CloudEvents? Is it just something that's used to move an event from one system to another or is it actually the event itself that is presented to the application and therefore the distinction between the CloudEvent properties and the data properties is blurred?

I've been assuming that CloudEvents was more about dealing with moving of events and that the "real" data that the application cares about is in data. Comparing this to an HTTP webapp, while there may be reasons why my app might go looking at the HTTP headers, that's not the real purpose of its processing. Its real purpose is usually to process the HTTP body. So, in the CloudEvents case, I view adding application level data JUST to the CloudEvents properties as a mistake unless its being added just for routing purposes. For example #225 (comment) mentioning "memoryLimitInMB" in a CloudEvents property (while technically legal from a spec perspective) isn't really something we should be encouraging - that should be in data someplace.

But I'd really like to hear what other people think so we can prove some guidance to people on this topic in the primer.

[Vlaaaaaaad]

I should add some context for the linked comment.

TL;DR: exploring the idea of using CloudEvents not just as a envelope/ transport spec, but inside the application too; some issues arise and I really want to avoid another "here, you get this huuuge XML, work on it and do app logic" scenario.

I'm currently helping a very very early startup architect their whole stack. Their desire is to do serverless with AWS Lambda.

Now the question comes: how does one build a fully serverless service? Serverless Inc.'s framework was chosen, several services identified and sketched up in rough terms.

Now, as there will be several functions sending and working with messages an event format should be used. A custom structure can be used or CloudEvents could be used. Since using a custom structure is... a solved issue, the interesting proposition is to use CloudEvents. So, how does one use CloudEvents with Lambda?

Forwarding

Wanting to use CloudEvents with Lambda came from the event-driven nature of the app. A flow could look like:

• event is generated
• event is put in a queue
• event is picked up, some work is done on it and it's then sent to 3 other queues. Yes, the same event with the same say eventTime

Sure, some functions could generate a new event or be the last hop, but others would do some work on it and forward it for further processing.

Lambda and CloudEvents

Lambda functions usually have a function( context, event) prototype.
The context object has stuff like what version of the function is used, what the memory limit is, AWS credentials and some other goodies. I do care about this data, even if only for observability.
The event object differs based on the event source, but is usually made of the payload and relevant metadata.

I saw two options:

• merge the context and the event in a single CloudEvent and have the function work on that. Clean from an input point of view( everything it needs is pre-packaged in that CloudEvent, including env variables and HTTP headers and whatever -- this leads to easy testability), but disgusting if all that is done is to forcefully squeeze the data in a structure
• repackage the data in event in a CloudEvent, but keep context separate( no idea regarding structure)

Currently exploring the first option and added an example in #225 (comment).

I'd like to keep the payload very very small( just what happened) and then add metadata as some "extensions" or extra fields.

Say the event is that a new file was created by somebody. Now, that somebody could've used Congito credentials, vanilla AWS credentials, a certificate or whatever. This hints to extension so I jumped at the idea of using the extensions field in CloudEvents. Bad idea since the extensions field is serialized as HTTP headers in the HTTP transport binding.

Of course, what I wanted to put in extensions could as easily go in data with some data/extensions and data/payload separation. Not sure how that would work with signing and/ or encryption, but still.
Or extensions could be serialized as part of the body and application-properties could be serialized as headers -- this feels wrong tho.

Either way, this is still very much early planning and exploratory stuff. We're considering all ideas, be they good or bad. And I'm probably missing big pieces( haven't even gotten that deep into frontend-backend interaction -- REST and GraphQL sure but how about generating an event directly and using the say iOS AWS SDK to push it to Kinesis/ SNS).

---

What is clear is that CloudEvents usage was mostly focused on transport. Is using CloudEvents in an application a wise usecase? Should that be supported or should usage be limited to envelope/ transport?

[duglin]

@Vlaaaaaaad thanks a ton for the description of your scenario.

If I have this right, I think there are a couple of key things here:

• you want to transfer an event from one system to one, or more, other systems
• you want to encapsulate all of the data from the first system into one object so you only transfer a single object
• you're not sure you can, or should, touch the original event due to things like signing

I can see how using CloudEvents made sense and probably can work. Let me ask this:

• does the original event have all of the CloudEvent properties already? Meaning, does the original event already have something like "source" and "eventType"?
• if the CloudEvent properties were not available to you, could you still get your job done? Very much related to the previous question
• what if the calling of the other systems involved were simple function calls and there wasn't any transport-layer involved? If I assume you want a single arg on that function call what would that arg look like? Would you still choose a CloudEvent or would you create a struct what had 2 fields (context and event)? Or something else?

My initial reaction is that whatever that 'arg' would be should be what you put into CloudEvent's data

But I'm curious to know other people's view.

[Vlaaaaaaad]

@duglin

does the original event have all of the CloudEvent properties already? Meaning, does the original event already have something like "source" and "eventType"?

Yes, it does have all the properties. Events are either created as CloudEvents from the start or they are repackaged( for new file creation in S3, the plan is for the event to be caught by a Lambda which repackages it into a CloudEvent and sends it on its way).

if the CloudEvent properties were not available to you, could you still get your job done? Very much related to the previous question

Unlikely. The time, event type and so on are all necessary. They could be duplicated in the payload, but that would be silly.

what if the calling of the other systems involved were simple function calls and there wasn't any transport-layer involved? If I assume you want a single arg on that function call what would that arg look like? Would you still choose a CloudEvent or would you create a struct what had 2 fields (context and event)? Or something else?

It would still be a CloudEvent. The context would either be part of the CloudEvent and/ or external to the app( in a database, cache, whatever). Stateless architecture and all that.

---

It looks more and more as everything should be part of the data attribute with the data/extensions and data/payload separation. Some functions will just have get into the data to modify the data/extensions( or whatever it will be called) to add some more details.

Not sure how that will all work when the signing/ encryption will come into play, but we'll see. There was a plan to have some extensions/ payloads encrypted, but I think that will be scrapped for now.

[cneijenhuis]

I've been assuming that CloudEvents was more about dealing with moving of events and that the "real" data that the application cares about is in data.

This has been my assumption as well.

As an analogy, if I'm sending a package, I'm only putting information on the outside that is of interest to the delivery service: Address, Stamps (or another proof that I've paid), maybe a "Fragile" sticker.
I'm not putting information there that is useless for the delivery service, e.g. "it's a birthday present" or "it contains a book and a toy".

Some examples of properties that our (eCommerce) customer want to route on that are not part of the eventTopic or source (and I'd duplicate these between data and context):

• payments: The payment processor (e.g. Stripe or Braintree) or the payment method (e.g. CC or PayPal)
• products: Brand/Manufacturer, categories (e.g. My eBay sync code is only interested in /sale/clearance, because these are the only products I'm selling via eBay)
• orders: Country (if I have different processes per country/region), Shipping Provider

[markpeek]

I've been assuming that CloudEvents was more about dealing with moving of events and that the "real" data that the application cares about is in data.

This has been my assumption as well.

Agreed. CloudEvents should add enough metadata information around the event data to move and route the event.

As an analogy, if I'm sending a package...

While I like the analogy I think the example "it contains a book and a toy" is not correct. With eventType we are making a stronger statement of the contents of the data.

[cneijenhuis]

Yes, you're right, the example isn't very good. (Still, from my understanding, the eventType would be e.g. com.example.package.created or com.example.PackageShipped. But in theory, for a very specialized structure, you could imagine something like com.example.package.book.shipped.)

A better example may be "it contains a Harry Potter book and a Lego truck".

[duglin]

Anyone else have some thoughts on this?

[duglin]

The Primer includes this text:

---

CloudEvents, at its core, defines a set of metadata, called attributes, about the event being transferred between systems, and how those pieces of metadata should appear in that message. This metadata is meant to be the minimal set of information needed to route the request to the proper component and to facilitate proper processing of the event by that component. So, while this might mean that some of the application data of the event itself might be duplicated as part of the CloudEvent's set of attributes, this is to be done solely for the purpose of proper delivery, and processing, of the message. Data that is not intended for that purpose should instead be placed within the event (the data attribute) itself.

---

I think that provides the clarity I was looking for so I'm going to propose that we close this issue. Please speak up if you think otherwise.

[duglin]

on the call today we agreed to close this issue - we can reopen if new info comes up.