From 8e74614cffddc7bd7172a7536321197ffcce23c0 Mon Sep 17 00:00:00 2001 From: Thomas Bouldin Date: Wed, 20 Jun 2018 13:40:24 -0700 Subject: [PATCH 1/4] Add sampling extension. This pull request supercedes #76 and builds upon #242. Per agreement at the f2f meeting 2018-06-15, this will be ratified as an extension but not currently as part of the core spec. The early inclusion of the spec and the ability for side-cars to impose sampling early in the pipeline make it reasonable to start with this as an extension. I know the original author hoped to make this a core feature, though there are two upsides: 1. Due to other decisions on 2018-06-15, this will have zero impact on JSON encoding (but will in Proto and may in SDKs) 2. Extensions now get a whole document. This lets us pitch sampling in more detail. Necessary additional changes: * Added `Integer` to our type system * Added convention for how scalar extensions should be documented. I'm not really excited about this proposal so others are welcome. Special thanks to @maplebed for kicking this effort off. Signed-off-by: Thomas Bouldin --- extensions.md | 9 +++++++-- extensions/sampled-rate.md | 36 ++++++++++++++++++++++++++++++++++++ spec.md | 7 ++++--- 3 files changed, 47 insertions(+), 5 deletions(-) create mode 100644 extensions/sampled-rate.md diff --git a/extensions.md b/extensions.md index 0e0ede1f9..235933dce 100644 --- a/extensions.md +++ b/extensions.md @@ -33,6 +33,11 @@ default placement for extensions, but an extension MAY require special representation when transported (e.g. tracing standards that require specific headers). -## Documented Extensions +As a convention, extensions of scalar types (`String`, `Binary`, `URI`, +`Number`) document their format with the attribute name `(self)` in the +documentation's "Attributes" section. -* [Distributed Tracing](extensions/distributed-tracing.md) \ No newline at end of file +## Known Extensions + +* [Distributed Tracing](extensions/distributed-tracing.md) +* [Sampling](extensions/sampled-rate.md) diff --git a/extensions/sampled-rate.md b/extensions/sampled-rate.md new file mode 100644 index 000000000..97946d6b5 --- /dev/null +++ b/extensions/sampled-rate.md @@ -0,0 +1,36 @@ +# Sampling extension + +There are many cases in an Event's life when a system (either the system +creating the event or a system transporting the event) may wish to only emit a +portion of the events that actually happened. In a high throughput system where +creating the event is costly, a system may wish to only create an event for +1/100 of the times that something happened. Additionally, during the +transmission of an event from the source to the eventual recipient, any step +along the way may choose to only pass along a fraction of the events it +receives. + +In order for the system receiving the event to understand what is actually +happening in the system that generated the event, information about how many +similar events may have happened must be included in the event itself. This +field provides a place for a system generating an event to indicate that the +emitted event represents a given number of other similar events. It also +provides a place for intermediary transport systems to modify the event when +they impose additional sampling. + +## Attributes + +### (self) +* Type: `Integer` +* Description: The rate at which this event has already been sampled. Represents + the number of similar events that happened but were not sent plus this event. + For example, if a system sees 30 occurrences and emits a single event, `rate` + would be 30 (29 not sent and 1 sent). + Consumers SHOULD assume a value of `1` when the extension is omitted. +* Constraints + * The rate MUST be positive. + +## Encoding + +### In-memory formats +The Sampling extension uses the key `sampledRate` for in-memory formats + diff --git a/spec.md b/spec.md index a7346981a..d6760ae9b 100644 --- a/spec.md +++ b/spec.md @@ -107,15 +107,16 @@ platform/vendor specific protocols (AWS Kinesis, Azure Event Grid). The following abstract data types are available for use in attributes. +- `Integer` - A 32-bit whole number. - `String` - Sequence of printable Unicode characters. - `Binary` - Sequence of bytes. -- `Map` - `String`-indexed dictionary of `Object`-typed values -- `Object` - Either a `String`, or a `Binary`, or a `Map` +- `Map` - `String`-indexed dictionary of `Object`-typed values. +- `Object` - Either a `String`, or a `Binary`, or a `Map`, or an `Integer`. - `URI` - String expression conforming to `URI-reference` as defined in [RFC 3986 ยง4.1](https://tools.ietf.org/html/rfc3986#section-4.1). - `Timestamp` - String expression as defined in - [RFC 3339](https://tools.ietf.org/html/rfc3339) + [RFC 3339](https://tools.ietf.org/html/rfc3339). This specification does not define numeric or logical types. From 60ad09d2cf6863fa4e3836a05b9da207f4b7df98 Mon Sep 17 00:00:00 2001 From: Thomas Bouldin Date: Wed, 25 Jul 2018 10:38:17 -0700 Subject: [PATCH 2/4] PR feedback; also add conttent to extensions.md Signed-off-by: Thomas Bouldin --- extensions.md | 10 +++++++--- extensions/sampled-rate.md | 13 ++++++++----- 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/extensions.md b/extensions.md index 235933dce..e919b3afa 100644 --- a/extensions.md +++ b/extensions.md @@ -31,11 +31,15 @@ decided by those representations. Transport bindings (e.g. [AMPQ](amqp-transport-binding.md), [NATS](nats-transport-binding.md)) provide default placement for extensions, but an extension MAY require special representation when transported (e.g. tracing standards that require -specific headers). +specific headers). Extension authors SHOULD only require special +representation in transport bindings where extensions integrate with +pre-existing specs; extensions with custom transport bindings are much +more likely to be dropped by middleware that does not understand the +extension. As a convention, extensions of scalar types (`String`, `Binary`, `URI`, -`Number`) document their format with the attribute name `(self)` in the -documentation's "Attributes" section. +`Number`) document their `Value` and structured types document their +`Attributes`. ## Known Extensions diff --git a/extensions/sampled-rate.md b/extensions/sampled-rate.md index 97946d6b5..7849b6396 100644 --- a/extensions/sampled-rate.md +++ b/extensions/sampled-rate.md @@ -17,20 +17,23 @@ emitted event represents a given number of other similar events. It also provides a place for intermediary transport systems to modify the event when they impose additional sampling. -## Attributes +## Value -### (self) * Type: `Integer` * Description: The rate at which this event has already been sampled. Represents the number of similar events that happened but were not sent plus this event. For example, if a system sees 30 occurrences and emits a single event, `rate` would be 30 (29 not sent and 1 sent). - Consumers SHOULD assume a value of `1` when the extension is omitted. + A value of `1` is the equivalent of this extension not being used at all. * Constraints - * The rate MUST be positive. + * The rate MUST be greater than zero. ## Encoding ### In-memory formats -The Sampling extension uses the key `sampledRate` for in-memory formats +The Sampling extension uses the key `sampledRate` for in-memory formats. + +### Transport bindings +The Sampling extension does not customize any transport binding's storage +for extensions. From c1c3ea4e4a95ccb210a75a932ec6289c461c405d Mon Sep 17 00:00:00 2001 From: Thomas Bouldin Date: Wed, 25 Jul 2018 11:23:08 -0700 Subject: [PATCH 3/4] =?UTF-8?q?@douglin=20doesn't=20like=20the=20world=20'?= =?UTF-8?q?may'=20=F0=9F=98=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Thomas Bouldin --- extensions/sampled-rate.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/extensions/sampled-rate.md b/extensions/sampled-rate.md index 7849b6396..4a96f0aa5 100644 --- a/extensions/sampled-rate.md +++ b/extensions/sampled-rate.md @@ -1,17 +1,17 @@ # Sampling extension There are many cases in an Event's life when a system (either the system -creating the event or a system transporting the event) may wish to only emit a -portion of the events that actually happened. In a high throughput system where -creating the event is costly, a system may wish to only create an event for -1/100 of the times that something happened. Additionally, during the +creating the event or a system transporting the event) might wish to only emit +a portion of the events that actually happened. In a high throughput system +where creating the event is costly, a system might wish to only create an event +for 1/100 of the times that something happened. Additionally, during the transmission of an event from the source to the eventual recipient, any step -along the way may choose to only pass along a fraction of the events it +along the way might choose to only pass along a fraction of the events it receives. In order for the system receiving the event to understand what is actually happening in the system that generated the event, information about how many -similar events may have happened must be included in the event itself. This +similar events happened would need to be included in the event itself. This field provides a place for a system generating an event to indicate that the emitted event represents a given number of other similar events. It also provides a place for intermediary transport systems to modify the event when From 9cdae21a311d185e19e58d3bc258e62bbb8782d8 Mon Sep 17 00:00:00 2001 From: Thomas Bouldin Date: Tue, 14 Aug 2018 17:55:30 -0700 Subject: [PATCH 4/4] Add 'e.g.' to list of scalar types in conventions Signed-off-by: Thomas Bouldin --- extensions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions.md b/extensions.md index e919b3afa..485953dc1 100644 --- a/extensions.md +++ b/extensions.md @@ -37,7 +37,7 @@ pre-existing specs; extensions with custom transport bindings are much more likely to be dropped by middleware that does not understand the extension. -As a convention, extensions of scalar types (`String`, `Binary`, `URI`, +As a convention, extensions of scalar types (e.g. `String`, `Binary`, `URI`, `Number`) document their `Value` and structured types document their `Attributes`.