From ed9d498a7784d272d8564e7754a60110f2c7be84 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Sun, 14 Oct 2018 11:51:25 +1000 Subject: [PATCH 01/20] structured logging rfc --- rfcs/0000-structured-logging.md | 1748 +++++++++++++++++++++++++++++++ 1 file changed, 1748 insertions(+) create mode 100644 rfcs/0000-structured-logging.md diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md new file mode 100644 index 000000000..0d7db6d24 --- /dev/null +++ b/rfcs/0000-structured-logging.md @@ -0,0 +1,1748 @@ +# Summary +[summary]: #summary + +Add support for structured logging to the `log` crate in both `std` and `no_std` environments, allowing log records to carry typed data beyond a textual message. This document serves as an introduction to what structured logging is all about, and as an RFC for an implementation in the `log` crate. + +`log` will depend on `serde` for structured logging support, which will be enabled by default. See [the implications for `log` users](#implications-for-dependents) for some more details. The API is heavily inspired by the `slog` logging framework. + +> NOTE: Code in this RFC uses recent language features like `impl Trait`, but can be implemented without them. + +# Contents + +- [Motivation](#motivation) + - [What is structured logging?](#what-is-structured-logging) + - [Why do we need structured logging in `log`?](#why-do-we-need-structured-logging-in-log) +- [Guide-level explanation](#guide-level-explanation) + - [Capturing structured logs](#capturing-structured-logs) + - [Consuming structured logs](#consuming-structured-logs) +- [Reference-level explanation](#reference-level-explanation) + - [Design considerations](#design-considerations) + - [Implications for dependents](#implications-for-dependents) + - [Cargo features](#cargo-features) + - [Public API](#public-api) + - [`Record` and `RecordBuilder`](#record-and-recordbuilder) + - [`ToValue`](#tovalue) + - [`Value`](#value) + - [`ToKey`](#tokey) + - [`Key`](#key) + - [`Visitor`](#visitor) + - [`Error`](#error) + - [`KeyValueSource`](#keyvaluesource) + - [The `log!` macros](#the-log-macros) +- [Drawbacks](#drawbacks) + - [`Display + Serialize`](#display--serialize) + - [`serde`](#serde) +- [Rationale and alternatives](#rationale-and-alternatives) + - [Just use `Display`](#just-use-display) + - [Don't use a blanket implementation](#dont-use-a-blanket-implementation) + - [Define our own serialization trait](#define-our-own-serialization-trait) + - [Don't enable structured logging by default](#dont-enable-structured-logging-by-default) +- [Prior art](#prior-art) + - [Rust](#rust) + - [Go](#go) + - [.NET](#net) +- [Unresolved questions](#unresolved-questions) + +# Motivation +[motivation]: #motivation + +## What is structured logging? + +Information in log records can be traditionally captured as a blob of text, including a level, a message, and maybe a few other pieces of metadata. There's a lot of potentially valuable information we throw away when we format data as text. Arbitrary textual representations often result in log records that are neither easy for humans to read, nor for machines to parse. + +Structured logs can retain their original structure in a machine-readable format. They can be changed programmatically within a logging pipeline before reaching their destination. Once there, they can be analyzed using common database tools. + +As an example of structured logging, a textual log like this: + +``` +[INF 2018-09-27T09:32:03Z basic] [service: database, correlation: 123] Operation completed successfully in 18ms +``` + +could be represented as a structured log like this: + +```json +{ + "ts": 1538040723000, + "lvl": "INFO", + "msg": "Operation completed successfully in 18ms", + "module": "basic", + "service": "database", + "correlation": 123, + "took": 18 +} +``` + +When log records are kept in a format like this, potentially interesting queries like _what are all records where the correlation is 123?_, or _how many errors were there in the last hour?_ can be computed efficiently. + +Even when logging to a console for immediate consumption, the human-readable message can be presented better when it's not trying to include ambient metadata inline: + +``` +[INF 2018-09-27T09:32:03Z] Operation completed successfully in 18ms +module: "basic" +service: "database" +correlation: 123 +took: 18 +``` + +Having a way to capture additional metadata is good for human-centric formats. Having a way to retain the structure of that metadata is good for machine-centric formats. + +## Why do we need structured logging in `log`? + +Why add structured logging support to the `log` crate when libraries like `slog` already exist and support it? `log` needs to support structured logging to make the experience of using `slog` and other logging tools in the Rust ecosystem more compatible. + +On the surface there doesn't seem to be a lot of difference between `log` and `slog`, so why not just deprecate one in favour of the other? Conceptually, `log` and `slog` are different libraries that fill different use-cases, even if there's some overlap. + +`slog` is a logging _framework_. It offers all the fundamental tools needed out-of-the-box to capture log records, define and implement the composable pieces of a logging pipeline, and pass them through that pipeline to an eventual destination. It has conventions and trade-offs baked into the design of its API. Loggers are treated explicitly as values in data structures and as arguments, and callers can control whether to pass owned or borrowed data. + +`log` is a logging _facade_. It's only concerned with a standard, minimal API for capturing log records, and surfacing those records to some consumer. The tools provided by `log` are only those that are fundamental to the operation of the `log!` macro. From `log`'s point of view, a logging framework like `slog` is a black-box implementation of the `Log` trait. In this role, the `Log` trait can act as a common entrypoint for capturing log records. That means the `Record` type can act as a common container for describing a log record. `log` has its own set of trade-offs baked into the design of its API. The `log!` macro assumes a single, global entrypoint, and all data in a log record is borrowed from the callsite. + +A healthy logging ecosystem needs both `log` and frameworks like `slog`. As a standard API, `log` can support a diverse but cohesive ecosystem of logging tools in Rust by acting as the glue between libraries, frameworks, and applications. A lot of libraries already depend on it. In order to really fulfil this role though, `log` needs to support structured logging so that libraries and their consumers can take advantage of it in a framework-agnostic way. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +## Capturing structured logs + +Structured logging is supported in `log` by allowing typed key-value pairs to be associated with a log record. + +### Structured vs unstructured + +A `;` separates structured key-value pairs from values that are replaced into the message: + +```rust +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + correlation = correlation_id, + user +); +``` + +Any `value` or `key = value` expressions before the `;` in the macro will be interpolated into the message as unstructured text. This is the `log!` macro we have today. Any `value` or `key = value` expressions after the `;` will be captured as structured key-value pairs. These structured key-value pairs can be inspected or serialized, retaining some notion of their original type. That means in the above example, the `message` key is unstructured, and the `correlation` and `user` keys are structured: + +``` +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + ^^^^^^^^^^^^^^^^^^^ + unstructured + + correlation = correlation_id, + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + structured + + user + ^^^^ + structured +); +``` + +Any type that implements both `Display` and `serde::Serialize` can be used as the value in a structured key-value pair. In the previous example, the values used in the macro require the following trait bounds: + +``` +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + ^^^^^^^^^ + Display + + correlation = correlation_id, + ^^^^^^^^^^^^^^ + Display + Serialize + + user + ^^^^ + Display + Serialize +); +``` + +### Logging data that isn't `Display + Serialize` + +The `Display + Serialize` bounds we've been using thus far are mostly accurate, but don't tell the whole story. Structured values don't _technically_ require `Display + Serialize`. They require a trait for capturing structured values, `ToValue`, which we implement for any type that also implements `Display + Serialize`. So in truth the trait bounds required by the macro look like this: + +``` +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + ^^^^^^^^^ + Display + + correlation = correlation_id, + ^^^^^^^^^^^^^^ + ToValue + + user + ^^^^ + ToValue +); +``` + +The `ToValue` trait makes it possible for types to be logged even if they don't implement `Display + Serialize`; they only need to implement `ToValue`. One of the goals of this RFC is to avoid creating another fundamental trait that needs to be implemented throughout the ecosystem though. So instead of implementing `ToValue` directly, we provide a few helper macros to wrap a value satisfying one set of trait bounds into a value that satisfies `ToValue`. The following example uses these helper macros to change the trait bounds required by `correlation_id` and `user`: + +```rust +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + correlation = log_fmt!(correlation_id, Debug::fmt), + user = log_serde!(user) +); +``` + +They now look like this: + +``` +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + ^^^^^^^^^ + Display + + correlation = log_fmt!(correlation_id, Debug::fmt), + ^^^^^^^^^^^^^^ + Debug + + user = log_serde!(user) + ^^^^ + Serialize +); +``` + +This same pattern can be applied to other types in the standard library and wider ecosystem that are likely to be logged, but don't typically satisfy `Display + Serialize`. Some examples are `Path`, and implementations of `Error` or `Fail`. + +So why do we use `Display + Serialize` as the effective bounds in the first place? Practically, it's required to keep a consistent API in `no_std` environments, where an object-safe wrapper over `serde` requires standard library features. `Display` is a natural trait to lean on when `Serialize` isn't available, so requiring both means there's no change in the trait bounds when records can be captured using `serde` and when they can't (later on we'll see how we can still use `serde` for primitive types in `no_std` environments). + +`Display` also suggests the data we are logging should have some canonical representation that's useful for someone to look at. Finding small but sufficient representations of potentially large pieces of state to log will make those logs easier to ingest and analyze later. + +## Consuming structured logs + +Capturing structured logs is only half the story. Implementors of the `Log` trait also need to be able to work with any key-value pairs associated with a log record. + +### Using `Visitor` to print or serialize key-value pairs + +Structured key-value pairs can be inspected using the `Visitor` trait. Take the terminal log format from before: + +``` +[INF 2018-09-27T09:32:03Z] Operation completed successfully in 18ms +module: "basic" +service: "database" +correlation: 123 +took: 18 +``` + +Each key-value pair, shown as `$key: $value`, can be written using a `Visitor` that hands values to a `serde::Serializer`. The implementation of that `Visitor` could look like this: + +```rust +fn write_pretty(w: impl Write, r: &Record) -> io::Result<()> { + // Write the first line of the log record + ... + + // Write each key-value pair using the `WriteKeyValues` visitor + record + .key_values() + .visit(&mut WriteKeyValues(w)) + .map_err(|e| io::Error::new(io::ErrorKind::Other, e.into_error())) +} + +struct WriteKeyValues(W); + +impl<'kvs, W> Visitor<'kvs> for WriteKeyValues +where + W: Write, +{ + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { + // Write the key + // `Key` is a wrapper around a string + // It can be formatted directly + write!(&mut self.0, "{}: ", k)?; + + // Write the value + // `Value` is a wrapper around `serde::Serialize` + // It can't be formatted directly + v.serialize(&mut Serializer::new(&mut self.0))?; + + Ok(()) + } +} +``` + +Needing `serde` for the human-centric format above seems a bit unnecessary. The value becomes clearer when using a machine-centric format for the entire log record, like json. Take the following json format: + +```json +{ + "ts": 1538040723000, + "lvl": "INFO", + "msg": "Operation completed successfully in 18ms", + "module": "basic", + "service": "database", + "correlation": 123, + "took": 18 +} +``` + +Defining a serializable structure for this format could be done using `serde_derive`, and then written using `serde_json`: + +```rust +fn write_json(w: impl Write, r: &Record) -> io::Result<()> { + let r = SerializeRecord { + lvl: r.level(), + ts: epoch_millis(), + msg: r.args().to_string(), + props: r.key_values().serialize_as_map(), + }; + + serde_json::to_writer(w, &r)?; + + Ok(()) +} + +#[derive(Serialize)] +struct SerializeRecord { + lvl: Level, + ts: u64, + msg: String, + #[serde(flatten)] + props: KVS, +} +``` + +There's no explicit `Visitor` in this case, because the `serialize_as_map` method wraps one up internally so all key-value pairs are serializable as a map using `serde`. + +### Using `KeyValueSource` to capture key-value pairs + +What exactly are the key-value pairs on a record? The previous examples used a `key_values()` method on `Record` to get _something_ that could be visited or serialized using a `Visitor`. That something is an implementation of a trait, `KeyValueSource`, which holds the actual `Key` and `Value` pairs: + +``` +fn write_pretty(w: impl Write, r: &Record) -> io::Result<()> { + ... + + record + .key_values() + ^^^^^^^^^^^^^ + `Record::key_values` returns `impl KeyValueSource` + + .visit(&mut WriteKeyValues(w)) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + `KeyValueSource::visit` takes `&mut Visitor` +} +``` + +As an example of a `KeyValueSource`, the `log!` macros can capture key-value pairs into an array of `(&str, &dyn ToValue)` tuples that can be visited in sequence: + +```rust +impl<'a> KeyValueSource for [(&'a str, &'a dyn ToValue)] { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) { + for &(k, v) in self { + visitor.visit_pair(k.to_key(), v.to_value())?; + } + + Ok(()) + } +} +``` + +A `KeyValueSource` doesn't have to just contain key-value pairs directly like this though. It could also act like an adapter, like we have for iterators in the standard library. As another example, the following `KeyValueSource` doesn't store any key-value pairs of its own, it will sort and de-duplicate pairs read from another source by first reading them into a map before forwarding them on: + +```rust +pub struct SortRetainLast(KVS); + +impl KeyValueSource for SortRetainLast +where + KVS: KeyValueSource, +{ + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) { + // `Seen` is a visitor that will capture key-value pairs + // in a `BTreeMap`. We use it internally to sort and de-duplicate + // the key-value pairs that `SortRetainLast` is wrapping. + struct Seen<'kvs>(BTreeMap, Value<'kvs>>); + + impl<'kvs> Visitor<'kvs> for Seen<'kvs> { + fn visit_pair<'vis>(&'vis mut self, k: Key<'kvs>, v: Value<'kvs>) { + self.0.insert(k, v); + } + } + + // Visit the inner source and collect its key-value pairs into `seen` + let mut seen = Seen(BTreeMap::new()); + self.0.visit(&mut seen); + + // Iterate through the seen key-value pairs in order + // and pass them to the `visitor`. + for (k, v) in seen.0 { + visitor.visit_pair(k, v); + } + } +} +``` + +This API is similar to `serde` and `slog`, and is very flexible. Other structured logging concepts like contextual logging, where a record is enriched with information from its envirnment, can be built on top of `KeyValueSource` and `Visitor`. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Design considerations + +### Don't break anything + +Allow structured logging to be added in the current `0.4.x` series of `log`. + +### Leverage the ecosystem + +Rather than trying to define our own serialization trait and require libraries in the ecosystem implement it, we leverage `serde`. Any new types that emerge in the ecosystem don't have another fundamental trait they need to think about. + +### Object safety + +`log` is already designed to be object-safe so this new structured logging API needs to be object-safe too. + +### Borrowed vs owned + +`Record` borrows all data from the call-site so records need to be handled directly on-thread as they're produced. On the one hand that means that log records need to be serialized before they can be sent across threads. On the other hand it means callers don't need to make assumptions about whether records need to be owned or borrowed. + +## Implications for dependents + +Dependents of `log` will notice the following: + +In `no_std` environments (which is the default for `log`): + +- `serde` will enter the `Cargo.lock` if it wasn't there already. This will impact compile-times. +- Artifact size of `log` will increase. + +In `std` environments (which is common when using `env_logger` and other crates that implement `log`): + +- `serde` and `erased-serde` will enter the `Cargo.lock` if it wasn't there already. This will impact compile-times. +- Artifact size of `log` will increase. + +In either case, `serde` will become a public dependency of the `log` crate, so any breaking changes to `serde` will result in breaking changes to `log`. + +## Cargo features + +Structured logging will be supported in either `std` or `no_std` contexts using Cargo features: + +```toml +[features] +# support structured logging by default +default = ["structured"] + +# semantic name for the structured logging feature +structured = ["serde"] + +# when `std` is available, always support structured logging +# with `erased-serde` +std = ["structured", "erased-serde"] +``` + +Using default features, structured logging will be supported by `log` in `no_std` environments. Structured logging will always be available when using the `std` feature (usually pulled in by libraries that implement the `Log` trait). + +## Public API + +For context, ignoring the `log!` macros, this is roughly the additional public API this RFC proposes to support structured logging: + +```rust +impl<'a> RecordBuilder<'a> { + /// Set the key-value pairs on a log record. + #[cfg(feature = "serde")] + pub fn key_values(&mut self, kvs: ErasedKeyValues<'a>) -> &mut RecordBuilder<'a>; +} + +impl<'a> Record<'a> { + /// Get the key-value pairs. + #[cfg(feature = "serde")] + pub fn key_values(&self) -> ErasedKeyValues; + + /// Get a builder that's preconfigured from this record. + pub fn to_builder(&self) -> RecordBuilder; +} + +#[cfg(features = "serde")] +pub mod key_values { + /// A visitor for a set of key-value pairs. + /// + /// The visitor is driven by an implementation of `KeyValueSource`. + /// The visitor expects keys and values that satisfy a given lifetime. + pub trait Visitor<'kvs> { + /// Visit a single key-value pair. + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; + } + + /// A source for key-value pairs. + pub trait KeyValueSource { + /// Serialize the key value pairs. + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; + + /// Erase this `KeyValueSource` so it can be used without + /// requiring generic type parameters. + fn erase(&self) -> ErasedKeyValueSource + where + Self: Sized {} + + /// Find the value for a given key. + /// + /// If the key is present multiple times, this method will + /// return the *last* value for the given key. + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: ToKey {} + + /// An adapter to borrow self. + fn by_ref(&self) -> &Self {} + + /// Chain two `KeyValueSource`s together. + fn chain(self, other: KVS) -> Chain + where + Self: Sized {} + + /// Apply a function to each key-value pair. + fn try_for_each(self, f: F) -> Result<(), Error> + where + Self: Sized, + F: FnMut(Key, Value) -> Result<(), E>, + E: Into {} + + /// Serialize the key-value pairs as a map. + fn serialize_as_map(self) -> SerializeAsMap + where + Self: Sized {} + } + + /// An erased `KeyValueSource`. + pub struct ErasedKeyValueSource<'a> {} + + impl<'a> ErasedKeyValueSource<'a> { + /// Capture a `KeyValueSource` and erase its concrete type. + pub fn new(kvs: &'a impl KeyValueSource) -> Self {} + } + + impl<'a> Clone for ErasedKeyValueSource<'a> {} + impl<'a> Default for ErasedKeyValueSource<'a> {} + impl<'a> KeyValueSource for ErasedKeyValueSource<'a> {} + + /// A `KeyValueSource` adapter that visits key-value pairs + /// in sequence. + /// + /// This is the result of calling `chain` on a `KeyValueSource`. + pub struct Chain {} + + impl KeyValueSource for Chain + where + A: KeyValueSource, + B: KeyValueSource {} + + /// A `KeyValueSource` adapter that can be serialized as + /// a map using `serde`. + /// + /// This is the result of calling `serialize_as_map` on + /// a `KeyValueSource`. + pub struct SerializeAsMap {} + + impl Serialize for SerializeAsMap + where + KVS: KeyValueSource {} + + impl KeyValueSource for (K, V) + where + K: ToKey, + V: ToValue {} + + impl KeyValueSource for [KVS] + where + KVS: KeyValueSource {} + + #[cfg(feature = "std")] + impl KeyValueSource for Vec + where + KVS: KeyValueSource {} + + #[cfg(feature = "std")] + impl KeyValueSource for BTreeMap + where + K: Borrow + Ord, + V: ToValue {} + + #[cfg(feature = "std")] + impl KeyValueSource for HashMap + where + K: Borrow + Eq + Hash, + V: ToValue {} + + /// A type that can be converted into a borrowed key. + pub trait ToKey { + /// Perform the conversion. + fn to_key(&self) -> Key; + } + + impl ToKey for str {} + + #[cfg(feature = "std")] + impl ToKey for String {} + + #[cfg(feature = "std")] + impl<'a> ToKey for Cow<'a, str> {} + + /// A key in a key-value pair. + /// + /// The key can be treated like `&str`. + pub struct Key<'kvs> {} + + impl<'kvs> Key<'kvs> { + /// Get a key from a borrowed string. + pub fn from_str(key: &'a (impl AsRef + ?Sized)) -> Self; + + /// Get a reference to the key as a string. + pub fn as_str(&self) -> &str; + } + + impl<'kvs> Serialize for Key<'kvs> {} + impl<'kvs> PartialEq for Key<'kvs> {} + impl<'kvs> Eq for Key<'kvs> {} + impl<'kvs> PartialOrd for Key<'kvs> {} + impl<'kvs> Ord for Key<'kvs> {} + impl<'kvs> Hash for Key<'kvs> {} + impl<'kvs> AsRef for Key<'kvs> {} + + #[cfg(feature = "std")] + impl<'kvs> Borrow for Key<'kvs> {} + + /// A type that can be converted into a borrowed value. + pub trait ToValue { + /// Perform the conversion. + fn to_value(&self) -> Value; + } + + impl ToValue for T + where + T: Display + Serialize {} + + /// A value in a key-value pair. + /// + /// The value can be treated like `serde::Serialize`. + pub struct Value<'kvs> {} + + impl<'kvs> Value<'kvs> { + /// Get a value that will choose either the `Display` + /// or `Serialize` implementation based on the platform. + /// + /// If the standard library is available, the `Serialize` + /// implementation will be used. If the standard library + /// is not available, the `Display` implementation will + /// probably be used. + pub fn new(v: &'kvs (impl Display + Serialize)) -> Self; + + /// Get a value that can be serialized as a string using + /// its `Display` implementation. + pub fn from_display(v: &'kvs impl Display) -> Self; + + /// Get a value that can be serialized as structured + /// data using its `Serialize` implementation. + #[cfg(feature = "std")] + pub fn from_serde(v: &'kvs impl Serialize) -> Self; + } + + impl<'kvs> Serialize for Value<'kvs> {} + impl<'kvs> ToValue for Value<'kvs> {} + impl<'a, 'kvs> ToValue for &'a Value<'kvs> {} + + /// An error encountered while visiting key-value pairs. + pub struct Error {} + + impl Error { + /// Create an error from a static message. + pub fn msg(msg: &'static str) -> Self {} + + /// Get a reference to a standard error. + #[cfg(feature = "std")] + pub fn as_error(&self) -> &(dyn std::error::Error + Send + Sync + 'static) {} + + /// Convert into a standard error. + #[cfg(feature = "std")] + pub fn into_error(self) -> Box {} + + /// Convert into a `serde` error. + pub fn into_serde(self) -> E + where + E: serde::ser::Error {} + } + + impl From for Error + where + E: std::error::Error {} + + impl From for Box {} + impl AsRef for Error {} +} +``` + +### `Record` and `RecordBuilder` + +Structured key-value pairs can be set on a `RecordBuilder`: + +```rust +impl<'a> RecordBuilder<'a> { + /// Set key values + #[cfg(feature = "serde")] + pub fn key_values(&mut self, kvs: ErasedKeyValueSource<'a>) -> &mut RecordBuilder<'a> { + self.record.kvs = kvs; + self + } +} +``` + +These key-value pairs can then be accessed on the built `Record`: + +```rust +#[derive(Clone, Debug)] +pub struct Record<'a> { + ... + + #[cfg(feature = "serde")] + kvs: ErasedKeyValueSource<'a>, +} + +impl<'a> Record<'a> { + /// The key value pairs attached to this record. + /// + /// Pairs aren't guaranteed to be unique (the same key may be repeated with different values). + #[cfg(feature = "serde")] + pub fn key_values(&self) -> ErasedKeyValueSource { + self.kvs.clone() + } +} +``` + +### `ToValue` + +A `ToValue` is a potentially long-lived structure that can be converted into a `Value`: + +```rust +/// A type that can be converted into a borrowed value. +pub trait ToValue { + /// Perform the conversion. + fn to_value(&self) -> Value; +} + +impl<'a> ToValue for &'a dyn ToValue { + fn to_value(&self) -> Value { + (*self).to_value() + } +} +``` + +#### Implementors + +`ToValue` requires a blanket implementation to be most useful. This covers any `V: Display + Serialize`: + +```rust +impl ToValue for T { + fn to_value(&self) -> Value { + Value::new(self) + } +} +``` + +### `Value` + +A `Value` is a short-lived structure that can be serialized using `serde`. This might require losing some type information about the underlying value and serializing it as a string: + +```rust +/// A value in a key-value pair. +/// +/// The value can be treated like `serde::Serialize`. +pub struct Value<'kvs> { + inner: ValueInner<'kvs>, +} + +#[derive(Clone, Copy)] +enum ValueInner<'kvs> { + /// The value will be serialized as a string + /// using its `Display` implementation. + Display(&'kvs dyn fmt::Display), + /// The value will be serialized as a structured + /// type using its `Serialize` implementation. + #[cfg(feature = "erased-serde")] + Serde(&'kvs dyn erased_serde::Serialize), + /// The value will be serialized as a structured + /// type using its primitive `Serialize` implementation. + #[cfg(not(feature = "erased-serde"))] + Primitive(&'kvs dyn ToPrimitive), +} + +impl<'kvs> ToValue for Value<'kvs> { + fn to_value(&self) -> Value { + Value { inner: self.inner } + } +} + +impl<'a, 'kvs> ToValue for &'a Value<'kvs> { + fn to_value(&self) -> Value { + Value { inner: self.inner } + } +} + +impl<'kvs> Serialize for Value<'kvs> { + fn serialize(&self, serializer: S) -> Result + where + S: Serializer, + { + match self.inner { + ValueInner::Display(v) => serializer.collect_str(&v), + + #[cfg(feature = "erased-serde")] + ValueInner::Serde(v) => v.serialize(serializer), + + #[cfg(not(feature = "erased-serde"))] + ValueInner::Primitive(v) => { + // We expect `Value::new` to correctly determine + // whether or not a value is a simple primitive + let v = v + .to_primitive() + .ok_or_else(|| S::Error::custom("captured value is not primitive"))?; + + v.serialize(serializer) + }, + } + } +} +``` + +#### Capturing values + +Methods on `Value` allow it to capture and erase types that implement combinations of `Serialize` and `Display`: + +```rust +impl<'kvs> Value<'kvs> { + /// Create a new value. + /// + /// The value must implement both `serde::Serialize` and `fmt::Display`. + /// Either implementation will be used depending on whether the standard + /// library is available, but is exposed through the same API. + /// + /// In environments where the standard library is available, the `Serialize` + /// implementation will be used. + /// + /// In environments where the standard library is not available, some + /// primitive stack-based values can retain their structure instead of falling + /// back to `Display`. + pub fn new(v: &'kvs (impl Serialize + Display)) -> Self { + Value { + inner: { + #[cfg(feature = "erased-serde")] + { + ValueInner::Serde(v) + } + + #[cfg(not(feature = "erased-serde"))] + { + // Try capture a primitive value + if v.to_primitive().is_some() { + ValueInner::Primitive(v) + } else { + ValueInner::Display(v) + } + } + } + } + } + + /// Get a `Value` from a displayable reference. + pub fn from_display(v: &'kvs impl Display) -> Self { + Value { + inner: ValueInner::Display(v), + } + } + + /// Get a `Value` from a serializable reference. + #[cfg(feature = "erased-serde")] + pub fn from_serde(v: &'kvs impl Serialize) -> Self { + Value { + inner: ValueInner::Serde(v), + } + } +} +``` + +`Value::new` will choose either the `Serialize` or `Display` implementation, depending on whether `std` is available. If it is, `Value::new` will use `Serialize` and is equivalent to `Value::from_serde`, if it's not `Value::new` will use `Display` and is equivalent to `Value::from_display`. + +`Value::new` can use the `Serialize` implementation for some fixed set of primitives, like `i32` and `bool` even if `std` is not available though. This can be done by capturing those values at the point that the serializer is known into a `Primitive` wrapper: + +```rust +/// Convert a value into a primitive with a known type. +/// +/// The `ToPrimitive` trait lets us pass trait objects around +/// that are always the same size, rather than bloating values +/// to the size of the largest primitive. +pub trait ToPrimitive { + /// Perform the conversion. + fn to_primitive(&self) -> Option; +} + +impl ToPrimitive for T +where + T: Serialize, +{ + fn to_primitive(&self) -> Option { + self.serialize(PrimitiveSerializer).ok() + } +} + +#[derive(Clone, Copy)] +pub struct Primitive(PrimitiveInner); + +#[derive(Clone, Copy)] +enum PrimitiveInner { + Unsigned(u64), + Signed(i64), + Float(f64), + Bool(bool), + Char(char), + + #[cfg(feature = "i128")] + BigUnsigned(u128), + + #[cfg(feature = "i128")] + BigSigned(i128), +} + +impl Serialize for Primitive {} + +struct PrimitiveSerializer; + +impl Serializer for PrimitiveSerializer { + type Ok = Primitive; + type Error = Invalid; + + ... +} +``` + +The `Primitive` type stored in a `Value` is another trait object. This is just to ensure the size of `Value` doesn't grow to the size of `Primitive`'s largest variant, which is a 128bit number. + +The `Value::from_serde` method requires `std` because it uses `erased_serde` as an object-safe wrapper around `serde`, which itself requires `std`. + +#### Ownership + +The `Value` type borrows from its inner value. + +#### Thread-safety + +The `Value` type doesn't try to guarantee that values are `Send` or `Sync`, and doesn't offer any way of retaining that information when erasing. + +### `ToKey` + +A `ToKey` is a potentially long-lived structure that can be converted into a `Key`: + +```rust +/// A type that can be converted into a borrowed key. +pub trait ToKey { + /// Perform the conversion. + fn to_key(&self) -> Key; +} + +impl<'a, K: ?Sized> ToKey for &'a K +where + K: ToKey, +{ + fn to_key(&self) -> Key { + (*self).to_key() + } +} +``` + +#### Implementors + +`ToKey` is implemented for `str`. This is supported in both `std` and `no_std` contexts: + +```rust +impl ToKey for str { + fn to_key(&self) -> Key { + Key::from_str(self) + } +} +``` + +When `std` is available, `Key` is also implemented for other string containers: + +```rust +#[cfg(feature = "std")] +impl ToKey for String { + fn to_key(&self) -> Key { + Key::from_str(self) + } +} + +#[cfg(feature = "std")] +impl<'a> ToKey for borrow::Cow<'a, str> { + fn to_key(&self) -> Key { + Key::from_str(self.as_ref()) + } +} +``` + +### `Key` + +A `Key` is a short-lived structure that can be represented as a UTF-8 string. This might be possible without allocating, or it might require a destination to write into: + +```rust +/// A key in a key-value pair. +/// +/// The key can be treated like `&str`. +#[derive(PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Key<'kvs> { + inner: &'kvs str, +} + +impl<'kvs> ToKey for Key<'kvs> { + fn to_key(&self) -> Key { + Key { inner: self.inner } + } +} + +impl<'kvs> Key<'kvs> { + /// Get a `Key` from a borrowed string. + pub fn from_str(key: &'kvs (impl AsRef + ?Sized)) -> Self { + Key { + inner: key.as_ref(), + } + } + + /// Get a borrowed string from a `Key`. + pub fn as_str(&self) -> &str { + &self.inner + } +} + +impl<'kvs> AsRef for Key<'kvs> { + fn as_ref(&self) -> &str { + self.as_str() + } +} + +#[cfg(feature = "std")] +impl<'kvs> Borrow for Key<'kvs> { + fn borrow(&self) -> &str { + self.as_str() + } +} + +impl<'kvs> Serialize for Key<'kvs> { + fn serialize(&self, serializer: S) -> Result + where + S: Serializer, + { + serializer.serialize_str(self.inner) + } +} + +impl<'kvs> Display for Key<'kvs> { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + self.inner.fmt(f) + } +} + +impl<'kvs> Debug for Key<'kvs> { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + self.inner.fmt(f) + } +} +``` + +Other standard implementations could be added for any `K: AsRef` in the same fashion. + +#### Ownership + +The `Key` type borrows its inner value. + +#### Thread-safety + +The `Key` type is probably `Send` + `Sync`, but that's not guaranteed. + +### `Visitor` + +The `Visitor` trait used by `KeyValueSource` can visit a single key-value pair: + +```rust +pub trait Visitor<'kvs> { + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; +} + +impl<'a, 'kvs, T: ?Sized> Visitor<'kvs> for &'a mut T +where + T: Visitor<'kvs>, +{ + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { + (*self).visit_pair(k, v) + } +} +``` + +A `Visitor` may serialize the keys and values as it sees them. It may also do other work, like sorting or de-duplicating them. Operations that involve ordering keys will probably require allocations. + +#### Implementors + +There aren't any public implementors of `Visitor` in the `log` crate, but the `KeyValueSource::try_for_each` and `KeyValueSource::serialize_as_map` methods use the trait internally. + +Other crates that use key-value pairs will implement `Visitor`. + +#### Object safety + +The `Visitor` trait is object-safe. + +### `Error` + +Just about the only thing you can do with a `Value` in the `Visitor::visit_pair` method is serialize it with `serde`. Serialization might fail, so to allow errors to get carried back to callers the `visit_pair` method needs to return a `Result`. + +```rust +pub struct Error(Inner); + +impl Error { + pub fn msg(msg: &'static str) -> Self { + Error(Inner::Static(msg)) + } + + #[cfg(feature = "std")] + pub fn as_error(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { + &self.0 + } + + #[cfg(feature = "std")] + pub fn into_error(self) -> Box { + Box::new(self.0) + } + + pub fn into_serde(self) -> E + where + E: serde::ser::Error, + { + E::custom(self) + } +} + +#[cfg(feature = "std")] +impl From for Error +where + E: std::error::Error, +{ + fn from(err: E) -> Self { + Error(Inner::Owned(err.to_string())) + } +} + +#[cfg(feature = "std")] +impl From for Box { + fn from(err: Error) -> Self { + err.into_error() + } +} + +impl AsRef for Error { + fn as_ref(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { + self.as_error() + } +} + +enum Inner { + Static(&'static str), + #[cfg(feature = "std")] + Owned(String), +} + +#[cfg(feature = "std")] +impl std::error::Error for Inner { + fn description(&self) -> &str { + match self { + Inner::Static(msg) => msg, + Inner::Owned(msg) => msg, + } + } +} +``` + +There's no really universal way to handle errors in a logging pipeline. Knowing that some error occurred, and knowing where, should be enough for implementations of `Log` to decide how to handle it. The `Error` type doesn't try to be a general-purpose error management tool, it tries to make it easy to early-return with other errors encountered during `Visitor::visit_pair`. + +To make it possible to carry any arbitrary `S::Error` type, where we don't know how long the value can live for and whether it's `Send` or `Sync`, without extra work, the `Error` type does not attempt to store the error value itself. It just converts it into a `String`. + +### `KeyValueSource` + +The `KeyValueSource` trait is a bit like `Serialize`. It gives us a way to inspect some arbitrary collection of key-value pairs using a visitor pattern: + +```rust +pub trait KeyValueSource { + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error>; + + ... +} + +impl<'a, T: ?Sized> KeyValueSource for &'a T +where + T: KeyValueSource, +{ + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + (*self).visit(visitor) + } +} +``` + +`KeyValueSource` doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. + +#### Adapters + +Some useful adapters exist as provided methods on the `KeyValueSource` trait. They're similar to adapters on the standard `Iterator` trait: + +```rust +pub trait KeyValueSource { + ... + + /// Erase this `KeyValueSource` so it can be used without + /// requiring generic type parameters. + fn erase(&self) -> ErasedKeyValueSource + where + Self: Sized, + { + ErasedKeyValueSource::erased(self) + } + + /// An adapter to borrow self. + fn by_ref(&self) -> &Self { + self + } + + /// Chain two `KeyValueSource`s together. + fn chain(self, other: KVS) -> Chained + where + Self: Sized, + { + Chained(self, other) + } + + /// Find the value for a given key. + /// + /// If the key is present multiple times, this method will + /// return the *last* value for the given key. + /// + /// The default implementation will scan all key-value pairs. + /// Implementors are encouraged provide a more efficient version + /// if they can. Standard collections like `BTreeMap` and `HashMap` + /// will do an indexed lookup instead of a scan. + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: ToKey, + { + struct Get<'k, 'v>(Key<'k>, Option>); + + impl<'k, 'kvs> Visitor<'kvs> for Get<'k, 'kvs> { + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { + if k == self.0 { + self.1 = Some(v); + } + + Ok(()) + } + } + + let mut visitor = Get(key.to_key(), None); + let _ = self.visit(&mut visitor); + + visitor.1 + } + + /// Apply a function to each key-value pair. + fn try_for_each(self, f: F) -> Result<(), Error> + where + Self: Sized, + F: FnMut(Key, Value) -> Result<(), E>, + E: Into, + { + struct ForEach(F, std::marker::PhantomData); + + impl<'kvs, F, E> Visitor<'kvs> for ForEach + where + F: FnMut(Key, Value) -> Result<(), E>, + E: Into, + { + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { + (self.0)(k, v).map_err(Into::into) + } + } + + self.visit(&mut ForEach(f, Default::default())) + } + + /// Serialize the key-value pairs as a map. + fn serialize_as_map(self) -> SerializeAsMap + where + Self: Sized, + { + SerializeAsMap(self) + } +} +``` + +- `by_ref` to get a reference to a `KeyValueSource` within a method chain. +- `chain` to concatenate one source with another. This is useful for composing implementations of `Log` together for contextual logging. +- `get` to try find the value associated with a key. +- `try_for_each` to try execute some closure over all key-value pairs. This is a convenient way to do something with each key-value pair without having to create and implement a `Visitor`. +- `serialize_as_map` to get a serializable map. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. + +None of these methods are required for the core API. They're helpful tools for working with key-value pairs with minimal machinery. Even if we don't necessarily include them right away it's worth having an API that can support them later without breakage. + +#### Object safety + +`KeyValueSource` is not object-safe because of the provided adapter methods not being object-safe. The only required method, `visit`, is safe though, so an object-safe version of `KeyValueSource` that forwards this method can be reasonably written. + +```rust +/// An erased `KeyValueSource`. +#[derive(Clone)] +pub struct ErasedKeyValueSource<'a>(&'a dyn ErasedKeyValueSourceBridge); + +impl<'a> ErasedKeyValueSource<'a> { + /// Capture a `KeyValueSource` and erase its concrete type. + pub fn new(kvs: &'a impl KeyValueSource) -> Self { + ErasedKeyValueSource(kvs) + } +} + +impl<'a> Default for ErasedKeyValueSource<'a> { + fn default() -> Self { + ErasedKeyValueSource(&(&[] as &[(&str, &dyn ToValue)])) + } +} + +impl<'a> KeyValueSource for ErasedKeyValueSource<'a> { + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + self.0.erased_visit(visitor) + } + + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: ToKey, + { + let key = key.to_key(); + self.0.erased_get(key.as_ref()) + } +} + +/// A trait that erases a `KeyValueSource` so it can be stored +/// in a `Record` without requiring any generic parameters. +trait ErasedKeyValueSourceBridge { + fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; + fn erased_get<'kvs>(&'kvs self, key: &str) -> Option>; +} + +impl ErasedKeyValueSourceBridge for KVS +where + KVS: KeyValueSource + ?Sized, +{ + fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + self.visit(visitor) + } + + fn erased_get<'kvs>(&'kvs self, key: &str) -> Option> { + self.get(key) + } +} +``` + +#### Implementors + +A `KeyValueSource` with a single pair is implemented for a tuple of a key and value: + +```rust +impl KeyValueSource for (K, V) +where + K: ToKey, + V: ToValue, +{ + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + visitor.visit_pair(self.0.to_key(), self.1.to_value()) + } +} +``` + +A `KeyValueSource` with multiple pairs is implemented for arrays of `KeyValueSource`s: + +```rust +impl KeyValueSource for [KVS] where KVS: KeyValueSource { + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + for kv in self { + kv.visit(&mut visitor)?; + } + + Ok(()) + } +} +``` + +When `std` is available, `KeyValueSource` is implemented for some standard collections too: + +```rust +#[cfg(feature = "std")] +impl KeyValueSource for Vec where KVS: KeyValueSource { + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + self.as_slice().visit(visitor) + } +} + +#[cfg(feature = "std")] +impl KeyValueSource for collections::BTreeMap +where + K: Borrow + Ord, + V: ToValue, +{ + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + for (k, v) in self { + visitor.visit_pair(k.to_key(), v.to_value())?; + } + + Ok(()) + } + + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: ToKey, + { + let key = key.to_key(); + collections::BTreeMap::get(self, key.as_ref()).map(ToValue::to_value) + } +} + +#[cfg(feature = "std")] +impl KeyValueSource for collections::HashMap +where + K: Borrow + Eq + Hash, + V: ToValue, +{ + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + for (k, v) in self { + visitor.visit_pair(k.to_key(), v.to_value())?; + } + + Ok(()) + } + + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: ToKey, + { + let key = key.to_key(); + collections::HashMap::get(self, key.as_ref()).map(ToValue::to_value) + } +} +``` + +The `BTreeMap` and `HashMap` implementations provide more efficient implementations of `KeyValueSource::get`. + +## The `log!` macros + +The `log!` macro will initially support a fairly spartan syntax for capturing structured data. The current `log!` macro looks like this: + +```rust +log!(); +``` + +This RFC proposes an additional semi-colon-separated part of the macro for capturing key-value pairs: + +```rust +log!( ; ) +``` + +The `;` and structured values are optional. If they're not present then the behaviour of the `log!` macro is the same as it is today. + +As an example, this is what a `log!` statement containing structured key-value pairs could look like: + +```rust +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + correlation = correlation_id, + user = user +); +``` + +There's a *big* design space around the syntax for capturing log records we could explore, especially when you consider procedural macros. The syntax proposed here for the `log!` macro is not designed to be really ergonomic. It's designed to be *ok*, and to encourage an exploration of the design space by offering a consistent base that other macros could build off. + +Having said that, there are a few unintrusive quality-of-life features that make the `log!` macros nicer to use with structured data. + +### Expansion + +Styructured key-value pairs in the `log!` macro expand to statements that borrow from their environment. + +```rust +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + correlation = correlation_id, + user = user +); +``` + +Will expand to something like: + +```rust +{ + let lvl = log::Level::Info; + + if lvl <= ::STATIC_MAX_LEVEL && lvl <= ::max_level() { + let correlation &correlation_id; + let user = &user; + + let kvs: &[(&str, &dyn::key_values::ToValue)] = + &[("correlation", &correlation), ("user", &user)]; + + ::__private_api_log( + ::std::fmt::Arguments::new_v1( + &["This is the rendered ", ". It is not structured"], + &match (&"message",) { + (arg0,) => [::std::fmt::ArgumentV1::new(arg0, ::std::fmt::Display::fmt)], + }, + ), + lvl, + &("bin", "mod", "mod.rs", 13u32), + &kvs, + ); + } +}; +``` + +### Logging values that aren't `ToValue` + +Not every type a user might want to log will satisfy the default `Display + Serialize` bound. To reduce friction in these cases, there are a few helper macros that can be used to tweak the way structured data is captured. + +The pattern here is pretty general, you could imagine other macros being created for capturing other useful trait implementors, like `T: Fail`. + +#### `log_serde!` + +The `log_serde!` macro allows any type that implements just `Serialize` to be logged: + +```rust +info!( + "A log statement"; + user = log_serde!(user) +); +``` + +The macro definition looks like this: + +```rust +macro_rules! log_serde { + ($v:expr) => { + $crate::adapter::log_serde(&$v) + } +} + +#[cfg(feature = "erased-serde")] +pub fn log_serde(v: impl Serialize) -> impl ToValue { + struct SerdeAdapter(T); + + impl ToValue for SerdeAdapter + where + T: Serialize, + { + fn to_value(&self) -> Value { + Value::from_serde(&self.0) + } + } + + SerdeAdapter(v) +} +``` + +#### `log_fmt!` + +The `log_fmt!` macro allows any type to be logged as a formatted string: + +```rust +info!( + "A log statement"; + user = log_fmt!(user, Debug::fmt) +); +``` + +The macro definition looks like this: + +```rust +macro_rules! log_fmt { + ($v:expr, $f:expr) => { + $crate::adapter::log_fmt(&$v, $f) + } +} + +pub fn log_fmt(value: T, adapter: impl Fn(&T, &mut fmt::Formatter) -> fmt::Result) -> impl ToValue { + struct FmtAdapter { + value: T, + adapter: F, + } + + impl Display for FmtAdapter + where + F: Fn(&T, &mut Formatter) -> fmt::Result, + { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + (self.adapter)(&self.value, f) + } + } + + impl ToValue for FmtAdapter + where + F: Fn(&T, &mut Formatter) -> fmt::Result, + { + fn to_value(&self) -> Value { + Value::from_display(self) + } + } + + FmtAdapter { value, adapter } +} +``` + +#### `log_path!` + +The `log_path!` macro allows `Path` and `PathBuf` to be logged: + +```rust +info!( + "A log statement"; + path = log_path!(path) +); +``` + +The macro definition looks like this: + +```rust +macro_rules! log_path { + ($v:expr) => { + $crate::adapter::log_path(&$v) + } +} + +#[cfg(feature = "std")] +pub fn log_path(v: impl AsRef) -> impl ToValue { + #[derive(Debug)] + struct PathAdapter(T); + + impl Display for PathAdapter + where + T: AsRef, + { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + let path = self.0.as_ref(); + + match path.to_str() { + Some(path) => Display::fmt(path, f), + None => Debug::fmt(path, f), + } + } + } + + impl serde::Serialize for PathAdapter + where + T: AsRef, + { + fn serialize(&self, serializer: S) -> Result + where + S: serde::Serializer + { + serializer.collect_str(&self) + } + } + + PathAdapter(v) +} +``` + +# Drawbacks +[drawbacks]: #drawbacks + +Structured logging is a non-trivial feature to support. + +## `Display + Serialize` + +Using `Display + Serialize` as a blanket implementation for `ToValue` means we immediately need to work around some values that will probably be logged, but don't satisfy the bound: + +- `Path` and `PathBuf` +- Most `std::error::Error`s +- Most `failure::Fail`s + +Using `Debug + Serialize` would allow `Path`s to be logged with no extra effort, but `Display` is probably a more appropriate trait to use. + +In `no_std` contexts, more machinery is required to try retain the structure of primitive values, rather than falling back to string serialization through the `Display` bound. + +## `serde` + +Using `serde` as the serialization framework for structured logging introduces a lot of complexity that consumers of key-value pairs need to deal with. Inspecting values requires an implementation of a `Serializer`, which is a complex trait. + +Having `serde` enabled by default means it'll be effectively impossible to compile `log` in any reasonably sized dependency graph without also compiling `serde`. + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +## Just use `Display` + +This API could be a lot simpler if key-value pairs were only required to implement the `Display` trait. Unfortunately, this doesn't really provide structured logging, because `Display` doesn't retain any structure. Take the json example from before: + +```json +{ + "ts": 1538040723000, + "lvl": "INFO", + "msg": "Operation completed successfully in 18ms", + "module": "basic", + "service": "database", + "correlation": 123, + "took": 18 +} +``` + +If key-value pairs were implemented as `Display` then this json object would look like this: + +```json +{ + "ts": "1538040723000", + "lvl": "INFO", + "msg": "Operation completed successfully in 18ms", + "module": "basic", + "service": "database", + "correlation": "123", + "took": "18" +} +``` + +Now without knowing that `ts` is a number we can't reasonably use that field for querying over ranges of events. + +## Don't use a blanket implementation + +Without a blanket implementation of `ToValue`, a set of primitive and standard types could manually implement the trait in `log`. That could include `Path` and `PathBuf`, which don't currently satify the blanket `Display + Serialize` bounds. This would require libraries providing types in the ecosystem to depend on `log` and implement `ToValue` themselves. + +## Define our own serialization trait + +Rather than rely on `serde`, define our own simplified, object-safe serialization trait. This would avoid the complexity of erasing `Serialize` implementations, but would introduce the same ecosystem leakage as not using a blanket implementation. It would also be a trait that looks a lot like `Serialize`, without necessarily keeping up with improvements that are made in `serde`. + +## Don't enable structured logging by default + +Make structured logging a purely optional feature, so it wouldn't necessarily need to support `no_std` environments at all and could avoid the `Display + Serialize` blanket implementation bounds on `ToValue`. This seems appealing, but isn't ideal because it makes a fundamental modern logging feature much less discoverable. + +# Prior art +[prior-art]: #prior-art + +Structured logging is a paradigm that's supported by logging frameworks in many language ecosystems. + +## Rust + +The `slog` library is a structured logging framework for Rust. Its API predates a stable `serde` crate so it defines its own traits that are similar to `serde::Serialize`. A log record consists of a rendered message and bag of structured key-value pairs. `slog` goes further than this RFC proposes by requiring callers of its `log!` macros to state whether key-values are owned or borrowed by the record, and whether the data is safe to share across threads. + +This RFC proposes an API that's inspired by `slog`, but doesn't directly support distinguishing between owned or borrowed key-value pairs. Everything is borrowed. That means the only way to send a `Record` to another thread is to serialize it into a different type. + +## Go + +The `logrus` library is a structured logging framework for Go. It uses a similar separation of the textual log message from structured key-value pairs that this API proposes. + +## .NET + +The C# community has mostly standardised around using message templates for packaging a log message with structured key-value pairs. Instead of logging a rendered message and separate bag of structured data, the log record contains a template that allows key-value pairs to be interpolated from the same bag of structured data. It avoids duplicating the same information multiple times. + +Supporting something like message templates in Rust using the `log!` macros would probably require procedural macros. A macro like that could be built on top of the API proposed by this RFC. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the RFC process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +----- + From b8182230330d318e604218c490e510e8bbb22882 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Mon, 29 Oct 2018 09:55:13 +1000 Subject: [PATCH 02/20] update RFC text based on feedback so far --- rfcs/0000-structured-logging.md | 2025 +++++++++++++++++++------------ 1 file changed, 1233 insertions(+), 792 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 0d7db6d24..05ad9338f 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -3,7 +3,7 @@ Add support for structured logging to the `log` crate in both `std` and `no_std` environments, allowing log records to carry typed data beyond a textual message. This document serves as an introduction to what structured logging is all about, and as an RFC for an implementation in the `log` crate. -`log` will depend on `serde` for structured logging support, which will be enabled by default. See [the implications for `log` users](#implications-for-dependents) for some more details. The API is heavily inspired by the `slog` logging framework. +`log` will provide a lightweight fundamental serialization API out-of-the-box that can be expanded to full `serde` support using crate features. It doesn't require crates implement any traits from `log` before their datastructures can be used in the `log!` macros. Instead, crates should implement `std::fmt::Debug` and `serde::Serialize`, which are traits they should by considering already. The API is heavily inspired by the `slog` logging framework. See [the implications for `log` users](#implications-for-dependents) for some more details. > NOTE: Code in this RFC uses recent language features like `impl Trait`, but can be implemented without them. @@ -13,30 +13,25 @@ Add support for structured logging to the `log` crate in both `std` and `no_std` - [What is structured logging?](#what-is-structured-logging) - [Why do we need structured logging in `log`?](#why-do-we-need-structured-logging-in-log) - [Guide-level explanation](#guide-level-explanation) - - [Capturing structured logs](#capturing-structured-logs) - - [Consuming structured logs](#consuming-structured-logs) + - [Logging structured key-value pairs](#logging-structured-key-value-pairs) + - [Supporting key-value pairs in `Log` implementations](#supporting-key-value-pairs-in-log-implementations) + - [Integrating log frameworks with `log`](#integrating-log-frameworks-with-log) + - [Writing your own `value::Visitor`](#writing-your-own-valuevisitor) - [Reference-level explanation](#reference-level-explanation) - [Design considerations](#design-considerations) - [Implications for dependents](#implications-for-dependents) - [Cargo features](#cargo-features) - [Public API](#public-api) - [`Record` and `RecordBuilder`](#record-and-recordbuilder) - - [`ToValue`](#tovalue) + - [`Visit`](#Visit) - [`Value`](#value) - - [`ToKey`](#tokey) - [`Key`](#key) - - [`Visitor`](#visitor) + - [`value::Visitor`](#valuevisitor) - [`Error`](#error) - - [`KeyValueSource`](#keyvaluesource) + - [`Source`](#source) + - [`source::Visitor`](#sourcevisitor) - [The `log!` macros](#the-log-macros) -- [Drawbacks](#drawbacks) - - [`Display + Serialize`](#display--serialize) - - [`serde`](#serde) -- [Rationale and alternatives](#rationale-and-alternatives) - - [Just use `Display`](#just-use-display) - - [Don't use a blanket implementation](#dont-use-a-blanket-implementation) - - [Define our own serialization trait](#define-our-own-serialization-trait) - - [Don't enable structured logging by default](#dont-enable-structured-logging-by-default) +- [Drawbacks, rationale, and alternatives](#drawbacks-rationale-and-alternatives) - [Prior art](#prior-art) - [Rust](#rust) - [Go](#go) @@ -101,13 +96,9 @@ A healthy logging ecosystem needs both `log` and frameworks like `slog`. As a st # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -## Capturing structured logs +## Logging structured key-value pairs -Structured logging is supported in `log` by allowing typed key-value pairs to be associated with a log record. - -### Structured vs unstructured - -A `;` separates structured key-value pairs from values that are replaced into the message: +Structured logging is supported in `log` by allowing typed key-value pairs to be associated with a log record. A `;` separates structured key-value pairs from values that are replaced into the message: ```rust info!( @@ -137,58 +128,62 @@ info!( ); ``` -Any type that implements both `Display` and `serde::Serialize` can be used as the value in a structured key-value pair. In the previous example, the values used in the macro require the following trait bounds: - -``` -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - ^^^^^^^^^ - Display +### What can be logged? - correlation = correlation_id, - ^^^^^^^^^^^^^^ - Display + Serialize +Anything that's `std::fmt::Debug + serde::Serialize` can be logged as a structured value. Using default crate features though, only a fixed set of types from the standard library are supported: - user - ^^^^ - Display + Serialize -); -``` +- Standard formats: `Arguments` +- Primitives: `bool`, `char` +- Unsigned integers: `u8`, `u16`, `u32`, `u64`, `u128` +- Signed integers: `i8`, `i16`, `i32`, `i64`, `i128` +- Strings: `&str`, `String` +- Bytes: `&[u8]`, `Vec` +- Paths: `Path`, `PathBuf` +- Special types: `Option` and `()` -### Logging data that isn't `Display + Serialize` +This keeps the `log` crate lightweight by default. In the example from before, `correlation_id` and `user` can be used as structured values if they're in that set of types: -The `Display + Serialize` bounds we've been using thus far are mostly accurate, but don't tell the whole story. Structured values don't _technically_ require `Display + Serialize`. They require a trait for capturing structured values, `ToValue`, which we implement for any type that also implements `Display + Serialize`. So in truth the trait bounds required by the macro look like this: +```rust +let user = "a user id"; +let correlation_id = "some correlation id"; -``` info!( "This is the rendered {message}. It is not structured", message = "message"; - ^^^^^^^^^ - Display - correlation = correlation_id, - ^^^^^^^^^^^^^^ - ToValue - user - ^^^^ - ToValue ); ``` -The `ToValue` trait makes it possible for types to be logged even if they don't implement `Display + Serialize`; they only need to implement `ToValue`. One of the goals of this RFC is to avoid creating another fundamental trait that needs to be implemented throughout the ecosystem though. So instead of implementing `ToValue` directly, we provide a few helper macros to wrap a value satisfying one set of trait bounds into a value that satisfies `ToValue`. The following example uses these helper macros to change the trait bounds required by `correlation_id` and `user`: +What if the `correlation_id` is a `uuid::Uuid` instead of a string? What if the `user` is some other datastructure containing an id along with some other metadata? We still want to be able to log these values. The set of loggable values above can be expanded to also include any other type that implements both `std::fmt::Debug` and `serde::Serialize` using a crate feature: + +```toml +[dependencies.log] +features = ["kv_serde"] + +[dependencies.uuid] +features = ["serde"] +``` ```rust +#[derive(Debug, Serialize)] +struct User { + id: Uuid, + .. +} + +let user = User { id, .. }; +let correlation_id = Uuid::new_v4(); + info!( "This is the rendered {message}. It is not structured", message = "message"; - correlation = log_fmt!(correlation_id, Debug::fmt), - user = log_serde!(user) + correlation = correlation_id, + user ); ``` -They now look like this: +So the effective trait bounds for structured values are `Debug + Serialize`: ``` info!( @@ -197,29 +192,21 @@ info!( ^^^^^^^^^ Display - correlation = log_fmt!(correlation_id, Debug::fmt), - ^^^^^^^^^^^^^^ - Debug + correlation = correlation_id, + ^^^^^^^^^^^^^^ + Debug + Serialize - user = log_serde!(user) - ^^^^ - Serialize + user + ^^^^ + Debug + Serialize ); ``` -This same pattern can be applied to other types in the standard library and wider ecosystem that are likely to be logged, but don't typically satisfy `Display + Serialize`. Some examples are `Path`, and implementations of `Error` or `Fail`. - -So why do we use `Display + Serialize` as the effective bounds in the first place? Practically, it's required to keep a consistent API in `no_std` environments, where an object-safe wrapper over `serde` requires standard library features. `Display` is a natural trait to lean on when `Serialize` isn't available, so requiring both means there's no change in the trait bounds when records can be captured using `serde` and when they can't (later on we'll see how we can still use `serde` for primitive types in `no_std` environments). - -`Display` also suggests the data we are logging should have some canonical representation that's useful for someone to look at. Finding small but sufficient representations of potentially large pieces of state to log will make those logs easier to ingest and analyze later. - -## Consuming structured logs +If you come across a data type in the Rust ecosystem that you can't log, then try looking for a `serde` feature on the crate that data type comes from. If there isn't one already then adding it will be useful not just for you, but for anyone that might want to serialize those types for other reasons. -Capturing structured logs is only half the story. Implementors of the `Log` trait also need to be able to work with any key-value pairs associated with a log record. +## Supporting key-value pairs in `Log` implementations -### Using `Visitor` to print or serialize key-value pairs - -Structured key-value pairs can be inspected using the `Visitor` trait. Take the terminal log format from before: +Capturing structured logs is only half the story. Implementors of the `Log` trait also need to be able to work with any key-value pairs associated with a log record. Take the terminal log format from before: ``` [INF 2018-09-27T09:32:03Z] Operation completed successfully in 18ms @@ -229,43 +216,24 @@ correlation: 123 took: 18 ``` -Each key-value pair, shown as `$key: $value`, can be written using a `Visitor` that hands values to a `serde::Serializer`. The implementation of that `Visitor` could look like this: +Each key-value pair, shown as `$key: $value`, can be formatted using the `std::fmt` machinery: ```rust +use log::kv::Source; + fn write_pretty(w: impl Write, r: &Record) -> io::Result<()> { // Write the first line of the log record ... - // Write each key-value pair using the `WriteKeyValues` visitor + // Write each key-value pair on a new line record .key_values() - .visit(&mut WriteKeyValues(w)) + .try_for_each(|k, v| writeln!("{}: {}", k, v)) .map_err(|e| io::Error::new(io::ErrorKind::Other, e.into_error())) } - -struct WriteKeyValues(W); - -impl<'kvs, W> Visitor<'kvs> for WriteKeyValues -where - W: Write, -{ - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { - // Write the key - // `Key` is a wrapper around a string - // It can be formatted directly - write!(&mut self.0, "{}: ", k)?; - - // Write the value - // `Value` is a wrapper around `serde::Serialize` - // It can't be formatted directly - v.serialize(&mut Serializer::new(&mut self.0))?; - - Ok(()) - } -} ``` -Needing `serde` for the human-centric format above seems a bit unnecessary. The value becomes clearer when using a machine-centric format for the entire log record, like json. Take the following json format: +Now take the following json format: ```json { @@ -279,9 +247,18 @@ Needing `serde` for the human-centric format above seems a bit unnecessary. The } ``` -Defining a serializable structure for this format could be done using `serde_derive`, and then written using `serde_json`: +Defining a serializable structure based on a log record for this format could be done using `serde_derive`, and then written using `serde_json`. This requires the `kv_serde` feature: + +```toml +[dependencies.log] +features = ["kv_serde"] +``` + +The structured key-value pairs can then be naturally serialized as a map: ```rust +use log::kv::Source; + fn write_json(w: impl Write, r: &Record) -> io::Result<()> { let r = SerializeRecord { lvl: r.level(), @@ -305,76 +282,140 @@ struct SerializeRecord { } ``` -There's no explicit `Visitor` in this case, because the `serialize_as_map` method wraps one up internally so all key-value pairs are serializable as a map using `serde`. +The crate that produces log records might not be the same crate that consumes them. A producer can depend on the `kv_serde` feature to log more types, but still have those types handled by a consumer that doesn't use the `kv_serde` feature. -### Using `KeyValueSource` to capture key-value pairs +## Integrating log frameworks with `log` -What exactly are the key-value pairs on a record? The previous examples used a `key_values()` method on `Record` to get _something_ that could be visited or serialized using a `Visitor`. That something is an implementation of a trait, `KeyValueSource`, which holds the actual `Key` and `Value` pairs: +The `Source` trait describes some container for structured key-value pairs. Other log frameworks that want to integrate with `log::Log` should build `Record`s that contain some implementation of `Source`. -``` -fn write_pretty(w: impl Write, r: &Record) -> io::Result<()> { - ... +```rust +use log::kv::source::{self, Source}; - record - .key_values() - ^^^^^^^^^^^^^ - `Record::key_values` returns `impl KeyValueSource` +struct MySource { + data: BTreeMap, +} - .visit(&mut WriteKeyValues(w)) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - `KeyValueSource::visit` takes `&mut Visitor` +impl Source for MySource { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { + self.data.visit(visitor) + } } ``` -As an example of a `KeyValueSource`, the `log!` macros can capture key-value pairs into an array of `(&str, &dyn ToValue)` tuples that can be visited in sequence: +`BTreeMap` happens to already implement the `Source` trait. Let's assume `BTreeMap` didn't implement `Source`. A manual implementation converting the `(String, serde_json::Value)` pairs into types that can be visited could look like this: ```rust -impl<'a> KeyValueSource for [(&'a str, &'a dyn ToValue)] { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) { - for &(k, v) in self { - visitor.visit_pair(k.to_key(), v.to_value())?; +use log::kv::{ + source::{self, Source}, + value, +}; + +struct MySource { + data: BTreeMap, +} + +impl Source for MySource { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { + for (k, v) in self.data { + visitor.visit_pair(source::Key::new(k), source::Value::new(v)) } + } +} +``` - Ok(()) +The `Key::new` method accepts any `T: Borrow`. The `Value::new` accepts any `T: std::fmt::Debug + serde::Serialize`. Values that can't implement `Debug + Serialize` can still be visited using the `source::Value::any` method. This method lets us provide an inline function that will visit the value: + +```rust +use log::kv::{ + source::{self, Source}, + value, +}; + +struct MySource { + data: BTreeMap, +} + +impl Source for MySource { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { + for (k, v) in self.data { + let key = source::Key::new(k); + + // Let's assume `MyValue` doesn't implement `Serialize` + // Instead it implements `Display`. + let value = source::Value::any(v, |v, visitor| { + // Let's assume `MyValue` implements `Display` + visitor.visit_fmt(format_args!("{}", v)) + }); + + visitor.visit_pair(key, value) } } ``` -A `KeyValueSource` doesn't have to just contain key-value pairs directly like this though. It could also act like an adapter, like we have for iterators in the standard library. As another example, the following `KeyValueSource` doesn't store any key-value pairs of its own, it will sort and de-duplicate pairs read from another source by first reading them into a map before forwarding them on: +A `Source` doesn't have to just contain key-value pairs directly like `BTreeMap` though. It could also act like an adapter, like we have for iterators in the standard library. As another example, the following `Source` doesn't store any key-value pairs of its own, it will sort and de-duplicate pairs read from another source by first reading them into a map before forwarding them on: ```rust +use log::kv::source::{self, Source, Visitor}; + pub struct SortRetainLast(KVS); -impl KeyValueSource for SortRetainLast +impl Source for SortRetainLast where - KVS: KeyValueSource, + KVS: Source, { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { // `Seen` is a visitor that will capture key-value pairs // in a `BTreeMap`. We use it internally to sort and de-duplicate // the key-value pairs that `SortRetainLast` is wrapping. - struct Seen<'kvs>(BTreeMap, Value<'kvs>>); + struct Seen<'kvs>(BTreeMap, source::Value<'kvs>>); impl<'kvs> Visitor<'kvs> for Seen<'kvs> { - fn visit_pair<'vis>(&'vis mut self, k: Key<'kvs>, v: Value<'kvs>) { + fn visit_pair<'vis>(&'vis mut self, k: source::Key<'kvs>, v: source::Value<'kvs>) -> Result<(), source::Error> { self.0.insert(k, v); + + Ok(()) } } // Visit the inner source and collect its key-value pairs into `seen` let mut seen = Seen(BTreeMap::new()); - self.0.visit(&mut seen); + self.0.visit(&mut seen)?; // Iterate through the seen key-value pairs in order // and pass them to the `visitor`. for (k, v) in seen.0 { - visitor.visit_pair(k, v); + visitor.visit_pair(k, v)?; } + + Ok(()) } } ``` -This API is similar to `serde` and `slog`, and is very flexible. Other structured logging concepts like contextual logging, where a record is enriched with information from its envirnment, can be built on top of `KeyValueSource` and `Visitor`. +This API is similar to `serde` and `slog`, and is very flexible. Other structured logging concepts like contextual logging, where a record is enriched with information from its envirnment, can be built on top of `Source` and `Visitor`. + +## Writing your own `value::Visitor` + +Consumers of key-value pairs can visit structured values without needing a `serde::Serializer`. Instead they can implement a `value::Visitor`. A `Visitor` can always visit any structured value by formatting it using its `Debug` implementation: + +```rust +use log::kv::value::{self, Value}; + +struct WriteVisitor(W); + +impl value::Visitor for WriteVisitor +where + W: Write, +{ + fn visit_any(&mut self, v: Value) -> Result<(), value::Error> { + write!(&mut self.0, "{:?}", v)?; + + Ok(()) + } +} +``` + +There are other methods besides `visit_any` that can be implemented. By default they all forward to `visit_any`. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation @@ -383,11 +424,11 @@ This API is similar to `serde` and `slog`, and is very flexible. Other structure ### Don't break anything -Allow structured logging to be added in the current `0.4.x` series of `log`. +Allow structured logging to be added in the current `0.4.x` series of `log`. This gives us the option of including structured logging in an `0.4.x` release, or bumping the minor crate version without introducing any actual breaking changes. -### Leverage the ecosystem +### Don't create a public dependency -Rather than trying to define our own serialization trait and require libraries in the ecosystem implement it, we leverage `serde`. Any new types that emerge in the ecosystem don't have another fundamental trait they need to think about. +Don't create a new serialization framework that causes `log` to become a public dependency of any library that wants their data types to be loggable. Logging is a truly cross-cutting concern, so if `log` was a public dependency it would probably be at least as pervasive as `serde` is now. ### Object safety @@ -397,41 +438,42 @@ Rather than trying to define our own serialization trait and require libraries i `Record` borrows all data from the call-site so records need to be handled directly on-thread as they're produced. On the one hand that means that log records need to be serialized before they can be sent across threads. On the other hand it means callers don't need to make assumptions about whether records need to be owned or borrowed. +A scheme for more efficient conversion from borrowed to owned sources of key-value pairs can be added later. + +## Cargo features + +Structured logging will be supported in either `std` or `no_std` contexts using Cargo features: + +```toml +[features] +# optional serde-support requires `std` +kv_serde = ["std", "serde", "erased-serde"] +``` + +Using default features, structured logging will be supported by `log` in `no_std` environments for a fixed set of types from the standard library. Using the `kv_serde` feature, any type that implements `Debug + Serialize` can be logged, and its potentially complex structure will be retained. + ## Implications for dependents Dependents of `log` will notice the following: +### Default crate features + +The API that's available with default features doesn't add any extra dependencies to the `log` crate, and shouldn't impact compile times or artifact size much. + +### After opting in to `kv_serde` + In `no_std` environments (which is the default for `log`): - `serde` will enter the `Cargo.lock` if it wasn't there already. This will impact compile-times. - Artifact size of `log` will increase. -In `std` environments (which is common when using `env_logger` and other crates that implement `log`): +In `std` environments (which is common when using `env_logger` and other crates that implement `Log`): - `serde` and `erased-serde` will enter the `Cargo.lock` if it wasn't there already. This will impact compile-times. - Artifact size of `log` will increase. In either case, `serde` will become a public dependency of the `log` crate, so any breaking changes to `serde` will result in breaking changes to `log`. -## Cargo features - -Structured logging will be supported in either `std` or `no_std` contexts using Cargo features: - -```toml -[features] -# support structured logging by default -default = ["structured"] - -# semantic name for the structured logging feature -structured = ["serde"] - -# when `std` is available, always support structured logging -# with `erased-serde` -std = ["structured", "erased-serde"] -``` - -Using default features, structured logging will be supported by `log` in `no_std` environments. Structured logging will always be available when using the `std` feature (usually pulled in by libraries that implement the `Log` trait). - ## Public API For context, ignoring the `log!` macros, this is roughly the additional public API this RFC proposes to support structured logging: @@ -439,206 +481,295 @@ For context, ignoring the `log!` macros, this is roughly the additional public A ```rust impl<'a> RecordBuilder<'a> { /// Set the key-value pairs on a log record. - #[cfg(feature = "serde")] - pub fn key_values(&mut self, kvs: ErasedKeyValues<'a>) -> &mut RecordBuilder<'a>; + pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a>; } impl<'a> Record<'a> { /// Get the key-value pairs. - #[cfg(feature = "serde")] - pub fn key_values(&self) -> ErasedKeyValues; + pub fn key_values(&self) -> ErasedSource; /// Get a builder that's preconfigured from this record. pub fn to_builder(&self) -> RecordBuilder; } -#[cfg(features = "serde")] -pub mod key_values { - /// A visitor for a set of key-value pairs. - /// - /// The visitor is driven by an implementation of `KeyValueSource`. - /// The visitor expects keys and values that satisfy a given lifetime. - pub trait Visitor<'kvs> { - /// Visit a single key-value pair. - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; - } +pub mod kv { + pub mod source { + pub use kv::Error; + + /// A source for key-value pairs. + pub trait Source { + /// Serialize the key value pairs. + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; + + /// Erase this `Source` so it can be used without + /// requiring generic type parameters. + fn erase(&self) -> ErasedSource + where + Self: Sized {} + + /// Find the value for a given key. + /// + /// If the key is present multiple times, this method will + /// return the *last* value for the given key. + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: Borrow {} + + /// An adapter to borrow self. + fn by_ref(&self) -> &Self {} + + /// Chain two `Source`s together. + fn chain(self, other: KVS) -> Chain + where + Self: Sized {} + + /// Apply a function to each key-value pair. + fn try_for_each(self, f: F) -> Result<(), Error> + where + Self: Sized, + F: FnMut(Key, Value) -> Result<(), E>, + E: Into {} + + /// Serialize the key-value pairs as a map. + fn serialize_as_map(self) -> SerializeAsMap + where + Self: Sized {} + + /// Serialize the key-value pairs as a sequence of tuples. + fn serialize_as_seq(self) -> SerializeAsSeq + where + Self: Sized {} + } + + /// A visitor for a set of key-value pairs. + /// + /// The visitor is driven by an implementation of `Source`. + /// The visitor expects keys and values that satisfy a given lifetime. + pub trait Visitor<'kvs> { + /// Visit a single key-value pair. + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; + } + + /// An erased `Source`. + pub struct ErasedSource<'a> {} + + impl<'a> ErasedSource<'a> { + /// Capture a `Source` and erase its concrete type. + pub fn new(kvs: &'a impl Source) -> Self {} + } + + impl<'a> Clone for ErasedSource<'a> {} + impl<'a> Default for ErasedSource<'a> {} + impl<'a> Source for ErasedSource<'a> {} - /// A source for key-value pairs. - pub trait KeyValueSource { - /// Serialize the key value pairs. - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; + /// A `Source` adapter that visits key-value pairs + /// in sequence. + /// + /// This is the result of calling `chain` on a `Source`. + pub struct Chain {} - /// Erase this `KeyValueSource` so it can be used without - /// requiring generic type parameters. - fn erase(&self) -> ErasedKeyValueSource + impl Source for Chain where - Self: Sized {} + A: Source, + B: Source {} - /// Find the value for a given key. + /// A `Source` adapter that can be serialized as + /// a map using `serde`. /// - /// If the key is present multiple times, this method will - /// return the *last* value for the given key. - fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + /// This is the result of calling `serialize_as_map` on + /// a `Source`. + pub struct SerializeAsMap {} + + impl Serialize for SerializeAsMap where - Q: ToKey {} + KVS: Source {} - /// An adapter to borrow self. - fn by_ref(&self) -> &Self {} + /// A `Source` adapter that can be serialized as + /// a sequence of tuples using `serde`. + /// + /// This is the result of calling `serialize_as_seq` on + /// a `Source`. + pub struct SerializeAsSeq {} - /// Chain two `KeyValueSource`s together. - fn chain(self, other: KVS) -> Chain + impl Serialize for SerializeAsSeq where - Self: Sized {} + KVS: Source {} - /// Apply a function to each key-value pair. - fn try_for_each(self, f: F) -> Result<(), Error> + impl Source for (K, V) where - Self: Sized, - F: FnMut(Key, Value) -> Result<(), E>, - E: Into {} + K: Borrow, + V: kv::value::Visit {} - /// Serialize the key-value pairs as a map. - fn serialize_as_map(self) -> SerializeAsMap + impl Source for [KVS] where - Self: Sized {} - } + KVS: Source {} + + #[cfg(feature = "std")] + impl Source for Box where KVS: Source {} + #[cfg(feature = "std")] + impl Source for Arc where KVS: Source {} + #[cfg(feature = "std")] + impl Source for Rc where KVS: Source {} - /// An erased `KeyValueSource`. - pub struct ErasedKeyValueSource<'a> {} + #[cfg(feature = "std")] + impl Source for Vec + where + KVS: Source {} + + #[cfg(feature = "std")] + impl Source for BTreeMap + where + K: Borrow + Ord, + V: kv::value::Visit {} + + #[cfg(feature = "std")] + impl Source for HashMap + where + K: Borrow + Eq + Hash, + V: kv::value::Visit {} - impl<'a> ErasedKeyValueSource<'a> { - /// Capture a `KeyValueSource` and erase its concrete type. - pub fn new(kvs: &'a impl KeyValueSource) -> Self {} + /// The key in a key-value pair. + pub struct Key<'kvs> {} + + /// The value in a key-value pair. + pub use kv::value::Value; } - impl<'a> Clone for ErasedKeyValueSource<'a> {} - impl<'a> Default for ErasedKeyValueSource<'a> {} - impl<'a> KeyValueSource for ErasedKeyValueSource<'a> {} + pub mod value { + pub use kv::Error; - /// A `KeyValueSource` adapter that visits key-value pairs - /// in sequence. - /// - /// This is the result of calling `chain` on a `KeyValueSource`. - pub struct Chain {} + /// An arbitrary structured value. + pub struct Value<'v> { + /// Create a new borrowed value. + pub fn new(v: &'v impl Visit) -> Self {} - impl KeyValueSource for Chain - where - A: KeyValueSource, - B: KeyValueSource {} + /// Create a new borrowed value from an arbitrary type. + pub fn any(&'v T, fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self {} - /// A `KeyValueSource` adapter that can be serialized as - /// a map using `serde`. - /// - /// This is the result of calling `serialize_as_map` on - /// a `KeyValueSource`. - pub struct SerializeAsMap {} + /// Visit the value with the given serializer. + pub fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> {} + } - impl Serialize for SerializeAsMap - where - KVS: KeyValueSource {} + impl<'v> Debug for Value<'v> {} + impl<'v> Display for Value<'v> {} - impl KeyValueSource for (K, V) - where - K: ToKey, - V: ToValue {} + /// A serializer for primitive values. + pub trait Visitor { + /// Visit an arbitrary value. + fn visit_any(&mut self, v: Value) -> Result<(), Error>; - impl KeyValueSource for [KVS] - where - KVS: KeyValueSource {} + /// Visit a signed integer. + fn visit_i64(&mut self, v: i64) -> Result<(), Error> {} - #[cfg(feature = "std")] - impl KeyValueSource for Vec - where - KVS: KeyValueSource {} + /// Visit an unsigned integer. + fn visit_u64(&mut self, v: u64) -> Result<(), Error> {} - #[cfg(feature = "std")] - impl KeyValueSource for BTreeMap - where - K: Borrow + Ord, - V: ToValue {} + /// Visit a floating point number. + fn visit_f64(&mut self, v: f64) -> Result<(), Error> {} - #[cfg(feature = "std")] - impl KeyValueSource for HashMap - where - K: Borrow + Eq + Hash, - V: ToValue {} + /// Visit a boolean. + fn visit_bool(&mut self, v: bool) -> Result<(), Error> {} - /// A type that can be converted into a borrowed key. - pub trait ToKey { - /// Perform the conversion. - fn to_key(&self) -> Key; - } + /// Visit a single character. + fn visit_char(&mut self, v: char) -> Result<(), Error> {} - impl ToKey for str {} + /// Visit a UTF8 string. + fn visit_str(&mut self, v: &str) -> Result<(), Error> {} - #[cfg(feature = "std")] - impl ToKey for String {} + /// Visit a raw byte buffer. + fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> {} - #[cfg(feature = "std")] - impl<'a> ToKey for Cow<'a, str> {} + /// Visit an empty value. + fn visit_none(&mut self) -> Result<(), Error> {} - /// A key in a key-value pair. - /// - /// The key can be treated like `&str`. - pub struct Key<'kvs> {} + /// Visit standard arguments. + fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error> {} + } - impl<'kvs> Key<'kvs> { - /// Get a key from a borrowed string. - pub fn from_str(key: &'a (impl AsRef + ?Sized)) -> Self; + impl<'a, T: ?Sized> Visitor for &'a mut T + where + T: Visitor {} - /// Get a reference to the key as a string. - pub fn as_str(&self) -> &str; - } + /// Covnert a type into a value. + /// + /// ** This trait can't be implemented manually ** + pub trait Visit: private::Sealed { + /// Visit this value. + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; + + /// Convert a reference to this value into an erased `Value`. + fn to_value(&self) -> Value + where + Self: Sized, + { + Value::new(self) + } + } - impl<'kvs> Serialize for Key<'kvs> {} - impl<'kvs> PartialEq for Key<'kvs> {} - impl<'kvs> Eq for Key<'kvs> {} - impl<'kvs> PartialOrd for Key<'kvs> {} - impl<'kvs> Ord for Key<'kvs> {} - impl<'kvs> Hash for Key<'kvs> {} - impl<'kvs> AsRef for Key<'kvs> {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for u8 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for u16 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for u32 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for u64 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for u128 {} + + #[cfg(not(feature = "kv_serde"))] + impl Visit for i8 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for i16 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for i32 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for i64 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for i128 {} + + #[cfg(not(feature = "kv_serde"))] + impl Visit for f32 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for f64 {} + + #[cfg(not(feature = "kv_serde"))] + impl Visit for char {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for bool {} + + #[cfg(not(feature = "kv_serde"))] + impl Visit for Option + where + T: Visit {} - #[cfg(feature = "std")] - impl<'kvs> Borrow for Key<'kvs> {} + #[cfg(all(not(feature = "kv_serde"), feature = "std"))] + impl Visit for Box + where + T: Visit {} - /// A type that can be converted into a borrowed value. - pub trait ToValue { - /// Perform the conversion. - fn to_value(&self) -> Value; - } + #[cfg(not(feature = "kv_serde"))] + impl<'a> Visit for &'a str {} + #[cfg(all(not(feature = "kv_serde"), feature = "std"))] + impl Visit for String {} - impl ToValue for T - where - T: Display + Serialize {} + #[cfg(not(feature = "kv_serde"))] + impl<'a> Visit for &'a [u8] {} + #[cfg(all(not(feature = "kv_serde"), feature = "std"))] + impl Visit for Vec {} - /// A value in a key-value pair. - /// - /// The value can be treated like `serde::Serialize`. - pub struct Value<'kvs> {} + #[cfg(not(feature = "kv_serde"))] + impl<'a, T> Visit for &'a T + where + T: Visit {} - impl<'kvs> Value<'kvs> { - /// Get a value that will choose either the `Display` - /// or `Serialize` implementation based on the platform. - /// - /// If the standard library is available, the `Serialize` - /// implementation will be used. If the standard library - /// is not available, the `Display` implementation will - /// probably be used. - pub fn new(v: &'kvs (impl Display + Serialize)) -> Self; - - /// Get a value that can be serialized as a string using - /// its `Display` implementation. - pub fn from_display(v: &'kvs impl Display) -> Self; - - /// Get a value that can be serialized as structured - /// data using its `Serialize` implementation. - #[cfg(feature = "std")] - pub fn from_serde(v: &'kvs impl Serialize) -> Self; + #[cfg(feature = "kv_serde")] + impl Visit for T + where + T: Debug + Serialize {} } - impl<'kvs> Serialize for Value<'kvs> {} - impl<'kvs> ToValue for Value<'kvs> {} - impl<'a, 'kvs> ToValue for &'a Value<'kvs> {} + pub use source::Source; /// An error encountered while visiting key-value pairs. pub struct Error {} @@ -656,16 +787,24 @@ pub mod key_values { pub fn into_error(self) -> Box {} /// Convert into a `serde` error. + #[cfg(feature = "kv_serde")] pub fn into_serde(self) -> E where E: serde::ser::Error {} } + #[cfg(not(feature = "std"))] + impl From for Error {} + + #[cfg(feature = "std")] impl From for Error where E: std::error::Error {} + #[cfg(feature = "std")] impl From for Box {} + + #[cfg(feature = "std")] impl AsRef for Error {} } ``` @@ -677,8 +816,7 @@ Structured key-value pairs can be set on a `RecordBuilder`: ```rust impl<'a> RecordBuilder<'a> { /// Set key values - #[cfg(feature = "serde")] - pub fn key_values(&mut self, kvs: ErasedKeyValueSource<'a>) -> &mut RecordBuilder<'a> { + pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a> { self.record.kvs = kvs; self } @@ -692,285 +830,774 @@ These key-value pairs can then be accessed on the built `Record`: pub struct Record<'a> { ... - #[cfg(feature = "serde")] - kvs: ErasedKeyValueSource<'a>, + kvs: ErasedSource<'a>, } impl<'a> Record<'a> { /// The key value pairs attached to this record. /// /// Pairs aren't guaranteed to be unique (the same key may be repeated with different values). - #[cfg(feature = "serde")] - pub fn key_values(&self) -> ErasedKeyValueSource { + pub fn key_values(&self) -> ErasedSource { self.kvs.clone() } } ``` -### `ToValue` +### `Error` -A `ToValue` is a potentially long-lived structure that can be converted into a `Value`: +Just about the only things you can do with a structured value are format it or serialize it. Serialization and writing might fail, so to allow errors to get carried back to callers there needs to be a general error type that they can early return with: ```rust -/// A type that can be converted into a borrowed value. -pub trait ToValue { - /// Perform the conversion. - fn to_value(&self) -> Value; +pub struct Error(Inner); + +enum Inner { + Static(&'static str), + #[cfg(feature = "std")] + Owned(String), +} + +impl Error { + pub fn msg(msg: &'static str) -> Self { + Error(Inner::Static(msg)) + } + + #[cfg(feature = "std")] + pub fn as_error(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { + &self.0 + } + + #[cfg(feature = "std")] + pub fn into_error(self) -> Box { + Box::new(self.0) + } + + #[cfg(feature = "kv_serde")] + pub fn into_serde(self) -> E + where + E: serde::ser::Error, + { + E::custom(self) + } +} + +#[cfg(feature = "std")] +impl From for Error +where + E: std::error::Error, +{ + fn from(err: E) -> Self { + Error(Inner::Owned(err.to_string())) + } } -impl<'a> ToValue for &'a dyn ToValue { - fn to_value(&self) -> Value { - (*self).to_value() +#[cfg(feature = "std")] +impl From for Box { + fn from(err: Error) -> Self { + err.into_error() + } +} + +impl AsRef for Error { + fn as_ref(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { + self.as_error() + } +} + +#[cfg(feature = "std")] +impl std::error::Error for Inner { + fn description(&self) -> &str { + match self { + Inner::Static(msg) => msg, + Inner::Owned(msg) => msg, + } } } ``` -#### Implementors +There's no really universal way to handle errors in a logging pipeline. Knowing that some error occurred, and knowing where, should be enough for implementations of `Log` to decide how to handle it. The `Error` type doesn't try to be a general-purpose error management tool, it tries to make it easy to early-return with other errors. + +To make it possible to carry any arbitrary `S::Error` type, where we don't know how long the value can live for and whether it's `Send` or `Sync`, without extra work, the `Error` type does not attempt to store the error value itself. It just converts it into a `String`. + +### `value::Visit` -`ToValue` requires a blanket implementation to be most useful. This covers any `V: Display + Serialize`: +The `Visit` trait can be treated like a lightweight subset of `serde::Serialize` that can interoperate with `serde` without necessarily depending on it: ```rust -impl ToValue for T { - fn to_value(&self) -> Value { +/// A type that can be converted into a borrowed value. +pub trait Visit: private::Sealed { + /// Visit this value. + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; + + /// Convert a reference to this value into an erased `Value`. + fn to_value(&self) -> Value + where + Self: Sized, + { Value::new(self) } } + +mod private { + #[cfg(not(feature = "kv_serde"))] + pub trait Sealed: Debug {} + + #[cfg(feature = "kv_serde")] + pub trait Sealed: Debug + Serialize {} +} ``` -### `Value` +We'll look at the `Visitor` trait shortly. It's like `serde::Serializer`. + +`Visit` is the trait bound that structured values need to satisfy before they can be logged. The trait can't be implemented outside of the `log` crate, because it uses blanket implementations depending on Cargo features. If a crate defines a datastructure that users might want to log, instead of trying to implement `Visit`, it should implement the `serde::Serialize` and `std::fmt::Debug` traits. This means that `Visit` can piggyback off `serde::Serialize` as the pervasive public dependency, so that `Visit` itself doesn't need to be one. -A `Value` is a short-lived structure that can be serialized using `serde`. This might require losing some type information about the underlying value and serializing it as a string: +The trait bounds on `private::Sealed` ensure that any generic `T: Visit` carries some additional traits that are needed for the blanket implementation of `Serialize`. As an example, any `Option` can also be treated as `Option` and therefore implement `Serialize` itself. The `Visit` trait is responsible for a lot of type system mischief. + +With default features, the types that implement `Visit` are a subset of `T: Debug + Serialize`: + +``` +-------- feature = "kv_serde" -------- +| | +| T: Debug + Serialize | +| | +| | +| - not(feature = "kv_serde") - | +| | | | +| | u8, i8, &str, &[u8], bool | | +| | etc... | | +| | | | +| ----------------------------- | +| | +| | +-------------------------------------- +``` + +Enabling the `kv_serde` feature expands the set of types that implement `Visit` from this subset to all `T: Debug + Serialize`. + +#### Object safety + +The `Visit` trait is not object-safe, but has a simple object-safe wrapper used by `Value`. + +#### Without `serde` + +Without the `kv_serde` feature, the `Visit` trait is implemented for a fixed set of fundamental types from the standard library: ```rust -/// A value in a key-value pair. -/// -/// The value can be treated like `serde::Serialize`. -pub struct Value<'kvs> { - inner: ValueInner<'kvs>, +impl Visit for u8 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u64(*self as u64) + } } -#[derive(Clone, Copy)] -enum ValueInner<'kvs> { - /// The value will be serialized as a string - /// using its `Display` implementation. - Display(&'kvs dyn fmt::Display), - /// The value will be serialized as a structured - /// type using its `Serialize` implementation. - #[cfg(feature = "erased-serde")] - Serde(&'kvs dyn erased_serde::Serialize), - /// The value will be serialized as a structured - /// type using its primitive `Serialize` implementation. - #[cfg(not(feature = "erased-serde"))] - Primitive(&'kvs dyn ToPrimitive), +impl Visit for u16 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u64(*self as u64) + } } -impl<'kvs> ToValue for Value<'kvs> { - fn to_value(&self) -> Value { - Value { inner: self.inner } +impl Visit for u32 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u64(*self as u64) } } -impl<'a, 'kvs> ToValue for &'a Value<'kvs> { - fn to_value(&self) -> Value { - Value { inner: self.inner } +impl Visit for u64 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u64(*self) } } -impl<'kvs> Serialize for Value<'kvs> { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - match self.inner { - ValueInner::Display(v) => serializer.collect_str(&v), +impl Visit for i8 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_i64(*self as i64) + } +} + +impl Visit for i16 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_i64(*self as i64) + } +} - #[cfg(feature = "erased-serde")] - ValueInner::Serde(v) => v.serialize(serializer), +impl Visit for i32 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_i64(*self as i64) + } +} + +impl Visit for i64 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_i64(*self) + } +} - #[cfg(not(feature = "erased-serde"))] - ValueInner::Primitive(v) => { - // We expect `Value::new` to correctly determine - // whether or not a value is a simple primitive - let v = v - .to_primitive() - .ok_or_else(|| S::Error::custom("captured value is not primitive"))?; +impl Visit for f32 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_f64(*self as f64) + } +} - v.serialize(serializer) - }, +impl Visit for f64 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_f64(*self) + } +} + +impl Visit for char { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_char(*self) + } +} + +impl Visit for bool { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_bool(*self) + } +} + +impl Visit for () { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_none() + } +} + +#[cfg(feature = "i128")] +impl Visit for u128 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u128(*self) + } +} + +#[cfg(feature = "i128")] +impl Visit for i128 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_i128(*self) + } +} + +impl Visit for Option +where + T: Visit, +{ + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + match self { + Some(v) => v.visit(visitor), + None => visitor.visit_none(), + } + } +} + +impl<'a> Visit for fmt::Arguments<'a> { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_fmt(self) + } +} + +impl<'a> Visit for &'a str { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_str(self) + } +} + +impl<'a> Visit for &'a [u8] { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_bytes(self) + } +} + +impl<'v> Visit for Value<'v> { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + self.visit(visitor) + } +} + +#[cfg(feature = "std")] +impl Visit for Box +where + T: Visit +{ + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + (**self).visit(visitor) + } +} + +#[cfg(feature = "std")] +impl<'a> Visit for &'a Path { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + match self.to_str() { + Some(s) => visitor.visit_str(s), + None => visitor.visit_fmt(&format_args!("{:?}", self)), } } } + +#[cfg(feature = "std")] +impl Visit for String { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_str(&*self) + } +} + +#[cfg(feature = "std")] +impl Visit for Vec { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_bytes(&*self) + } +} + +#[cfg(feature = "std")] +impl Visit for PathBuf { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + self.as_path().visit(visitor) + } +} ``` -#### Capturing values +#### With `serde` -Methods on `Value` allow it to capture and erase types that implement combinations of `Serialize` and `Display`: +With the `kv_serde` feature, the `Visit` trait is implemented for any type that is `Debug + Serialize`: ```rust -impl<'kvs> Value<'kvs> { - /// Create a new value. - /// - /// The value must implement both `serde::Serialize` and `fmt::Display`. - /// Either implementation will be used depending on whether the standard - /// library is available, but is exposed through the same API. - /// - /// In environments where the standard library is available, the `Serialize` - /// implementation will be used. - /// - /// In environments where the standard library is not available, some - /// primitive stack-based values can retain their structure instead of falling - /// back to `Display`. - pub fn new(v: &'kvs (impl Serialize + Display)) -> Self { - Value { - inner: { - #[cfg(feature = "erased-serde")] - { - ValueInner::Serde(v) - } +#[cfg(feature = "kv_serde")] +impl Visit for T +where + T: Debug + Serialize {} +``` - #[cfg(not(feature = "erased-serde"))] - { - // Try capture a primitive value - if v.to_primitive().is_some() { - ValueInner::Primitive(v) - } else { - ValueInner::Display(v) - } - } - } +#### Ensuring the fixed set is a subset of the blanket implementation + +Changing trait implementations based on Cargo features is a dangerous game. Cargo features are additive, so any observable changes to trait implementations must also be purely additive, otherwise you can end up with libraries that can't compile if a feature is active. This can be very subtle when references and generics are involved. + +When the `kv_serde` feature is active, the implementaiton of `Visit` changes from a fixed set to an open one. We have to guarantee that the open set is a superset of the fixed one. That means any valid `T: Visit` without the `kv_serde` feature remains a valid `T: Visit` with the `kv_serde` feature. + +There are a few ways we could achieve this, depending on the quality of the docs we want to produce. + +For more readable documentation at the risk of incorrectly implementing `Visit`, we can use a private trait like `EnsureVisit: Visit` that is implemented alongside the concrete `Visit` trait regardless of any blanket implementations of `Visit`: + +```rust +// The blanket implemention of `Visit` when `kv_serde` is enabled +#[cfg(feature = "kv_serde")] +impl Visit for T where T: Debug + Serialize {} + +/// This trait is a private implementation detail for testing. +/// +/// All it does is make sure that our set of concrete types +/// that implement `Visit` always implement the `Visit` trait, +/// regardless of crate features and blanket implementations. +trait EnsureVisit: Visit {} + +// Ensure any reference to a `Visit` implements `Visit` +impl<'a, T> EnsureVisit for &'a T where T: Visit {} + +// These impl blocks always exists +impl EnsureVisit for Option where T: Visit {} +// This impl block only exists if the `kv_serde` isn't active +#[cfg(not(feature = "kv_serde"))] +impl private::Sealed for Option where T: Visit {} +#[cfg(not(feature = "kv_serde"))] +impl Visit for Option where T: Visit { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + + } +} +``` + +In the above example, we can ensure that `Option` always implements the `Visit` trait, whether it's done manually or as part of a blanket implementation. All types that implement `Visit` manually with any `#[cfg]` _must_ also always implement `EnsureVisit` manually (with no `#[cfg]`) with the exact same type bounds. It's pretty subtle, but the subtlety can be localized to a single module within the `log` crate so it can be managed. + +Using a trait for this type checking means the `impl Visit for Option` and `impl EnsureVisit for Option` can be wrapped up in a macro so that we never miss adding them. The below macro is an example of a (not very pretty) one that can add the needed implementations of `EnsureVisit` along with the regular `Visit`: + +```rust +macro_rules! impl_to_value { + () => {}; + ( + impl: { $($params:tt)* } + where: { $($where:tt)* } + $ty:ty: { $($serialize:tt)* } + $($rest:tt)* + ) => { + impl<$($params)*> EnsureVisit for $ty + where + $($where)* {} + + #[cfg(not(feature = "kv_serde"))] + impl<$($params)*> private::Sealed for $ty + where + $($where)* {} + + #[cfg(not(feature = "kv_serde"))] + impl<$($params)*> Visit for $ty + where + $($where)* + { + $($serialize)* + } + + impl_to_value!($($rest)*); + }; + ( + impl: { $($params:tt)* } + $ty:ty: { $($serialize:tt)* } + $($rest:tt)* + ) => { + impl_to_value! { + impl: {$($params)*} where: {} $ty: { $($serialize)* } $($rest)* + } + }; + ( + $ty:ty: { $($serialize:tt)* } + $($rest:tt)* + ) => { + impl_to_value! { + impl: {} where: {} $ty: { $($serialize)* } $($rest)* } } +} + +// Ensure any reference to a `Visit` is also `Visit` +impl<'a, T> EnsureVisit for &'a T where T: Visit {} - /// Get a `Value` from a displayable reference. - pub fn from_display(v: &'kvs impl Display) -> Self { - Value { - inner: ValueInner::Display(v), +impl_to_value! { + u8: { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u64(*self as u64) } } - /// Get a `Value` from a serializable reference. - #[cfg(feature = "erased-serde")] - pub fn from_serde(v: &'kvs impl Serialize) -> Self { - Value { - inner: ValueInner::Serde(v), + impl: { T: Visit } Option: { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + match self { + Some(v) => v.to_value().visit(visitor), + None => visitor.visit_none(), + } } } + + ... } ``` -`Value::new` will choose either the `Serialize` or `Display` implementation, depending on whether `std` is available. If it is, `Value::new` will use `Serialize` and is equivalent to `Value::from_serde`, if it's not `Value::new` will use `Display` and is equivalent to `Value::from_display`. +We don't necessarily need a macro to make new implementations accessible for new contributors safely though. + +##### What about specialization? -`Value::new` can use the `Serialize` implementation for some fixed set of primitives, like `i32` and `bool` even if `std` is not available though. This can be done by capturing those values at the point that the serializer is known into a `Primitive` wrapper: +In a future Rust with specialization we might be able to avoid all the machinery needed to keep the manual impls consistent with the blanket one, and allow consumers to implement `Visit` without needing `serde`. The specifics of specialization are still up in the air though. Under the proposed _always applicable_ rule, manual implementations like `impl Visit for Option where T: Visit` wouldn't be allowed. The ` where specialize(T: Visit)` scheme might make it possible though, although this would probably be a breaking change in any case. + +### `Value` + +A `Value` is an erased container for a `Visit`, with a potentially short-lived lifetime: ```rust -/// Convert a value into a primitive with a known type. -/// -/// The `ToPrimitive` trait lets us pass trait objects around -/// that are always the same size, rather than bloating values -/// to the size of the largest primitive. -pub trait ToPrimitive { - /// Perform the conversion. - fn to_primitive(&self) -> Option; +/// The value in a key-value pair. +pub struct Value<'v>(ValueInner<'v>); + +enum ValueInner<'v> { + Erased(&'v dyn ErasedVisit), + Any(Any<'v>), +} + +impl<'v> Value<'v> { + /// Create a value. + pub fn new(v: &'v impl Visit) -> Self { + Value(ValueInner::Erased(v)) + } + + /// Create a value from an anonymous type. + /// + /// The value must be provided with a compatible visit method. + pub fn any(v: &'v T, visit: fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self + where + T: 'static, + { + Value(ValueInner::Any(Any::new(v, visit))) + } + + /// Visit the contents of this value with a visitor. + pub fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + match self.0 { + ValueInner::Erased(v) => v.erased_visit(visitor), + ValueInner::Any(ref v) => v.visit(visitor), + } + } } +``` + +#### `ErasedVisit` + +The `ErasedVisit` trait is an object-safe wrapper for the `Visit` trait. `Visit` itself isn't technically object-safe because it needs the non-object-safe `serde::Serialize` as a supertrait to carry in generic contexts: -impl ToPrimitive for T +```rust +#[cfg(not(feature = "kv_serde"))] +trait ErasedVisit: fmt::Debug { + fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; +} + +#[cfg(feature = "kv_serde")] +trait ErasedVisit: fmt::Debug + erased_serde::Serialize { + fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; +} + +impl ErasedVisit for T where - T: Serialize, + T: Visit, { - fn to_primitive(&self) -> Option { - self.serialize(PrimitiveSerializer).ok() + fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + self.visit(visitor) } } +``` -#[derive(Clone, Copy)] -pub struct Primitive(PrimitiveInner); +#### `Any` + +Other logging frameworks that want to integrate with `log` might not want to pull in a `serde` dependency, and so they couldn't implement the `Visit` trait. The `Any` type uses some `std::fmt` inspired black-magic to allow values that don't implement the `Visit` trait to be erased in a `Value`. It does this by taking a borrowed value along with a function pointer that looks like `Visit::visit`: + +```rust +struct Void { + _priv: (), + _oibit_remover: PhantomData<*mut dyn Fn()>, +} + +struct Any<'a> { + data: &'a Void, + visit: fn(&Void, &mut dyn Visitor) -> Result<(), Error>, +} + +impl<'a> Any<'a> { + fn new(data: &'a T, visit: fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self + where + T: 'static, + { + unsafe { + Any { + data: mem::transmute::<&'a T, &'a Void>(data), + visit: mem::transmute::< + fn(&T, &mut dyn Visitor) -> Result<(), Error>, + fn(&Void, &mut dyn Visitor) -> Result<(), Error>> + (visit), + } + } + } + + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + (self.visit)(self.data, visitor) + } +} +``` + +There's some scary code in `Any`, which is really just something like an ad-hoc trait object. + +#### Formatting + +`Value` always implements `Debug` and `Display` by forwarding to its inner value: + +```rust +impl<'v> fmt::Debug for Value<'v> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self.0 { + ValueInner::Erased(v) => v.fmt(f), + ValueInner::Any(ref v) => { + struct ValueFmt<'a, 'b>(&'a mut fmt::Formatter<'b>); + + impl<'a, 'b> Visitor for ValueFmt<'a, 'b> { + fn visit_any(&mut self, v: Value) -> Result<(), Error> { + write!(self.0, "{:?}", v)?; + + Ok(()) + } + } + + let mut visitor = ValueFmt(f); + v.visit(&mut visitor).map_err(|_| fmt::Error) + } + } + } +} + +impl<'v> fmt::Display for Value<'v> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{:?}", self) + } +} +``` + +#### Serialization + +When the `kv_serde` feature is enabled, `Value` implements the `Serialize` trait by forwarding to its inner value: + +```rust +impl<'v> Serialize for Value<'v> { + fn serialize(&self, serializer: S) -> Result + where + S: Serializer, + { + match self.0 { + ValueInner::Erased(v) => { + erased_serde::serialize(v, serializer) + }, + ValueInner::Any(ref v) => { + struct ErasedVisitSerde { + serializer: Option, + ok: Option, + } + + impl Visitor for ErasedVisitSerde + where + S: Serializer, + { + fn visit_any(&mut self, v: Value) -> Result<(), Error> { + let ok = v.serialize(self.serializer.take().expect("missing serializer"))?; + self.ok = Some(ok); + + Ok(()) + } + } + + let mut visitor = ErasedVisitSerde { + serializer: Some(serializer), + ok: None, + }; + + v.visit(&mut visitor).map_err(|e| e.into_serde())?; + Ok(visitor.ok.expect("missing return value")) + }, + } + } +} +``` + +#### Ownership + +The `Value` type borrows from its inner value. + +#### Thread-safety -#[derive(Clone, Copy)] -enum PrimitiveInner { - Unsigned(u64), - Signed(i64), - Float(f64), - Bool(bool), - Char(char), +The `Value` type doesn't try to guarantee that values are `Send` or `Sync`, and doesn't offer any way of retaining that information when erasing. - #[cfg(feature = "i128")] - BigUnsigned(u128), - - #[cfg(feature = "i128")] - BigSigned(i128), -} +### `value::Visitor` -impl Serialize for Primitive {} +A visitor for a `Value` that can interogate its structure: -struct PrimitiveSerializer; +```rust +/// A serializer for primitive values. +pub trait Visitor { + /// Visit an arbitrary value. + /// + /// Depending on crate features there are a few things + /// you can do with a value. You can: + /// + /// - format it using `Debug`. + /// - serialize it using `serde`. + fn visit_any(&mut self, v: Value) -> Result<(), Error>; -impl Serializer for PrimitiveSerializer { - type Ok = Primitive; - type Error = Invalid; + /// Visit a signed integer. + fn visit_i64(&mut self, v: i64) -> Result<(), Error> { + self.visit_any(v.to_value()) + } - ... -} -``` + /// Visit an unsigned integer. + fn visit_u64(&mut self, v: u64) -> Result<(), Error> { + self.visit_any(v.to_value()) + } -The `Primitive` type stored in a `Value` is another trait object. This is just to ensure the size of `Value` doesn't grow to the size of `Primitive`'s largest variant, which is a 128bit number. + /// Visit a 128bit signed integer. + #[cfg(feature = "i128")] + fn visit_i128(&mut self, v: i128) -> Result<(), Error> { + self.visit_any(v.to_value()) + } -The `Value::from_serde` method requires `std` because it uses `erased_serde` as an object-safe wrapper around `serde`, which itself requires `std`. + /// Visit a 128bit unsigned integer. + #[cfg(feature = "i128")] + fn visit_u128(&mut self, v: u128) -> Result<(), Error> { + self.visit_any(v.to_value()) + } -#### Ownership + /// Visit a floating point number. + fn visit_f64(&mut self, v: f64) -> Result<(), Error> { + self.visit_any(v.to_value()) + } -The `Value` type borrows from its inner value. + /// Visit a boolean. + fn visit_bool(&mut self, v: bool) -> Result<(), Error> { + self.visit_any(v.to_value()) + } -#### Thread-safety + /// Visit a single character. + fn visit_char(&mut self, v: char) -> Result<(), Error> { + let mut b = [0; 4]; + self.visit_str(&*v.encode_utf8(&mut b)) + } -The `Value` type doesn't try to guarantee that values are `Send` or `Sync`, and doesn't offer any way of retaining that information when erasing. + /// Visit a UTF8 string. + fn visit_str(&mut self, v: &str) -> Result<(), Error> { + self.visit_any((&v).to_value()) + } -### `ToKey` + /// Visit a raw byte buffer. + fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { + self.visit_any((&v).to_value()) + } -A `ToKey` is a potentially long-lived structure that can be converted into a `Key`: + /// Visit standard arguments. + fn visit_none(&mut self) -> Result<(), Error> { + self.visit_any(().to_value()) + } -```rust -/// A type that can be converted into a borrowed key. -pub trait ToKey { - /// Perform the conversion. - fn to_key(&self) -> Key; + /// Visit standard arguments. + fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error> { + self.visit_any(v.to_value()) + } } -impl<'a, K: ?Sized> ToKey for &'a K +impl<'a, T: ?Sized> Visitor for &'a mut T where - K: ToKey, + T: Visitor, { - fn to_key(&self) -> Key { - (*self).to_key() + fn visit_any(&mut self, v: Value) -> Result<(), Error> { + (**self).visit_any(v) } -} -``` -#### Implementors + fn visit_i64(&mut self, v: i64) -> Result<(), Error> { + (**self).visit_i64(v) + } -`ToKey` is implemented for `str`. This is supported in both `std` and `no_std` contexts: + fn visit_u64(&mut self, v: u64) -> Result<(), Error> { + (**self).visit_u64(v) + } -```rust -impl ToKey for str { - fn to_key(&self) -> Key { - Key::from_str(self) + #[cfg(feature = "i128")] + fn visit_i128(&mut self, v: i128) -> Result<(), Error> { + (**self).visit_i128(v) } -} -``` -When `std` is available, `Key` is also implemented for other string containers: + #[cfg(feature = "i128")] + fn visit_u128(&mut self, v: u128) -> Result<(), Error> { + (**self).visit_u128(v) + } -```rust -#[cfg(feature = "std")] -impl ToKey for String { - fn to_key(&self) -> Key { - Key::from_str(self) + fn visit_f64(&mut self, v: f64) -> Result<(), Error> { + (**self).visit_f64(v) } -} -#[cfg(feature = "std")] -impl<'a> ToKey for borrow::Cow<'a, str> { - fn to_key(&self) -> Key { - Key::from_str(self.as_ref()) + fn visit_bool(&mut self, v: bool) -> Result<(), Error> { + (**self).visit_bool(v) + } + + fn visit_char(&mut self, v: char) -> Result<(), Error> { + (**self).visit_char(v) + } + + fn visit_str(&mut self, v: &str) -> Result<(), Error> { + (**self).visit_str(v) + } + + fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { + (**self).visit_bytes(v) + } + + fn visit_none(&mut self) -> Result<(), Error> { + (**self).visit_none() + } + + fn visit_fmt(&mut self, args: &fmt::Arguments) -> Result<(), Error> { + (**self).visit_fmt(args) } } ``` @@ -988,7 +1615,7 @@ pub struct Key<'kvs> { inner: &'kvs str, } -impl<'kvs> ToKey for Key<'kvs> { +impl<'kvs> Borrow for Key<'kvs> { fn to_key(&self) -> Key { Key { inner: self.inner } } @@ -1043,19 +1670,19 @@ impl<'kvs> Debug for Key<'kvs> { } ``` -Other standard implementations could be added for any `K: AsRef` in the same fashion. +Other standard implementations could be added for any `K: Borrow` in the same fashion. #### Ownership -The `Key` type borrows its inner value. +The `Key` type can either borrow or own its inner value. #### Thread-safety The `Key` type is probably `Send` + `Sync`, but that's not guaranteed. -### `Visitor` +### `source::Visitor` -The `Visitor` trait used by `KeyValueSource` can visit a single key-value pair: +The `Visitor` trait used by `Source` can visit a single key-value pair: ```rust pub trait Visitor<'kvs> { @@ -1076,7 +1703,7 @@ A `Visitor` may serialize the keys and values as it sees them. It may also do ot #### Implementors -There aren't any public implementors of `Visitor` in the `log` crate, but the `KeyValueSource::try_for_each` and `KeyValueSource::serialize_as_map` methods use the trait internally. +There aren't any public implementors of `Visitor` in the `log` crate, but the `Source::try_for_each` and `Source::serialize_as_map` methods use the trait internally. Other crates that use key-value pairs will implement `Visitor`. @@ -1084,118 +1711,80 @@ Other crates that use key-value pairs will implement `Visitor`. The `Visitor` trait is object-safe. -### `Error` +### `Source` -Just about the only thing you can do with a `Value` in the `Visitor::visit_pair` method is serialize it with `serde`. Serialization might fail, so to allow errors to get carried back to callers the `visit_pair` method needs to return a `Result`. +The `Source` trait is a bit like `Serialize`. It gives us a way to inspect some arbitrary collection of key-value pairs using a visitor pattern: ```rust -pub struct Error(Inner); - -impl Error { - pub fn msg(msg: &'static str) -> Self { - Error(Inner::Static(msg)) - } - - #[cfg(feature = "std")] - pub fn as_error(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { - &self.0 - } - - #[cfg(feature = "std")] - pub fn into_error(self) -> Box { - Box::new(self.0) - } +pub trait Source { + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error>; - pub fn into_serde(self) -> E - where - E: serde::ser::Error, - { - E::custom(self) - } + ... } -#[cfg(feature = "std")] -impl From for Error +impl<'a, T: ?Sized> Source for &'a T where - E: std::error::Error, + T: Source, { - fn from(err: E) -> Self { - Error(Inner::Owned(err.to_string())) + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + (*self).visit(visitor) } } +``` -#[cfg(feature = "std")] -impl From for Box { - fn from(err: Error) -> Self { - err.into_error() - } -} +`Source` doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. -impl AsRef for Error { - fn as_ref(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { - self.as_error() - } -} +#### Ownership -enum Inner { - Static(&'static str), - #[cfg(feature = "std")] - Owned(String), -} +The `Source` trait is probably the point where having some way to convert from a borrowed to an owned variant would make the most sense. -#[cfg(feature = "std")] -impl std::error::Error for Inner { - fn description(&self) -> &str { - match self { - Inner::Static(msg) => msg, - Inner::Owned(msg) => msg, - } +We could add a method to `Source` that allowed it to be converted into an owned variant with a default implementation: + +```rust +pub trait Source { + fn to_owned(&self) -> OwnedSource { + OwnedSource::serialized(self) } } ``` -There's no really universal way to handle errors in a logging pipeline. Knowing that some error occurred, and knowing where, should be enough for implementations of `Log` to decide how to handle it. The `Error` type doesn't try to be a general-purpose error management tool, it tries to make it easy to early-return with other errors encountered during `Visitor::visit_pair`. - -To make it possible to carry any arbitrary `S::Error` type, where we don't know how long the value can live for and whether it's `Send` or `Sync`, without extra work, the `Error` type does not attempt to store the error value itself. It just converts it into a `String`. - -### `KeyValueSource` - -The `KeyValueSource` trait is a bit like `Serialize`. It gives us a way to inspect some arbitrary collection of key-value pairs using a visitor pattern: +The `OwnedSource` could then encapsulte some sharable `dyn Source + Send + Sync`: ```rust -pub trait KeyValueSource { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error>; +#[derive(Clone)] +pub struct OwnedSource(Arc); - ... -} +impl OwnedSource { + fn new(impl Into>) -> Self { + OwnedSource(source.into()) + } -impl<'a, T: ?Sized> KeyValueSource for &'a T -where - T: KeyValueSource, -{ - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { - (*self).visit(visitor) + fn serialize(impl Source) -> Self { + // Serialize the `Source` to something like + // `Vec<(String, OwnedValue)>` + // where `OwnedValue` is like `serde_json::Value` + ... } } ``` -`KeyValueSource` doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. +Other implementations of `Source` would be encouraged to override the `to_owned` method if they could provide a more efficient implementation. As an example, if there's a `Source` that is already wrapped up in an `Arc` then it can implement `to_owned` by just cloning itself. #### Adapters -Some useful adapters exist as provided methods on the `KeyValueSource` trait. They're similar to adapters on the standard `Iterator` trait: +Some useful adapters exist as provided methods on the `Source` trait. They're similar to adapters on the standard `Iterator` trait: ```rust -pub trait KeyValueSource { +pub trait Source { ... - /// Erase this `KeyValueSource` so it can be used without + /// Erase this `Source` so it can be used without /// requiring generic type parameters. - fn erase(&self) -> ErasedKeyValueSource + fn erase(&self) -> ErasedSource where Self: Sized, { - ErasedKeyValueSource::erased(self) + ErasedSource::erased(self) } /// An adapter to borrow self. @@ -1203,7 +1792,7 @@ pub trait KeyValueSource { self } - /// Chain two `KeyValueSource`s together. + /// Chain two `Source`s together. fn chain(self, other: KVS) -> Chained where Self: Sized, @@ -1222,7 +1811,7 @@ pub trait KeyValueSource { /// will do an indexed lookup instead of a scan. fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> where - Q: ToKey, + Q: Borrow, { struct Get<'k, 'v>(Key<'k>, Option>); @@ -1265,69 +1854,80 @@ pub trait KeyValueSource { } /// Serialize the key-value pairs as a map. + #[cfg(feature = "kv_serde")] fn serialize_as_map(self) -> SerializeAsMap where Self: Sized, { SerializeAsMap(self) } + + /// Serialize the key-value pairs as a map. + #[cfg(feature = "kv_serde")] + fn serialize_as_seq(self) -> SerializeAsSeq + where + Self: Sized, + { + SerializeAsSeq(self) + } } ``` -- `by_ref` to get a reference to a `KeyValueSource` within a method chain. +- `by_ref` to get a reference to a `Source` within a method chain. - `chain` to concatenate one source with another. This is useful for composing implementations of `Log` together for contextual logging. - `get` to try find the value associated with a key. - `try_for_each` to try execute some closure over all key-value pairs. This is a convenient way to do something with each key-value pair without having to create and implement a `Visitor`. - `serialize_as_map` to get a serializable map. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. +- `serialize_as_seq` to get a serializable sequence of tuples. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. None of these methods are required for the core API. They're helpful tools for working with key-value pairs with minimal machinery. Even if we don't necessarily include them right away it's worth having an API that can support them later without breakage. #### Object safety -`KeyValueSource` is not object-safe because of the provided adapter methods not being object-safe. The only required method, `visit`, is safe though, so an object-safe version of `KeyValueSource` that forwards this method can be reasonably written. +`Source` is not object-safe because of the provided adapter methods not being object-safe. The only required method, `visit`, is safe though, so an object-safe version of `Source` that forwards this method can be reasonably written. ```rust -/// An erased `KeyValueSource`. +/// An erased `Source`. #[derive(Clone)] -pub struct ErasedKeyValueSource<'a>(&'a dyn ErasedKeyValueSourceBridge); +pub struct ErasedSource<'a>(&'a dyn ErasedSourceBridge); -impl<'a> ErasedKeyValueSource<'a> { - /// Capture a `KeyValueSource` and erase its concrete type. - pub fn new(kvs: &'a impl KeyValueSource) -> Self { - ErasedKeyValueSource(kvs) +impl<'a> ErasedSource<'a> { + /// Capture a `Source` and erase its concrete type. + pub fn new(kvs: &'a impl Source) -> Self { + ErasedSource(kvs) } } -impl<'a> Default for ErasedKeyValueSource<'a> { +impl<'a> Default for ErasedSource<'a> { fn default() -> Self { - ErasedKeyValueSource(&(&[] as &[(&str, &dyn ToValue)])) + ErasedSource(&(&[] as &[(&str, &dyn Visit)])) } } -impl<'a> KeyValueSource for ErasedKeyValueSource<'a> { +impl<'a> Source for ErasedSource<'a> { fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { self.0.erased_visit(visitor) } fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> where - Q: ToKey, + Q: Borrow, { let key = key.to_key(); self.0.erased_get(key.as_ref()) } } -/// A trait that erases a `KeyValueSource` so it can be stored +/// A trait that erases a `Source` so it can be stored /// in a `Record` without requiring any generic parameters. -trait ErasedKeyValueSourceBridge { +trait ErasedSourceBridge { fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; fn erased_get<'kvs>(&'kvs self, key: &str) -> Option>; } -impl ErasedKeyValueSourceBridge for KVS +impl ErasedSourceBridge for KVS where - KVS: KeyValueSource + ?Sized, + KVS: Source + ?Sized, { fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { self.visit(visitor) @@ -1341,13 +1941,13 @@ where #### Implementors -A `KeyValueSource` with a single pair is implemented for a tuple of a key and value: +A `Source` with a single pair is implemented for a tuple of a key and value: ```rust -impl KeyValueSource for (K, V) +impl Source for (K, V) where - K: ToKey, - V: ToValue, + K: Borrow, + V: Visit, { fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { visitor.visit_pair(self.0.to_key(), self.1.to_value()) @@ -1355,10 +1955,10 @@ where } ``` -A `KeyValueSource` with multiple pairs is implemented for arrays of `KeyValueSource`s: +A `Source` with multiple pairs is implemented for arrays of `Source`s: ```rust -impl KeyValueSource for [KVS] where KVS: KeyValueSource { +impl Source for [KVS] where KVS: Source { fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { for kv in self { kv.visit(&mut visitor)?; @@ -1369,21 +1969,42 @@ impl KeyValueSource for [KVS] where KVS: KeyValueSource { } ``` -When `std` is available, `KeyValueSource` is implemented for some standard collections too: +When `std` is available, `Source` is implemented for some standard collections too: ```rust #[cfg(feature = "std")] -impl KeyValueSource for Vec where KVS: KeyValueSource { +impl Source for Box where KVS: Source { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + (**self).visit(visitor) + } +} + +#[cfg(feature = "std")] +impl Source for Arc where KVS: Source { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + (**self).visit(visitor) + } +} + +#[cfg(feature = "std")] +impl Source for Rc where KVS: Source { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + (**self).visit(visitor) + } +} + +#[cfg(feature = "std")] +impl Source for Vec where KVS: Source { fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { self.as_slice().visit(visitor) } } #[cfg(feature = "std")] -impl KeyValueSource for collections::BTreeMap +impl Source for collections::BTreeMap where K: Borrow + Ord, - V: ToValue, + V: Visit, { fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { for (k, v) in self { @@ -1395,18 +2016,18 @@ where fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> where - Q: ToKey, + Q: Borrow, { let key = key.to_key(); - collections::BTreeMap::get(self, key.as_ref()).map(ToValue::to_value) + collections::BTreeMap::get(self, key.as_ref()).map(Visit::to_value) } } #[cfg(feature = "std")] -impl KeyValueSource for collections::HashMap +impl Source for collections::HashMap where K: Borrow + Eq + Hash, - V: ToValue, + V: Visit, { fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { for (k, v) in self { @@ -1418,15 +2039,15 @@ where fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> where - Q: ToKey, + Q: Borrow, { let key = key.to_key(); - collections::HashMap::get(self, key.as_ref()).map(ToValue::to_value) + collections::HashMap::get(self, key.as_ref()).map(Visit::to_value) } } ``` -The `BTreeMap` and `HashMap` implementations provide more efficient implementations of `KeyValueSource::get`. +The `BTreeMap` and `HashMap` implementations provide more efficient implementations of `Source::get`. ## The `log!` macros @@ -1482,7 +2103,7 @@ Will expand to something like: let correlation &correlation_id; let user = &user; - let kvs: &[(&str, &dyn::key_values::ToValue)] = + let kvs: &[(&str, &dyn::key_values::Visit)] = &[("correlation", &correlation), ("user", &user)]; ::__private_api_log( @@ -1500,221 +2121,48 @@ Will expand to something like: }; ``` -### Logging values that aren't `ToValue` - -Not every type a user might want to log will satisfy the default `Display + Serialize` bound. To reduce friction in these cases, there are a few helper macros that can be used to tweak the way structured data is captured. - -The pattern here is pretty general, you could imagine other macros being created for capturing other useful trait implementors, like `T: Fail`. +# Drawbacks, rationale, and alternatives +[drawbacks]: #drawbacks -#### `log_serde!` +Structured logging is a non-trivial feature to support. -The `log_serde!` macro allows any type that implements just `Serialize` to be logged: +## The `Debug + Serialize` blanket implementation of `Visit` -```rust -info!( - "A log statement"; - user = log_serde!(user) -); -``` +Making sure the `Visit` trait doesn't drop any implementations when the blanket implementation from `kv_serde` replaces the concrete ones is subtle and nonstandard. We have to be especially careful of references and generics. Any mistakes made here can result in dependencies that become uncompilable depending on Cargo features with no workaround besides removing that impl. Using a macro to define the small fixed set, and keeping all impls local to a single module, could help catch these cases. -The macro definition looks like this: +It's also possibly surprising that the way the `Visit` trait is implemented in the ecosystem is through an entirely unrelated combination of `serde` and `std` traits. At least it's surprising on the surface. For libraries that define loggable types, they just implement some standard traits for serialization without involving `log` at all. These are traits they should be considering anyway. For consumers of the `log!` macro, they are mostly going to capture structured values for types they didn't produce, so having `serde` as the answer to _how can I log a `Url`, or a `Uuid`?_ sounds reasonable. It also means libraries defining types like `Url` and `Uuid` don't have yet another public serialization trait to implement. -```rust -macro_rules! log_serde { - ($v:expr) => { - $crate::adapter::log_serde(&$v) - } -} +If a library provides a datatype that you'd reasonably want to log, but it doesn't implement `serde::Serialize` then adding support for that type isn't just beneficial to you, but to anyone else that might want to serialize that type. -#[cfg(feature = "erased-serde")] -pub fn log_serde(v: impl Serialize) -> impl ToValue { - struct SerdeAdapter(T); +The real question for `serde` is whether or not depending on it as the general serialization framework in `log` creates the potential for some kind of ecosystem dichotomy if an alternative framework becomes popular where half the ecosystem uses `serde` and the other half uses something else that's incompatible. In that case `log` might not reasonably be able to support both without breakage if it goes down this path. The options for mitigating this in the design now is by either require all loggable types implement `Visit` explicitly, or just requiring callers opt in to `serde` support at the callsite in `log!`. - impl ToValue for SerdeAdapter - where - T: Serialize, - { - fn to_value(&self) -> Value { - Value::from_serde(&self.0) - } - } +### Require all loggable types implement `Visit` - SerdeAdapter(v) -} -``` +We could entirely punt on `serde` and just provide an API for simple values that implement the simple `Visit` trait. That avoids the potential serialization dichotomy in `log` altogether. -#### `log_fmt!` +The problem here is that any pervasive public API has the chance to create rifts in the ecosystem. By creating a new fundamental API for logging via the `Visit` trait we're just expanding the potential for dichotomies. -The `log_fmt!` macro allows any type to be logged as a formatted string: +It also means we need to re-invent `serde`'s support for complex datastructures, the datatypes that implement its traits, and the formats that support it. We'll effectively turn `log` into a serialization framework of its own, and have to introduce arbitrary limitations on the kinds of values that can be logged. -```rust -info!( - "A log statement"; - user = log_fmt!(user, Debug::fmt) -); -``` +### Require callers opt in to `serde` support -The macro definition looks like this: +We could avoid a potential serialization dichotomy by requiring callers opt in to `serde` support. That way if a new framework came along it could be naturally supported in the same way. There are a few ways callers could opt in to `serde` in the `log!` macros. The specifics aren't really important, but it could look something like this: ```rust -macro_rules! log_fmt { - ($v:expr, $f:expr) => { - $crate::adapter::log_fmt(&$v, $f) - } -} - -pub fn log_fmt(value: T, adapter: impl Fn(&T, &mut fmt::Formatter) -> fmt::Result) -> impl ToValue { - struct FmtAdapter { - value: T, - adapter: F, - } - - impl Display for FmtAdapter - where - F: Fn(&T, &mut Formatter) -> fmt::Result, - { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - (self.adapter)(&self.value, f) - } - } - - impl ToValue for FmtAdapter - where - F: Fn(&T, &mut Formatter) -> fmt::Result, - { - fn to_value(&self) -> Value { - Value::from_display(self) - } - } - - FmtAdapter { value, adapter } -} -``` - -#### `log_path!` - -The `log_path!` macro allows `Path` and `PathBuf` to be logged: +use log::log_serde; -```rust -info!( - "A log statement"; - path = log_path!(path) -); +info!("A message"; user = log_serde!(user)); ``` -The macro definition looks like this: +That way an alternative framework could be supported as: ```rust -macro_rules! log_path { - ($v:expr) => { - $crate::adapter::log_path(&$v) - } -} - -#[cfg(feature = "std")] -pub fn log_path(v: impl AsRef) -> impl ToValue { - #[derive(Debug)] - struct PathAdapter(T); - - impl Display for PathAdapter - where - T: AsRef, - { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - let path = self.0.as_ref(); - - match path.to_str() { - Some(path) => Display::fmt(path, f), - None => Debug::fmt(path, f), - } - } - } - - impl serde::Serialize for PathAdapter - where - T: AsRef, - { - fn serialize(&self, serializer: S) -> Result - where - S: serde::Serializer - { - serializer.collect_str(&self) - } - } - - PathAdapter(v) -} -``` - -# Drawbacks -[drawbacks]: #drawbacks - -Structured logging is a non-trivial feature to support. - -## `Display + Serialize` - -Using `Display + Serialize` as a blanket implementation for `ToValue` means we immediately need to work around some values that will probably be logged, but don't satisfy the bound: - -- `Path` and `PathBuf` -- Most `std::error::Error`s -- Most `failure::Fail`s - -Using `Debug + Serialize` would allow `Path`s to be logged with no extra effort, but `Display` is probably a more appropriate trait to use. - -In `no_std` contexts, more machinery is required to try retain the structure of primitive values, rather than falling back to string serialization through the `Display` bound. - -## `serde` - -Using `serde` as the serialization framework for structured logging introduces a lot of complexity that consumers of key-value pairs need to deal with. Inspecting values requires an implementation of a `Serializer`, which is a complex trait. - -Having `serde` enabled by default means it'll be effectively impossible to compile `log` in any reasonably sized dependency graph without also compiling `serde`. +use log::log_other_framework; -# Rationale and alternatives -[rationale-and-alternatives]: #rationale-and-alternatives - -## Just use `Display` - -This API could be a lot simpler if key-value pairs were only required to implement the `Display` trait. Unfortunately, this doesn't really provide structured logging, because `Display` doesn't retain any structure. Take the json example from before: - -```json -{ - "ts": 1538040723000, - "lvl": "INFO", - "msg": "Operation completed successfully in 18ms", - "module": "basic", - "service": "database", - "correlation": 123, - "took": 18 -} -``` - -If key-value pairs were implemented as `Display` then this json object would look like this: - -```json -{ - "ts": "1538040723000", - "lvl": "INFO", - "msg": "Operation completed successfully in 18ms", - "module": "basic", - "service": "database", - "correlation": "123", - "took": "18" -} +info!("A message"; user = log_other_framework!(user)); ``` -Now without knowing that `ts` is a number we can't reasonably use that field for querying over ranges of events. - -## Don't use a blanket implementation - -Without a blanket implementation of `ToValue`, a set of primitive and standard types could manually implement the trait in `log`. That could include `Path` and `PathBuf`, which don't currently satify the blanket `Display + Serialize` bounds. This would require libraries providing types in the ecosystem to depend on `log` and implement `ToValue` themselves. - -## Define our own serialization trait - -Rather than rely on `serde`, define our own simplified, object-safe serialization trait. This would avoid the complexity of erasing `Serialize` implementations, but would introduce the same ecosystem leakage as not using a blanket implementation. It would also be a trait that looks a lot like `Serialize`, without necessarily keeping up with improvements that are made in `serde`. - -## Don't enable structured logging by default - -Make structured logging a purely optional feature, so it wouldn't necessarily need to support `no_std` environments at all and could avoid the `Display + Serialize` blanket implementation bounds on `ToValue`. This seems appealing, but isn't ideal because it makes a fundamental modern logging feature much less discoverable. +The problem with this approach is that it puts extra barriers in front of users that want to log. Instead of enabling crate features once and then logging structured values, each log statement needs to know how it can capture values. It also passes the burden of dealing with dichotomies onto every consumer of `log`. It seems like a reasonable idea from the perspective of the `log` crate, but is more hostile to end-users. # Prior art [prior-art]: #prior-art @@ -1739,10 +2187,3 @@ Supporting something like message templates in Rust using the `log!` macros woul # Unresolved questions [unresolved-questions]: #unresolved-questions - -- What parts of the design do you expect to resolve through the RFC process before this gets merged? -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? -- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? - ------ - From 17912c26b5283364734ca2ca29d92a118601a0ea Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Thu, 8 Nov 2018 17:31:47 +1000 Subject: [PATCH 03/20] address some pr comments and reorder sections --- rfcs/0000-structured-logging.md | 2640 ++++++++++++++++--------------- 1 file changed, 1357 insertions(+), 1283 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 05ad9338f..1f3b095f0 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -3,7 +3,9 @@ Add support for structured logging to the `log` crate in both `std` and `no_std` environments, allowing log records to carry typed data beyond a textual message. This document serves as an introduction to what structured logging is all about, and as an RFC for an implementation in the `log` crate. -`log` will provide a lightweight fundamental serialization API out-of-the-box that can be expanded to full `serde` support using crate features. It doesn't require crates implement any traits from `log` before their datastructures can be used in the `log!` macros. Instead, crates should implement `std::fmt::Debug` and `serde::Serialize`, which are traits they should by considering already. The API is heavily inspired by the `slog` logging framework. See [the implications for `log` users](#implications-for-dependents) for some more details. +`log` will provide a lightweight fundamental serialization API out-of-the-box that allows a fixed set of common types from the standard library to be logged as structured values. Using optional Cargo features, that set can be expanded to support anything that implements `serde::Serialize + std::fmt::Debug`. It doesn't turn `log` into a pervasive public dependency to support structured logging for types outside the standard library. + +The API is heavily inspired by the `slog` logging framework. > NOTE: Code in this RFC uses recent language features like `impl Trait`, but can be implemented without them. @@ -21,8 +23,7 @@ Add support for structured logging to the `log` crate in both `std` and `no_std` - [Design considerations](#design-considerations) - [Implications for dependents](#implications-for-dependents) - [Cargo features](#cargo-features) - - [Public API](#public-api) - - [`Record` and `RecordBuilder`](#record-and-recordbuilder) + - [Key-values API](#key-values-api) - [`Visit`](#Visit) - [`Value`](#value) - [`Key`](#key) @@ -30,6 +31,7 @@ Add support for structured logging to the `log` crate in both `std` and `no_std` - [`Error`](#error) - [`Source`](#source) - [`source::Visitor`](#sourcevisitor) + - [`Record` and `RecordBuilder`](#record-and-recordbuilder) - [The `log!` macros](#the-log-macros) - [Drawbacks, rationale, and alternatives](#drawbacks-rationale-and-alternatives) - [Prior art](#prior-art) @@ -37,6 +39,8 @@ Add support for structured logging to the `log` crate in both `std` and `no_std` - [Go](#go) - [.NET](#net) - [Unresolved questions](#unresolved-questions) +- [Appendix](#appendix) + - [Public API](#public-api) # Motivation [motivation]: #motivation @@ -109,7 +113,7 @@ info!( ); ``` -Any `value` or `key = value` expressions before the `;` in the macro will be interpolated into the message as unstructured text. This is the `log!` macro we have today. Any `value` or `key = value` expressions after the `;` will be captured as structured key-value pairs. These structured key-value pairs can be inspected or serialized, retaining some notion of their original type. That means in the above example, the `message` key is unstructured, and the `correlation` and `user` keys are structured: +Any `value` or `key = value` expressions before the `;` in the macro will be interpolated into the message as unstructured text using `std::fmt`. This is the `log!` macro we have today. Any `value` or `key = value` expressions after the `;` will be captured as structured key-value pairs. These structured key-value pairs can be inspected or serialized, retaining some notion of their original type. That means in the above example, the `message` key is unstructured, and the `correlation` and `user` keys are structured: ``` info!( @@ -130,7 +134,7 @@ info!( ### What can be logged? -Anything that's `std::fmt::Debug + serde::Serialize` can be logged as a structured value. Using default crate features though, only a fixed set of types from the standard library are supported: +Using default Cargo features, a closed set of types from the standard library are supported as structured values: - Standard formats: `Arguments` - Primitives: `bool`, `char` @@ -139,9 +143,9 @@ Anything that's `std::fmt::Debug + serde::Serialize` can be logged as a structur - Strings: `&str`, `String` - Bytes: `&[u8]`, `Vec` - Paths: `Path`, `PathBuf` -- Special types: `Option` and `()` +- Special types: `Option` and `()`. -This keeps the `log` crate lightweight by default. In the example from before, `correlation_id` and `user` can be used as structured values if they're in that set of types: +In the example from before, `correlation_id` and `user` can be used as structured values if they're in that set of concrete types: ```rust let user = "a user id"; @@ -155,7 +159,7 @@ info!( ); ``` -What if the `correlation_id` is a `uuid::Uuid` instead of a string? What if the `user` is some other datastructure containing an id along with some other metadata? We still want to be able to log these values. The set of loggable values above can be expanded to also include any other type that implements both `std::fmt::Debug` and `serde::Serialize` using a crate feature: +What if the `correlation_id` is a `uuid::Uuid` instead of a string? What if the `user` is some other datastructure containing an id along with some other metadata? Only being able to log a few types from the standard library is a bit limiting. To make logging other values possible, the `kv_serde` Cargo feature expands the set of loggable values above to also include any other type that implements both `std::fmt::Debug` and `serde::Serialize`: ```toml [dependencies.log] @@ -202,11 +206,21 @@ info!( ); ``` -If you come across a data type in the Rust ecosystem that you can't log, then try looking for a `serde` feature on the crate that data type comes from. If there isn't one already then adding it will be useful not just for you, but for anyone that might want to serialize those types for other reasons. +If you come across a data type in the Rust ecosystem that you can't log, then try looking for a `serde` feature on the crate that defines it. If there isn't one already then adding it will be useful not just for you, but for anyone that might want to serialize those types for other reasons. ## Supporting key-value pairs in `Log` implementations -Capturing structured logs is only half the story. Implementors of the `Log` trait also need to be able to work with any key-value pairs associated with a log record. Take the terminal log format from before: +Capturing structured logs is only half the story. Implementors of the `Log` trait also need to be able to work with any key-value pairs associated with a log record. Key-value pairs are accessible on a log record through the `key_values` method: + +```rust +impl Record { + pub fn key_values(&self) -> impl Source; +} +``` + +where `Source` is a trait for iterating over the individual key-value pairs. + +To demonstrate how to work with a `Source`, let's take the terminal log format from before: ``` [INF 2018-09-27T09:32:03Z] Operation completed successfully in 18ms @@ -216,7 +230,7 @@ correlation: 123 took: 18 ``` -Each key-value pair, shown as `$key: $value`, can be formatted using the `std::fmt` machinery: +Each key-value pair, shown as `$key: $value`, can be formatted from the `Source` using the `std::fmt` machinery: ```rust use log::kv::Source; @@ -228,12 +242,13 @@ fn write_pretty(w: impl Write, r: &Record) -> io::Result<()> { // Write each key-value pair on a new line record .key_values() - .try_for_each(|k, v| writeln!("{}: {}", k, v)) - .map_err(|e| io::Error::new(io::ErrorKind::Other, e.into_error())) + .try_for_each(|k, v| writeln!("{}: {}", k, v))?; + + Ok(()) } ``` -Now take the following json format: +In the above example, the `try_for_each` method iterates over each key-value pair and writes them to the terminal. Now take the following json format: ```json { @@ -282,11 +297,33 @@ struct SerializeRecord { } ``` -The crate that produces log records might not be the same crate that consumes them. A producer can depend on the `kv_serde` feature to log more types, but still have those types handled by a consumer that doesn't use the `kv_serde` feature. +This time, instead of using the `try_for_each` method, we use `serialize_as_map` to get an adapter that will serialize each key-value pair as an entry in a map. + +The crate that produces log records might not be the same crate that consumes them. A producer can depend on the `kv_serde` feature to log more types, and a consumer will always be able to handle them, even if they don't depend on the `kv_serde` feature. ## Integrating log frameworks with `log` -The `Source` trait describes some container for structured key-value pairs. Other log frameworks that want to integrate with `log::Log` should build `Record`s that contain some implementation of `Source`. +The `Source` trait describes some container for structured key-value pairs. Other log frameworks that want to integrate with the `log` crate should build `Record`s that contain some implementation of `Source` based on their own structured logging. + +The previous section demonstrated some of the methods available on `Source` like `Source::try_for_each` and `Source::serialize_as_map`. Both of those methods are provided on top of a single required `Source::visit` method. The `Source` trait itself looks something like this: + +```rust +trait Source { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; + + // Provided methods +} +``` + +where `source::Visitor` is another trait that accepts individual key-value pairs: + +```rust +trait Visitor<'kvs> { + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; +} +``` + +The following example wraps up a `BTreeMap` and implements the `Source` trait for it: ```rust use log::kv::source::{self, Source}; @@ -302,7 +339,7 @@ impl Source for MySource { } ``` -`BTreeMap` happens to already implement the `Source` trait. Let's assume `BTreeMap` didn't implement `Source`. A manual implementation converting the `(String, serde_json::Value)` pairs into types that can be visited could look like this: +The implementation is pretty trivial because `BTreeMap` happens to already implement the `Source` trait. Now let's assume `BTreeMap` didn't implement `Source`. A manual implementation iterating through the map and converting the `(String, serde_json::Value)` pairs into types that can be visited could look like this: ```rust use log::kv::{ @@ -342,7 +379,7 @@ impl Source for MySource { // Let's assume `MyValue` doesn't implement `Serialize` // Instead it implements `Display`. - let value = source::Value::any(v, |v, visitor| { + let value = source::Value::any(v: &MyValue, |v: &MyValue, visitor: &mut dyn value::Visitor| { // Let's assume `MyValue` implements `Display` visitor.visit_fmt(format_args!("{}", v)) }); @@ -352,6 +389,24 @@ impl Source for MySource { } ``` +The `value::Visitor` trait is similar to `serde::Serializer`, but only supports a few common types: + +```rust +trait Visitor { + fn visit_i64(&mut self, v: i64) -> Result<(), Error>; + fn visit_u64(&mut self, v: u64) -> Result<(), Error>; + fn visit_i128(&mut self, v: i128) -> Result<(), Error>; + fn visit_u128(&mut self, v: u128) -> Result<(), Error>; + fn visit_f64(&mut self, v: f64) -> Result<(), Error>; + fn visit_bool(&mut self, v: bool) -> Result<(), Error>; + fn visit_char(&mut self, v: char) -> Result<(), Error>; + fn visit_str(&mut self, v: &str) -> Result<(), Error>; + fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error>; + fn visit_none(&mut self) -> Result<(), Error>; + fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error>' +} +``` + A `Source` doesn't have to just contain key-value pairs directly like `BTreeMap` though. It could also act like an adapter, like we have for iterators in the standard library. As another example, the following `Source` doesn't store any key-value pairs of its own, it will sort and de-duplicate pairs read from another source by first reading them into a map before forwarding them on: ```rust @@ -392,8 +447,6 @@ where } ``` -This API is similar to `serde` and `slog`, and is very flexible. Other structured logging concepts like contextual logging, where a record is enriched with information from its envirnment, can be built on top of `Source` and `Visitor`. - ## Writing your own `value::Visitor` Consumers of key-value pairs can visit structured values without needing a `serde::Serializer`. Instead they can implement a `value::Visitor`. A `Visitor` can always visit any structured value by formatting it using its `Debug` implementation: @@ -430,28 +483,36 @@ Allow structured logging to be added in the current `0.4.x` series of `log`. Thi Don't create a new serialization framework that causes `log` to become a public dependency of any library that wants their data types to be loggable. Logging is a truly cross-cutting concern, so if `log` was a public dependency it would probably be at least as pervasive as `serde` is now. -### Object safety +### Support arbitrary producers and arbitrary consumers -`log` is already designed to be object-safe so this new structured logging API needs to be object-safe too. +Provide an API that's suitable for two independent logging frameworks to integrate through if they want. -### Borrowed vs owned +### Prioritize end-users of `log!` -`Record` borrows all data from the call-site so records need to be handled directly on-thread as they're produced. On the one hand that means that log records need to be serialized before they can be sent across threads. On the other hand it means callers don't need to make assumptions about whether records need to be owned or borrowed. +There are far more consumers of the `log!` macros that don't need to worry about the internals of the `log` crate than there are log frameworks and sinks that do so it makes sense to prioritize `log!` ergonomics. + +### Object safety -A scheme for more efficient conversion from borrowed to owned sources of key-value pairs can be added later. +`log` is already designed to be object-safe so this new structured logging API needs to be object-safe too. ## Cargo features -Structured logging will be supported in either `std` or `no_std` contexts using Cargo features: +Structured logging will be supported in either `std` or `no_std` contexts by default. ```toml [features] -# optional serde-support requires `std` kv_serde = ["std", "serde", "erased-serde"] +i128 = [] ``` +### `kv_serde` + Using default features, structured logging will be supported by `log` in `no_std` environments for a fixed set of types from the standard library. Using the `kv_serde` feature, any type that implements `Debug + Serialize` can be logged, and its potentially complex structure will be retained. +### `i128` + +Add support for 128bit numbers without bumping `log`'s current minimally supported version of `rustc`. + ## Implications for dependents Dependents of `log` will notice the following: @@ -474,1716 +535,1729 @@ In `std` environments (which is common when using `env_logger` and other crates In either case, `serde` will become a public dependency of the `log` crate, so any breaking changes to `serde` will result in breaking changes to `log`. -## Public API +## Key-values API -For context, ignoring the `log!` macros, this is roughly the additional public API this RFC proposes to support structured logging: +### `Error` -```rust -impl<'a> RecordBuilder<'a> { - /// Set the key-value pairs on a log record. - pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a>; -} +Just about the only things you can do with a structured value are format it or serialize it. Serialization and writing might fail, so to allow errors to get carried back to callers there needs to be a general error type that they can early return with: -impl<'a> Record<'a> { - /// Get the key-value pairs. - pub fn key_values(&self) -> ErasedSource; +```rust +pub struct Error(Inner); - /// Get a builder that's preconfigured from this record. - pub fn to_builder(&self) -> RecordBuilder; +enum Inner { + Static(&'static str), + #[cfg(feature = "std")] + Owned(String), } -pub mod kv { - pub mod source { - pub use kv::Error; - - /// A source for key-value pairs. - pub trait Source { - /// Serialize the key value pairs. - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; - - /// Erase this `Source` so it can be used without - /// requiring generic type parameters. - fn erase(&self) -> ErasedSource - where - Self: Sized {} +impl Error { + pub fn msg(msg: &'static str) -> Self { + Error(Inner::Static(msg)) + } - /// Find the value for a given key. - /// - /// If the key is present multiple times, this method will - /// return the *last* value for the given key. - fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> - where - Q: Borrow {} + #[cfg(feature = "std")] + pub fn as_error(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { + &self.0 + } - /// An adapter to borrow self. - fn by_ref(&self) -> &Self {} + #[cfg(feature = "std")] + pub fn into_error(self) -> Box { + Box::new(self.0) + } - /// Chain two `Source`s together. - fn chain(self, other: KVS) -> Chain - where - Self: Sized {} + #[cfg(feature = "kv_serde")] + pub fn into_serde(self) -> E + where + E: serde::ser::Error, + { + E::custom(self) + } +} - /// Apply a function to each key-value pair. - fn try_for_each(self, f: F) -> Result<(), Error> - where - Self: Sized, - F: FnMut(Key, Value) -> Result<(), E>, - E: Into {} +#[cfg(feature = "std")] +impl From for Error +where + E: std::error::Error, +{ + fn from(err: E) -> Self { + Error(Inner::Owned(err.to_string())) + } +} - /// Serialize the key-value pairs as a map. - fn serialize_as_map(self) -> SerializeAsMap - where - Self: Sized {} +#[cfg(feature = "std")] +impl From for Box { + fn from(err: Error) -> Self { + err.into_error() + } +} - /// Serialize the key-value pairs as a sequence of tuples. - fn serialize_as_seq(self) -> SerializeAsSeq - where - Self: Sized {} - } +#[cfg(feature = "std")] +impl From for io::Error { + fn from(err: Error) -> Self { - /// A visitor for a set of key-value pairs. - /// - /// The visitor is driven by an implementation of `Source`. - /// The visitor expects keys and values that satisfy a given lifetime. - pub trait Visitor<'kvs> { - /// Visit a single key-value pair. - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; - } + } +} - /// An erased `Source`. - pub struct ErasedSource<'a> {} +impl AsRef for Error { + fn as_ref(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { + self.as_error() + } +} - impl<'a> ErasedSource<'a> { - /// Capture a `Source` and erase its concrete type. - pub fn new(kvs: &'a impl Source) -> Self {} +#[cfg(feature = "std")] +impl std::error::Error for Inner { + fn description(&self) -> &str { + match self { + Inner::Static(msg) => msg, + Inner::Owned(msg) => msg, } + } +} +``` - impl<'a> Clone for ErasedSource<'a> {} - impl<'a> Default for ErasedSource<'a> {} - impl<'a> Source for ErasedSource<'a> {} - - /// A `Source` adapter that visits key-value pairs - /// in sequence. - /// - /// This is the result of calling `chain` on a `Source`. - pub struct Chain {} +There's no really universal way to handle errors in a logging pipeline. Knowing that some error occurred, and knowing where, should be enough for implementations of `Log` to decide how to handle it. The `Error` type doesn't try to be a general-purpose error management tool, it tries to make it easy to early-return with other errors. - impl Source for Chain - where - A: Source, - B: Source {} +To make it possible to carry any arbitrary `S::Error` type, where we don't know how long the value can live for and whether it's `Send` or `Sync`, without extra work, the `Error` type does not attempt to store the error value itself. It just converts it into a `String`. - /// A `Source` adapter that can be serialized as - /// a map using `serde`. - /// - /// This is the result of calling `serialize_as_map` on - /// a `Source`. - pub struct SerializeAsMap {} +### `value::Visit` - impl Serialize for SerializeAsMap - where - KVS: Source {} +The `Visit` trait can be treated like a lightweight subset of `serde::Serialize` that can interoperate with `serde` without necessarily depending on it: - /// A `Source` adapter that can be serialized as - /// a sequence of tuples using `serde`. - /// - /// This is the result of calling `serialize_as_seq` on - /// a `Source`. - pub struct SerializeAsSeq {} +```rust +/// A type that can be converted into a borrowed value. +pub trait Visit: private::Sealed { + /// Visit this value. + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; - impl Serialize for SerializeAsSeq - where - KVS: Source {} + /// Convert a reference to this value into an erased `Value`. + fn to_value(&self) -> Value + where + Self: Sized, + { + Value::new(self) + } +} - impl Source for (K, V) - where - K: Borrow, - V: kv::value::Visit {} +mod private { + #[cfg(not(feature = "kv_serde"))] + pub trait Sealed: Debug {} - impl Source for [KVS] - where - KVS: Source {} + #[cfg(feature = "kv_serde")] + pub trait Sealed: Debug + Serialize {} +} +``` - #[cfg(feature = "std")] - impl Source for Box where KVS: Source {} - #[cfg(feature = "std")] - impl Source for Arc where KVS: Source {} - #[cfg(feature = "std")] - impl Source for Rc where KVS: Source {} +We'll look at the `Visitor` trait shortly. It's like `serde::Serializer`. - #[cfg(feature = "std")] - impl Source for Vec - where - KVS: Source {} +`Visit` is the trait bound that structured values need to satisfy before they can be logged. The trait can't be implemented outside of the `log` crate, because it uses blanket implementations depending on Cargo features. If a crate defines a datastructure that users might want to log, instead of trying to implement `Visit`, it should implement the `serde::Serialize` and `std::fmt::Debug` traits. This means that `Visit` can piggyback off `serde::Serialize` as the pervasive public dependency, so that `Visit` itself doesn't need to be one. - #[cfg(feature = "std")] - impl Source for BTreeMap - where - K: Borrow + Ord, - V: kv::value::Visit {} +The trait bounds on `private::Sealed` ensure that any generic `T: Visit` carries some additional traits that are needed for the blanket implementation of `Serialize`. As an example, any `Option` can also be treated as `Option` and therefore implement `Serialize` itself. The `Visit` trait is responsible for a lot of type system mischief. - #[cfg(feature = "std")] - impl Source for HashMap - where - K: Borrow + Eq + Hash, - V: kv::value::Visit {} +With default features, the types that implement `Visit` are a subset of `T: Debug + Serialize`: - /// The key in a key-value pair. - pub struct Key<'kvs> {} +``` +-------- feature = "kv_serde" -------- +| | +| T: Debug + Serialize | +| | +| | +| - not(feature = "kv_serde") - | +| | | | +| | u8, i8, &str, &[u8], bool | | +| | etc... | | +| | | | +| ----------------------------- | +| | +| | +-------------------------------------- +``` - /// The value in a key-value pair. - pub use kv::value::Value; - } +Enabling the `kv_serde` feature expands the set of types that implement `Visit` from this subset to all `T: Debug + Serialize`. - pub mod value { - pub use kv::Error; +#### Object safety - /// An arbitrary structured value. - pub struct Value<'v> { - /// Create a new borrowed value. - pub fn new(v: &'v impl Visit) -> Self {} +The `Visit` trait is not object-safe, but has a simple object-safe wrapper used by `Value`. - /// Create a new borrowed value from an arbitrary type. - pub fn any(&'v T, fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self {} +#### Without `serde` - /// Visit the value with the given serializer. - pub fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> {} - } +Without the `kv_serde` feature, the `Visit` trait is implemented for a fixed set of fundamental types from the standard library: - impl<'v> Debug for Value<'v> {} - impl<'v> Display for Value<'v> {} +```rust +impl Visit for u8 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u64(*self as u64) + } +} - /// A serializer for primitive values. - pub trait Visitor { - /// Visit an arbitrary value. - fn visit_any(&mut self, v: Value) -> Result<(), Error>; +impl Visit for u16 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u64(*self as u64) + } +} - /// Visit a signed integer. - fn visit_i64(&mut self, v: i64) -> Result<(), Error> {} +impl Visit for u32 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u64(*self as u64) + } +} - /// Visit an unsigned integer. - fn visit_u64(&mut self, v: u64) -> Result<(), Error> {} +impl Visit for u64 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u64(*self) + } +} - /// Visit a floating point number. - fn visit_f64(&mut self, v: f64) -> Result<(), Error> {} +impl Visit for i8 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_i64(*self as i64) + } +} - /// Visit a boolean. - fn visit_bool(&mut self, v: bool) -> Result<(), Error> {} +impl Visit for i16 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_i64(*self as i64) + } +} - /// Visit a single character. - fn visit_char(&mut self, v: char) -> Result<(), Error> {} +impl Visit for i32 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_i64(*self as i64) + } +} - /// Visit a UTF8 string. - fn visit_str(&mut self, v: &str) -> Result<(), Error> {} +impl Visit for i64 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_i64(*self) + } +} - /// Visit a raw byte buffer. - fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> {} +impl Visit for f32 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_f64(*self as f64) + } +} - /// Visit an empty value. - fn visit_none(&mut self) -> Result<(), Error> {} +impl Visit for f64 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_f64(*self) + } +} - /// Visit standard arguments. - fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error> {} - } +impl Visit for char { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_char(*self) + } +} - impl<'a, T: ?Sized> Visitor for &'a mut T - where - T: Visitor {} +impl Visit for bool { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_bool(*self) + } +} - /// Covnert a type into a value. - /// - /// ** This trait can't be implemented manually ** - pub trait Visit: private::Sealed { - /// Visit this value. - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; +impl Visit for () { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_none() + } +} - /// Convert a reference to this value into an erased `Value`. - fn to_value(&self) -> Value - where - Self: Sized, - { - Value::new(self) - } - } +#[cfg(feature = "i128")] +impl Visit for u128 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u128(*self) + } +} - #[cfg(not(feature = "kv_serde"))] - impl Visit for u8 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for u16 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for u32 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for u64 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for u128 {} +#[cfg(feature = "i128")] +impl Visit for i128 { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_i128(*self) + } +} - #[cfg(not(feature = "kv_serde"))] - impl Visit for i8 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for i16 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for i32 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for i64 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for i128 {} +impl Visit for Option +where + T: Visit, +{ + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + match self { + Some(v) => v.visit(visitor), + None => visitor.visit_none(), + } + } +} - #[cfg(not(feature = "kv_serde"))] - impl Visit for f32 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for f64 {} +impl<'a> Visit for fmt::Arguments<'a> { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_fmt(self) + } +} - #[cfg(not(feature = "kv_serde"))] - impl Visit for char {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for bool {} +impl<'a> Visit for &'a str { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_str(self) + } +} - #[cfg(not(feature = "kv_serde"))] - impl Visit for Option - where - T: Visit {} +impl<'a> Visit for &'a [u8] { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_bytes(self) + } +} - #[cfg(all(not(feature = "kv_serde"), feature = "std"))] - impl Visit for Box - where - T: Visit {} +impl<'v> Visit for Value<'v> { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + self.visit(visitor) + } +} - #[cfg(not(feature = "kv_serde"))] - impl<'a> Visit for &'a str {} - #[cfg(all(not(feature = "kv_serde"), feature = "std"))] - impl Visit for String {} +#[cfg(feature = "std")] +impl Visit for Box +where + T: Visit +{ + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + (**self).visit(visitor) + } +} - #[cfg(not(feature = "kv_serde"))] - impl<'a> Visit for &'a [u8] {} - #[cfg(all(not(feature = "kv_serde"), feature = "std"))] - impl Visit for Vec {} +#[cfg(feature = "std")] +impl<'a> Visit for &'a Path { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + match self.to_str() { + Some(s) => visitor.visit_str(s), + None => visitor.visit_fmt(&format_args!("{:?}", self)), + } + } +} - #[cfg(not(feature = "kv_serde"))] - impl<'a, T> Visit for &'a T - where - T: Visit {} +#[cfg(feature = "std")] +impl Visit for String { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_str(&*self) + } +} - #[cfg(feature = "kv_serde")] - impl Visit for T - where - T: Debug + Serialize {} +#[cfg(feature = "std")] +impl Visit for Vec { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_bytes(&*self) } +} - pub use source::Source; +#[cfg(feature = "std")] +impl Visit for PathBuf { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + self.as_path().visit(visitor) + } +} +``` - /// An error encountered while visiting key-value pairs. - pub struct Error {} +#### With `serde` - impl Error { - /// Create an error from a static message. - pub fn msg(msg: &'static str) -> Self {} +With the `kv_serde` feature, the `Visit` trait is implemented for any type that is `Debug + Serialize`: - /// Get a reference to a standard error. - #[cfg(feature = "std")] - pub fn as_error(&self) -> &(dyn std::error::Error + Send + Sync + 'static) {} +```rust +#[cfg(feature = "kv_serde")] +impl Visit for T +where + T: Debug + Serialize {} +``` - /// Convert into a standard error. - #[cfg(feature = "std")] - pub fn into_error(self) -> Box {} +#### Ensuring the fixed set is a subset of the blanket implementation - /// Convert into a `serde` error. - #[cfg(feature = "kv_serde")] - pub fn into_serde(self) -> E - where - E: serde::ser::Error {} - } +Changing trait implementations based on Cargo features is a dangerous game. Cargo features are additive, so any observable changes to trait implementations must also be purely additive, otherwise you can end up with libraries that can't compile if a feature is active. This can be very subtle when references and generics are involved. - #[cfg(not(feature = "std"))] - impl From for Error {} +When the `kv_serde` feature is active, the implementaiton of `Visit` changes from a fixed set to an open one. We have to guarantee that the open set is a superset of the fixed one. That means any valid `T: Visit` without the `kv_serde` feature remains a valid `T: Visit` with the `kv_serde` feature. - #[cfg(feature = "std")] - impl From for Error - where - E: std::error::Error {} +There are a few ways we could achieve this, depending on the quality of the docs we want to produce. - #[cfg(feature = "std")] - impl From for Box {} +For more readable documentation at the risk of incorrectly implementing `Visit`, we can use a private trait like `EnsureVisit: Visit` that is implemented alongside the concrete `Visit` trait regardless of any blanket implementations of `Visit`: - #[cfg(feature = "std")] - impl AsRef for Error {} -} -``` +```rust +// The blanket implemention of `Visit` when `kv_serde` is enabled +#[cfg(feature = "kv_serde")] +impl Visit for T where T: Debug + Serialize {} -### `Record` and `RecordBuilder` +/// This trait is a private implementation detail for testing. +/// +/// All it does is make sure that our set of concrete types +/// that implement `Visit` always implement the `Visit` trait, +/// regardless of crate features and blanket implementations. +trait EnsureVisit: Visit {} -Structured key-value pairs can be set on a `RecordBuilder`: +// Ensure any reference to a `Visit` implements `Visit` +impl<'a, T> EnsureVisit for &'a T where T: Visit {} + +// These impl blocks always exists +impl EnsureVisit for Option where T: Visit {} +// This impl block only exists if the `kv_serde` isn't active +#[cfg(not(feature = "kv_serde"))] +impl private::Sealed for Option where T: Visit {} +#[cfg(not(feature = "kv_serde"))] +impl Visit for Option where T: Visit { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { -```rust -impl<'a> RecordBuilder<'a> { - /// Set key values - pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a> { - self.record.kvs = kvs; - self } } ``` -These key-value pairs can then be accessed on the built `Record`: +In the above example, we can ensure that `Option` always implements the `Visit` trait, whether it's done manually or as part of a blanket implementation. All types that implement `Visit` manually with any `#[cfg]` _must_ also always implement `EnsureVisit` manually (with no `#[cfg]`) with the exact same type bounds. It's pretty subtle, but the subtlety can be localized to a single module within the `log` crate so it can be managed. + +Using a trait for this type checking means the `impl Visit for Option` and `impl EnsureVisit for Option` can be wrapped up in a macro so that we never miss adding them. The below macro is an example of a (not very pretty) one that can add the needed implementations of `EnsureVisit` along with the regular `Visit`: ```rust -#[derive(Clone, Debug)] -pub struct Record<'a> { - ... +macro_rules! impl_to_value { + () => {}; + ( + impl: { $($params:tt)* } + where: { $($where:tt)* } + $ty:ty: { $($serialize:tt)* } + $($rest:tt)* + ) => { + impl<$($params)*> EnsureVisit for $ty + where + $($where)* {} + + #[cfg(not(feature = "kv_serde"))] + impl<$($params)*> private::Sealed for $ty + where + $($where)* {} - kvs: ErasedSource<'a>, + #[cfg(not(feature = "kv_serde"))] + impl<$($params)*> Visit for $ty + where + $($where)* + { + $($serialize)* + } + + impl_to_value!($($rest)*); + }; + ( + impl: { $($params:tt)* } + $ty:ty: { $($serialize:tt)* } + $($rest:tt)* + ) => { + impl_to_value! { + impl: {$($params)*} where: {} $ty: { $($serialize)* } $($rest)* + } + }; + ( + $ty:ty: { $($serialize:tt)* } + $($rest:tt)* + ) => { + impl_to_value! { + impl: {} where: {} $ty: { $($serialize)* } $($rest)* + } + } } -impl<'a> Record<'a> { - /// The key value pairs attached to this record. - /// - /// Pairs aren't guaranteed to be unique (the same key may be repeated with different values). - pub fn key_values(&self) -> ErasedSource { - self.kvs.clone() +// Ensure any reference to a `Visit` is also `Visit` +impl<'a, T> EnsureVisit for &'a T where T: Visit {} + +impl_to_value! { + u8: { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + visitor.visit_u64(*self as u64) + } + } + + impl: { T: Visit } Option: { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + match self { + Some(v) => v.to_value().visit(visitor), + None => visitor.visit_none(), + } + } } + + ... } ``` -### `Error` +We don't necessarily need a macro to make new implementations accessible for new contributors safely though. -Just about the only things you can do with a structured value are format it or serialize it. Serialization and writing might fail, so to allow errors to get carried back to callers there needs to be a general error type that they can early return with: +##### What about specialization? -```rust -pub struct Error(Inner); +In a future Rust with specialization we might be able to avoid all the machinery needed to keep the manual impls consistent with the blanket one, and allow consumers to implement `Visit` without needing `serde`. The specifics of specialization are still up in the air though. Under the proposed _always applicable_ rule, manual implementations like `impl Visit for Option where T: Visit` wouldn't be allowed. The ` where specialize(T: Visit)` scheme might make it possible though, although this would probably be a breaking change in any case. -enum Inner { - Static(&'static str), - #[cfg(feature = "std")] - Owned(String), -} +### `Value` -impl Error { - pub fn msg(msg: &'static str) -> Self { - Error(Inner::Static(msg)) - } +A `Value` is an erased container for a `Visit`, with a potentially short-lived lifetime: - #[cfg(feature = "std")] - pub fn as_error(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { - &self.0 - } +```rust +/// The value in a key-value pair. +pub struct Value<'v>(ValueInner<'v>); - #[cfg(feature = "std")] - pub fn into_error(self) -> Box { - Box::new(self.0) +enum ValueInner<'v> { + Erased(&'v dyn ErasedVisit), + Any(Any<'v>), +} + +impl<'v> Value<'v> { + /// Create a value. + pub fn new(v: &'v impl Visit) -> Self { + Value(ValueInner::Erased(v)) } - #[cfg(feature = "kv_serde")] - pub fn into_serde(self) -> E + /// Create a value from an anonymous type. + /// + /// The value must be provided with a compatible visit method. + pub fn any(v: &'v T, visit: fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self where - E: serde::ser::Error, + T: 'static, { - E::custom(self) + Value(ValueInner::Any(Any::new(v, visit))) } -} -#[cfg(feature = "std")] -impl From for Error -where - E: std::error::Error, -{ - fn from(err: E) -> Self { - Error(Inner::Owned(err.to_string())) + /// Visit the contents of this value with a visitor. + pub fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + match self.0 { + ValueInner::Erased(v) => v.erased_visit(visitor), + ValueInner::Any(ref v) => v.visit(visitor), + } } } +``` -#[cfg(feature = "std")] -impl From for Box { - fn from(err: Error) -> Self { - err.into_error() - } +#### `ErasedVisit` + +The `ErasedVisit` trait is an object-safe wrapper for the `Visit` trait. `Visit` itself isn't technically object-safe because it needs the non-object-safe `serde::Serialize` as a supertrait to carry in generic contexts: + +```rust +#[cfg(not(feature = "kv_serde"))] +trait ErasedVisit: fmt::Debug { + fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; } -impl AsRef for Error { - fn as_ref(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { - self.as_error() - } +#[cfg(feature = "kv_serde")] +trait ErasedVisit: fmt::Debug + erased_serde::Serialize { + fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; } -#[cfg(feature = "std")] -impl std::error::Error for Inner { - fn description(&self) -> &str { - match self { - Inner::Static(msg) => msg, - Inner::Owned(msg) => msg, - } +impl ErasedVisit for T +where + T: Visit, +{ + fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + self.visit(visitor) } } ``` -There's no really universal way to handle errors in a logging pipeline. Knowing that some error occurred, and knowing where, should be enough for implementations of `Log` to decide how to handle it. The `Error` type doesn't try to be a general-purpose error management tool, it tries to make it easy to early-return with other errors. - -To make it possible to carry any arbitrary `S::Error` type, where we don't know how long the value can live for and whether it's `Send` or `Sync`, without extra work, the `Error` type does not attempt to store the error value itself. It just converts it into a `String`. - -### `value::Visit` +#### `Any` -The `Visit` trait can be treated like a lightweight subset of `serde::Serialize` that can interoperate with `serde` without necessarily depending on it: +Other logging frameworks that want to integrate with `log` might not want to pull in a `serde` dependency, and so they couldn't implement the `Visit` trait. The `Any` type uses some `std::fmt` inspired black-magic to allow values that don't implement the `Visit` trait to be erased in a `Value`. It does this by taking a borrowed value along with a function pointer that looks like `Visit::visit`: ```rust -/// A type that can be converted into a borrowed value. -pub trait Visit: private::Sealed { - /// Visit this value. - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; +struct Void { + _priv: (), + _oibit_remover: PhantomData<*mut dyn Fn()>, +} - /// Convert a reference to this value into an erased `Value`. - fn to_value(&self) -> Value +struct Any<'a> { + data: &'a Void, + visit: fn(&Void, &mut dyn Visitor) -> Result<(), Error>, +} + +impl<'a> Any<'a> { + fn new(data: &'a T, visit: fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self where - Self: Sized, + T: 'static, { - Value::new(self) + unsafe { + Any { + data: mem::transmute::<&'a T, &'a Void>(data), + visit: mem::transmute::< + fn(&T, &mut dyn Visitor) -> Result<(), Error>, + fn(&Void, &mut dyn Visitor) -> Result<(), Error>> + (visit), + } + } } -} - -mod private { - #[cfg(not(feature = "kv_serde"))] - pub trait Sealed: Debug {} - #[cfg(feature = "kv_serde")] - pub trait Sealed: Debug + Serialize {} + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + (self.visit)(self.data, visitor) + } } ``` -We'll look at the `Visitor` trait shortly. It's like `serde::Serializer`. +There's some scary code in `Any`, which is really just something like an ad-hoc trait object. -`Visit` is the trait bound that structured values need to satisfy before they can be logged. The trait can't be implemented outside of the `log` crate, because it uses blanket implementations depending on Cargo features. If a crate defines a datastructure that users might want to log, instead of trying to implement `Visit`, it should implement the `serde::Serialize` and `std::fmt::Debug` traits. This means that `Visit` can piggyback off `serde::Serialize` as the pervasive public dependency, so that `Visit` itself doesn't need to be one. +#### Formatting -The trait bounds on `private::Sealed` ensure that any generic `T: Visit` carries some additional traits that are needed for the blanket implementation of `Serialize`. As an example, any `Option` can also be treated as `Option` and therefore implement `Serialize` itself. The `Visit` trait is responsible for a lot of type system mischief. +`Value` always implements `Debug` and `Display` by forwarding to its inner value: -With default features, the types that implement `Visit` are a subset of `T: Debug + Serialize`: +```rust +impl<'v> fmt::Debug for Value<'v> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self.0 { + ValueInner::Erased(v) => v.fmt(f), + ValueInner::Any(ref v) => { + struct ValueFmt<'a, 'b>(&'a mut fmt::Formatter<'b>); -``` --------- feature = "kv_serde" -------- -| | -| T: Debug + Serialize | -| | -| | -| - not(feature = "kv_serde") - | -| | | | -| | u8, i8, &str, &[u8], bool | | -| | etc... | | -| | | | -| ----------------------------- | -| | -| | --------------------------------------- -``` + impl<'a, 'b> Visitor for ValueFmt<'a, 'b> { + fn visit_any(&mut self, v: Value) -> Result<(), Error> { + write!(self.0, "{:?}", v)?; -Enabling the `kv_serde` feature expands the set of types that implement `Visit` from this subset to all `T: Debug + Serialize`. + Ok(()) + } + } -#### Object safety + let mut visitor = ValueFmt(f); + v.visit(&mut visitor).map_err(|_| fmt::Error) + } + } + } +} -The `Visit` trait is not object-safe, but has a simple object-safe wrapper used by `Value`. +impl<'v> fmt::Display for Value<'v> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{:?}", self) + } +} +``` -#### Without `serde` +#### Serialization -Without the `kv_serde` feature, the `Visit` trait is implemented for a fixed set of fundamental types from the standard library: +When the `kv_serde` feature is enabled, `Value` implements the `Serialize` trait by forwarding to its inner value: ```rust -impl Visit for u8 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u64(*self as u64) - } -} +impl<'v> Serialize for Value<'v> { + fn serialize(&self, serializer: S) -> Result + where + S: Serializer, + { + match self.0 { + ValueInner::Erased(v) => { + erased_serde::serialize(v, serializer) + }, + ValueInner::Any(ref v) => { + struct ErasedVisitSerde { + serializer: Option, + ok: Option, + } -impl Visit for u16 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u64(*self as u64) - } -} + impl Visitor for ErasedVisitSerde + where + S: Serializer, + { + fn visit_any(&mut self, v: Value) -> Result<(), Error> { + let ok = v.serialize(self.serializer.take().expect("missing serializer"))?; + self.ok = Some(ok); -impl Visit for u32 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u64(*self as u64) - } -} + Ok(()) + } + } -impl Visit for u64 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u64(*self) - } -} + let mut visitor = ErasedVisitSerde { + serializer: Some(serializer), + ok: None, + }; -impl Visit for i8 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_i64(*self as i64) + v.visit(&mut visitor).map_err(|e| e.into_serde())?; + Ok(visitor.ok.expect("missing return value")) + }, + } } } +``` -impl Visit for i16 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_i64(*self as i64) - } -} +#### Ownership -impl Visit for i32 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_i64(*self as i64) - } -} +The `Value` type borrows from its inner value. -impl Visit for i64 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_i64(*self) - } -} +#### Thread-safety -impl Visit for f32 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_f64(*self as f64) - } -} +The `Value` type doesn't try to guarantee that values are `Send` or `Sync`, and doesn't offer any way of retaining that information when erasing. -impl Visit for f64 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_f64(*self) - } -} +### `value::Visitor` -impl Visit for char { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_char(*self) +A visitor for a `Value` that can interogate its structure: + +```rust +/// A serializer for primitive values. +pub trait Visitor { + /// Visit an arbitrary value. + /// + /// Depending on crate features there are a few things + /// you can do with a value. You can: + /// + /// - format it using `Debug`. + /// - serialize it using `serde`. + fn visit_any(&mut self, v: Value) -> Result<(), Error>; + + /// Visit a signed integer. + fn visit_i64(&mut self, v: i64) -> Result<(), Error> { + self.visit_any(v.to_value()) } -} -impl Visit for bool { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_bool(*self) + /// Visit an unsigned integer. + fn visit_u64(&mut self, v: u64) -> Result<(), Error> { + self.visit_any(v.to_value()) } -} -impl Visit for () { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_none() + /// Visit a 128bit signed integer. + #[cfg(feature = "i128")] + fn visit_i128(&mut self, v: i128) -> Result<(), Error> { + self.visit_any(v.to_value()) } -} -#[cfg(feature = "i128")] -impl Visit for u128 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u128(*self) + /// Visit a 128bit unsigned integer. + #[cfg(feature = "i128")] + fn visit_u128(&mut self, v: u128) -> Result<(), Error> { + self.visit_any(v.to_value()) } -} -#[cfg(feature = "i128")] -impl Visit for i128 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_i128(*self) + /// Visit a floating point number. + fn visit_f64(&mut self, v: f64) -> Result<(), Error> { + self.visit_any(v.to_value()) } -} -impl Visit for Option -where - T: Visit, -{ - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - match self { - Some(v) => v.visit(visitor), - None => visitor.visit_none(), - } + /// Visit a boolean. + fn visit_bool(&mut self, v: bool) -> Result<(), Error> { + self.visit_any(v.to_value()) } -} -impl<'a> Visit for fmt::Arguments<'a> { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_fmt(self) + /// Visit a single character. + fn visit_char(&mut self, v: char) -> Result<(), Error> { + let mut b = [0; 4]; + self.visit_str(&*v.encode_utf8(&mut b)) } -} -impl<'a> Visit for &'a str { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_str(self) + /// Visit a UTF8 string. + fn visit_str(&mut self, v: &str) -> Result<(), Error> { + self.visit_any((&v).to_value()) } -} -impl<'a> Visit for &'a [u8] { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_bytes(self) + /// Visit a raw byte buffer. + fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { + self.visit_any((&v).to_value()) } -} -impl<'v> Visit for Value<'v> { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - self.visit(visitor) + /// Visit standard arguments. + fn visit_none(&mut self) -> Result<(), Error> { + self.visit_any(().to_value()) + } + + /// Visit standard arguments. + fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error> { + self.visit_any(v.to_value()) } } -#[cfg(feature = "std")] -impl Visit for Box +impl<'a, T: ?Sized> Visitor for &'a mut T where - T: Visit + T: Visitor, { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - (**self).visit(visitor) + fn visit_any(&mut self, v: Value) -> Result<(), Error> { + (**self).visit_any(v) } -} -#[cfg(feature = "std")] -impl<'a> Visit for &'a Path { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - match self.to_str() { - Some(s) => visitor.visit_str(s), - None => visitor.visit_fmt(&format_args!("{:?}", self)), - } + fn visit_i64(&mut self, v: i64) -> Result<(), Error> { + (**self).visit_i64(v) } -} -#[cfg(feature = "std")] -impl Visit for String { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_str(&*self) + fn visit_u64(&mut self, v: u64) -> Result<(), Error> { + (**self).visit_u64(v) } -} -#[cfg(feature = "std")] -impl Visit for Vec { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_bytes(&*self) + #[cfg(feature = "i128")] + fn visit_i128(&mut self, v: i128) -> Result<(), Error> { + (**self).visit_i128(v) } -} -#[cfg(feature = "std")] -impl Visit for PathBuf { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - self.as_path().visit(visitor) + #[cfg(feature = "i128")] + fn visit_u128(&mut self, v: u128) -> Result<(), Error> { + (**self).visit_u128(v) } -} -``` - -#### With `serde` - -With the `kv_serde` feature, the `Visit` trait is implemented for any type that is `Debug + Serialize`: - -```rust -#[cfg(feature = "kv_serde")] -impl Visit for T -where - T: Debug + Serialize {} -``` - -#### Ensuring the fixed set is a subset of the blanket implementation - -Changing trait implementations based on Cargo features is a dangerous game. Cargo features are additive, so any observable changes to trait implementations must also be purely additive, otherwise you can end up with libraries that can't compile if a feature is active. This can be very subtle when references and generics are involved. -When the `kv_serde` feature is active, the implementaiton of `Visit` changes from a fixed set to an open one. We have to guarantee that the open set is a superset of the fixed one. That means any valid `T: Visit` without the `kv_serde` feature remains a valid `T: Visit` with the `kv_serde` feature. - -There are a few ways we could achieve this, depending on the quality of the docs we want to produce. + fn visit_f64(&mut self, v: f64) -> Result<(), Error> { + (**self).visit_f64(v) + } -For more readable documentation at the risk of incorrectly implementing `Visit`, we can use a private trait like `EnsureVisit: Visit` that is implemented alongside the concrete `Visit` trait regardless of any blanket implementations of `Visit`: + fn visit_bool(&mut self, v: bool) -> Result<(), Error> { + (**self).visit_bool(v) + } -```rust -// The blanket implemention of `Visit` when `kv_serde` is enabled -#[cfg(feature = "kv_serde")] -impl Visit for T where T: Debug + Serialize {} + fn visit_char(&mut self, v: char) -> Result<(), Error> { + (**self).visit_char(v) + } -/// This trait is a private implementation detail for testing. -/// -/// All it does is make sure that our set of concrete types -/// that implement `Visit` always implement the `Visit` trait, -/// regardless of crate features and blanket implementations. -trait EnsureVisit: Visit {} + fn visit_str(&mut self, v: &str) -> Result<(), Error> { + (**self).visit_str(v) + } -// Ensure any reference to a `Visit` implements `Visit` -impl<'a, T> EnsureVisit for &'a T where T: Visit {} + fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { + (**self).visit_bytes(v) + } -// These impl blocks always exists -impl EnsureVisit for Option where T: Visit {} -// This impl block only exists if the `kv_serde` isn't active -#[cfg(not(feature = "kv_serde"))] -impl private::Sealed for Option where T: Visit {} -#[cfg(not(feature = "kv_serde"))] -impl Visit for Option where T: Visit { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + fn visit_none(&mut self) -> Result<(), Error> { + (**self).visit_none() + } + fn visit_fmt(&mut self, args: &fmt::Arguments) -> Result<(), Error> { + (**self).visit_fmt(args) } } ``` -In the above example, we can ensure that `Option` always implements the `Visit` trait, whether it's done manually or as part of a blanket implementation. All types that implement `Visit` manually with any `#[cfg]` _must_ also always implement `EnsureVisit` manually (with no `#[cfg]`) with the exact same type bounds. It's pretty subtle, but the subtlety can be localized to a single module within the `log` crate so it can be managed. +### `Key` -Using a trait for this type checking means the `impl Visit for Option` and `impl EnsureVisit for Option` can be wrapped up in a macro so that we never miss adding them. The below macro is an example of a (not very pretty) one that can add the needed implementations of `EnsureVisit` along with the regular `Visit`: +A `Key` is a short-lived structure that can be represented as a UTF-8 string. This might be possible without allocating, or it might require a destination to write into: ```rust -macro_rules! impl_to_value { - () => {}; - ( - impl: { $($params:tt)* } - where: { $($where:tt)* } - $ty:ty: { $($serialize:tt)* } - $($rest:tt)* - ) => { - impl<$($params)*> EnsureVisit for $ty - where - $($where)* {} - - #[cfg(not(feature = "kv_serde"))] - impl<$($params)*> private::Sealed for $ty - where - $($where)* {} - - #[cfg(not(feature = "kv_serde"))] - impl<$($params)*> Visit for $ty - where - $($where)* - { - $($serialize)* - } +/// A key in a key-value pair. +/// +/// The key can be treated like `&str`. +#[derive(PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Key<'kvs> { + inner: &'kvs str, +} - impl_to_value!($($rest)*); - }; - ( - impl: { $($params:tt)* } - $ty:ty: { $($serialize:tt)* } - $($rest:tt)* - ) => { - impl_to_value! { - impl: {$($params)*} where: {} $ty: { $($serialize)* } $($rest)* - } - }; - ( - $ty:ty: { $($serialize:tt)* } - $($rest:tt)* - ) => { - impl_to_value! { - impl: {} where: {} $ty: { $($serialize)* } $($rest)* - } +impl<'kvs> Borrow for Key<'kvs> { + fn to_key(&self) -> Key { + Key { inner: self.inner } } } -// Ensure any reference to a `Visit` is also `Visit` -impl<'a, T> EnsureVisit for &'a T where T: Visit {} - -impl_to_value! { - u8: { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u64(*self as u64) +impl<'kvs> Key<'kvs> { + /// Get a `Key` from a borrowed string. + pub fn from_str(key: &'kvs (impl AsRef + ?Sized)) -> Self { + Key { + inner: key.as_ref(), } } - impl: { T: Visit } Option: { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - match self { - Some(v) => v.to_value().visit(visitor), - None => visitor.visit_none(), - } - } + /// Get a borrowed string from a `Key`. + pub fn as_str(&self) -> &str { + &self.inner } - - ... } -``` - -We don't necessarily need a macro to make new implementations accessible for new contributors safely though. - -##### What about specialization? - -In a future Rust with specialization we might be able to avoid all the machinery needed to keep the manual impls consistent with the blanket one, and allow consumers to implement `Visit` without needing `serde`. The specifics of specialization are still up in the air though. Under the proposed _always applicable_ rule, manual implementations like `impl Visit for Option where T: Visit` wouldn't be allowed. The ` where specialize(T: Visit)` scheme might make it possible though, although this would probably be a breaking change in any case. - -### `Value` - -A `Value` is an erased container for a `Visit`, with a potentially short-lived lifetime: - -```rust -/// The value in a key-value pair. -pub struct Value<'v>(ValueInner<'v>); -enum ValueInner<'v> { - Erased(&'v dyn ErasedVisit), - Any(Any<'v>), +impl<'kvs> AsRef for Key<'kvs> { + fn as_ref(&self) -> &str { + self.as_str() + } } -impl<'v> Value<'v> { - /// Create a value. - pub fn new(v: &'v impl Visit) -> Self { - Value(ValueInner::Erased(v)) +#[cfg(feature = "std")] +impl<'kvs> Borrow for Key<'kvs> { + fn borrow(&self) -> &str { + self.as_str() } +} - /// Create a value from an anonymous type. - /// - /// The value must be provided with a compatible visit method. - pub fn any(v: &'v T, visit: fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self +impl<'kvs> Serialize for Key<'kvs> { + fn serialize(&self, serializer: S) -> Result where - T: 'static, + S: Serializer, { - Value(ValueInner::Any(Any::new(v, visit))) + serializer.serialize_str(self.inner) } +} - /// Visit the contents of this value with a visitor. - pub fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - match self.0 { - ValueInner::Erased(v) => v.erased_visit(visitor), - ValueInner::Any(ref v) => v.visit(visitor), - } +impl<'kvs> Display for Key<'kvs> { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + self.inner.fmt(f) + } +} + +impl<'kvs> Debug for Key<'kvs> { + fn fmt(&self, f: &mut Formatter) -> fmt::Result { + self.inner.fmt(f) } } ``` -#### `ErasedVisit` +Other standard implementations could be added for any `K: Borrow` in the same fashion. -The `ErasedVisit` trait is an object-safe wrapper for the `Visit` trait. `Visit` itself isn't technically object-safe because it needs the non-object-safe `serde::Serialize` as a supertrait to carry in generic contexts: +#### Ownership -```rust -#[cfg(not(feature = "kv_serde"))] -trait ErasedVisit: fmt::Debug { - fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; -} +The `Key` type can either borrow or own its inner value. -#[cfg(feature = "kv_serde")] -trait ErasedVisit: fmt::Debug + erased_serde::Serialize { - fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; +#### Thread-safety + +The `Key` type is probably `Send` + `Sync`, but that's not guaranteed. + +### `source::Visitor` + +The `Visitor` trait used by `Source` can visit a single key-value pair: + +```rust +pub trait Visitor<'kvs> { + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; } -impl ErasedVisit for T +impl<'a, 'kvs, T: ?Sized> Visitor<'kvs> for &'a mut T where - T: Visit, + T: Visitor<'kvs>, { - fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - self.visit(visitor) + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { + (*self).visit_pair(k, v) } } ``` -#### `Any` - -Other logging frameworks that want to integrate with `log` might not want to pull in a `serde` dependency, and so they couldn't implement the `Visit` trait. The `Any` type uses some `std::fmt` inspired black-magic to allow values that don't implement the `Visit` trait to be erased in a `Value`. It does this by taking a borrowed value along with a function pointer that looks like `Visit::visit`: +A `Visitor` may serialize the keys and values as it sees them. It may also do other work, like sorting or de-duplicating them. Operations that involve ordering keys will probably require allocations. -```rust -struct Void { - _priv: (), - _oibit_remover: PhantomData<*mut dyn Fn()>, -} +#### Implementors -struct Any<'a> { - data: &'a Void, - visit: fn(&Void, &mut dyn Visitor) -> Result<(), Error>, -} +There aren't any public implementors of `Visitor` in the `log` crate, but the `Source::try_for_each` and `Source::serialize_as_map` methods use the trait internally. -impl<'a> Any<'a> { - fn new(data: &'a T, visit: fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self - where - T: 'static, - { - unsafe { - Any { - data: mem::transmute::<&'a T, &'a Void>(data), - visit: mem::transmute::< - fn(&T, &mut dyn Visitor) -> Result<(), Error>, - fn(&Void, &mut dyn Visitor) -> Result<(), Error>> - (visit), - } - } - } +Other crates that use key-value pairs will implement `Visitor`. - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - (self.visit)(self.data, visitor) - } -} -``` +#### Object safety -There's some scary code in `Any`, which is really just something like an ad-hoc trait object. +The `Visitor` trait is object-safe. -#### Formatting +### `Source` -`Value` always implements `Debug` and `Display` by forwarding to its inner value: +The `Source` trait is a bit like `Serialize`. It gives us a way to inspect some arbitrary collection of key-value pairs using a visitor pattern: ```rust -impl<'v> fmt::Debug for Value<'v> { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self.0 { - ValueInner::Erased(v) => v.fmt(f), - ValueInner::Any(ref v) => { - struct ValueFmt<'a, 'b>(&'a mut fmt::Formatter<'b>); - - impl<'a, 'b> Visitor for ValueFmt<'a, 'b> { - fn visit_any(&mut self, v: Value) -> Result<(), Error> { - write!(self.0, "{:?}", v)?; - - Ok(()) - } - } +pub trait Source { + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error>; - let mut visitor = ValueFmt(f); - v.visit(&mut visitor).map_err(|_| fmt::Error) - } - } - } + ... } -impl<'v> fmt::Display for Value<'v> { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "{:?}", self) +impl<'a, T: ?Sized> Source for &'a T +where + T: Source, +{ + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + (*self).visit(visitor) } } ``` -#### Serialization - -When the `kv_serde` feature is enabled, `Value` implements the `Serialize` trait by forwarding to its inner value: - -```rust -impl<'v> Serialize for Value<'v> { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - match self.0 { - ValueInner::Erased(v) => { - erased_serde::serialize(v, serializer) - }, - ValueInner::Any(ref v) => { - struct ErasedVisitSerde { - serializer: Option, - ok: Option, - } +`Source` doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. - impl Visitor for ErasedVisitSerde - where - S: Serializer, - { - fn visit_any(&mut self, v: Value) -> Result<(), Error> { - let ok = v.serialize(self.serializer.take().expect("missing serializer"))?; - self.ok = Some(ok); +#### Ownership - Ok(()) - } - } +The `Source` trait is probably the point where having some way to convert from a borrowed to an owned variant would make the most sense. - let mut visitor = ErasedVisitSerde { - serializer: Some(serializer), - ok: None, - }; +We could add a method to `Source` that allowed it to be converted into an owned variant with a default implementation: - v.visit(&mut visitor).map_err(|e| e.into_serde())?; - Ok(visitor.ok.expect("missing return value")) - }, - } +```rust +pub trait Source { + fn to_owned(&self) -> OwnedSource { + OwnedSource::serialized(self) } } ``` -#### Ownership +The `OwnedSource` could then encapsulte some sharable `dyn Source + Send + Sync`: -The `Value` type borrows from its inner value. +```rust +#[derive(Clone)] +pub struct OwnedSource(Arc); -#### Thread-safety +impl OwnedSource { + fn new(impl Into>) -> Self { + OwnedSource(source.into()) + } -The `Value` type doesn't try to guarantee that values are `Send` or `Sync`, and doesn't offer any way of retaining that information when erasing. + fn serialize(impl Source) -> Self { + // Serialize the `Source` to something like + // `Vec<(String, OwnedValue)>` + // where `OwnedValue` is like `serde_json::Value` + ... + } +} +``` -### `value::Visitor` +Other implementations of `Source` would be encouraged to override the `to_owned` method if they could provide a more efficient implementation. As an example, if there's a `Source` that is already wrapped up in an `Arc` then it can implement `to_owned` by just cloning itself. -A visitor for a `Value` that can interogate its structure: +#### Adapters + +Some useful adapters exist as provided methods on the `Source` trait. They're similar to adapters on the standard `Iterator` trait: ```rust -/// A serializer for primitive values. -pub trait Visitor { - /// Visit an arbitrary value. - /// - /// Depending on crate features there are a few things - /// you can do with a value. You can: - /// - /// - format it using `Debug`. - /// - serialize it using `serde`. - fn visit_any(&mut self, v: Value) -> Result<(), Error>; +pub trait Source { + ... - /// Visit a signed integer. - fn visit_i64(&mut self, v: i64) -> Result<(), Error> { - self.visit_any(v.to_value()) + /// Erase this `Source` so it can be used without + /// requiring generic type parameters. + fn erase(&self) -> ErasedSource + where + Self: Sized, + { + ErasedSource::erased(self) } - /// Visit an unsigned integer. - fn visit_u64(&mut self, v: u64) -> Result<(), Error> { - self.visit_any(v.to_value()) + /// An adapter to borrow self. + fn by_ref(&self) -> &Self { + self } - /// Visit a 128bit signed integer. - #[cfg(feature = "i128")] - fn visit_i128(&mut self, v: i128) -> Result<(), Error> { - self.visit_any(v.to_value()) + /// Chain two `Source`s together. + fn chain(self, other: KVS) -> Chained + where + Self: Sized, + { + Chained(self, other) } - /// Visit a 128bit unsigned integer. - #[cfg(feature = "i128")] - fn visit_u128(&mut self, v: u128) -> Result<(), Error> { - self.visit_any(v.to_value()) - } + /// Find the value for a given key. + /// + /// If the key is present multiple times, this method will + /// return the *last* value for the given key. + /// + /// The default implementation will scan all key-value pairs. + /// Implementors are encouraged provide a more efficient version + /// if they can. Standard collections like `BTreeMap` and `HashMap` + /// will do an indexed lookup instead of a scan. + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: Borrow, + { + struct Get<'k, 'v>(Key<'k>, Option>); + + impl<'k, 'kvs> Visitor<'kvs> for Get<'k, 'kvs> { + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { + if k == self.0 { + self.1 = Some(v); + } + + Ok(()) + } + } - /// Visit a floating point number. - fn visit_f64(&mut self, v: f64) -> Result<(), Error> { - self.visit_any(v.to_value()) - } + let mut visitor = Get(key.to_key(), None); + let _ = self.visit(&mut visitor); - /// Visit a boolean. - fn visit_bool(&mut self, v: bool) -> Result<(), Error> { - self.visit_any(v.to_value()) + visitor.1 } - /// Visit a single character. - fn visit_char(&mut self, v: char) -> Result<(), Error> { - let mut b = [0; 4]; - self.visit_str(&*v.encode_utf8(&mut b)) - } + /// Apply a function to each key-value pair. + fn try_for_each(self, f: F) -> Result<(), Error> + where + Self: Sized, + F: FnMut(Key, Value) -> Result<(), E>, + E: Into, + { + struct ForEach(F, std::marker::PhantomData); - /// Visit a UTF8 string. - fn visit_str(&mut self, v: &str) -> Result<(), Error> { - self.visit_any((&v).to_value()) - } + impl<'kvs, F, E> Visitor<'kvs> for ForEach + where + F: FnMut(Key, Value) -> Result<(), E>, + E: Into, + { + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { + (self.0)(k, v).map_err(Into::into) + } + } - /// Visit a raw byte buffer. - fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { - self.visit_any((&v).to_value()) + self.visit(&mut ForEach(f, Default::default())) } - /// Visit standard arguments. - fn visit_none(&mut self) -> Result<(), Error> { - self.visit_any(().to_value()) + /// Serialize the key-value pairs as a map. + #[cfg(feature = "kv_serde")] + fn serialize_as_map(self) -> SerializeAsMap + where + Self: Sized, + { + SerializeAsMap(self) } - /// Visit standard arguments. - fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error> { - self.visit_any(v.to_value()) + /// Serialize the key-value pairs as a map. + #[cfg(feature = "kv_serde")] + fn serialize_as_seq(self) -> SerializeAsSeq + where + Self: Sized, + { + SerializeAsSeq(self) } } +``` -impl<'a, T: ?Sized> Visitor for &'a mut T -where - T: Visitor, -{ - fn visit_any(&mut self, v: Value) -> Result<(), Error> { - (**self).visit_any(v) - } +- `by_ref` to get a reference to a `Source` within a method chain. +- `chain` to concatenate one source with another. This is useful for composing implementations of `Log` together for contextual logging. +- `get` to try find the value associated with a key. +- `try_for_each` to try execute some closure over all key-value pairs. This is a convenient way to do something with each key-value pair without having to create and implement a `Visitor`. +- `serialize_as_map` to get a serializable map. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. +- `serialize_as_seq` to get a serializable sequence of tuples. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. - fn visit_i64(&mut self, v: i64) -> Result<(), Error> { - (**self).visit_i64(v) - } +None of these methods are required for the core API. They're helpful tools for working with key-value pairs with minimal machinery. Even if we don't necessarily include them right away it's worth having an API that can support them later without breakage. - fn visit_u64(&mut self, v: u64) -> Result<(), Error> { - (**self).visit_u64(v) - } +#### Object safety - #[cfg(feature = "i128")] - fn visit_i128(&mut self, v: i128) -> Result<(), Error> { - (**self).visit_i128(v) - } +`Source` is not object-safe because of the provided adapter methods not being object-safe. The only required method, `visit`, is safe though, so an object-safe version of `Source` that forwards this method can be reasonably written. - #[cfg(feature = "i128")] - fn visit_u128(&mut self, v: u128) -> Result<(), Error> { - (**self).visit_u128(v) - } +```rust +/// An erased `Source`. +#[derive(Clone)] +pub struct ErasedSource<'a>(&'a dyn ErasedSourceBridge); - fn visit_f64(&mut self, v: f64) -> Result<(), Error> { - (**self).visit_f64(v) +impl<'a> ErasedSource<'a> { + /// Capture a `Source` and erase its concrete type. + pub fn new(kvs: &'a impl Source) -> Self { + ErasedSource(kvs) } +} - fn visit_bool(&mut self, v: bool) -> Result<(), Error> { - (**self).visit_bool(v) +impl<'a> Default for ErasedSource<'a> { + fn default() -> Self { + ErasedSource(&(&[] as &[(&str, &dyn Visit)])) } +} - fn visit_char(&mut self, v: char) -> Result<(), Error> { - (**self).visit_char(v) +impl<'a> Source for ErasedSource<'a> { + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + self.0.erased_visit(visitor) } - fn visit_str(&mut self, v: &str) -> Result<(), Error> { - (**self).visit_str(v) + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: Borrow, + { + let key = key.to_key(); + self.0.erased_get(key.as_ref()) } +} - fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { - (**self).visit_bytes(v) - } +/// A trait that erases a `Source` so it can be stored +/// in a `Record` without requiring any generic parameters. +trait ErasedSourceBridge { + fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; + fn erased_get<'kvs>(&'kvs self, key: &str) -> Option>; +} - fn visit_none(&mut self) -> Result<(), Error> { - (**self).visit_none() +impl ErasedSourceBridge for KVS +where + KVS: Source + ?Sized, +{ + fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + self.visit(visitor) } - fn visit_fmt(&mut self, args: &fmt::Arguments) -> Result<(), Error> { - (**self).visit_fmt(args) + fn erased_get<'kvs>(&'kvs self, key: &str) -> Option> { + self.get(key) } } ``` -### `Key` +#### Implementors -A `Key` is a short-lived structure that can be represented as a UTF-8 string. This might be possible without allocating, or it might require a destination to write into: +A `Source` with a single pair is implemented for a tuple of a key and value: ```rust -/// A key in a key-value pair. -/// -/// The key can be treated like `&str`. -#[derive(PartialEq, Eq, PartialOrd, Ord, Hash)] -pub struct Key<'kvs> { - inner: &'kvs str, +impl Source for (K, V) +where + K: Borrow, + V: Visit, +{ + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + visitor.visit_pair(self.0.to_key(), self.1.to_value()) + } } +``` -impl<'kvs> Borrow for Key<'kvs> { - fn to_key(&self) -> Key { - Key { inner: self.inner } +A `Source` with multiple pairs is implemented for arrays of `Source`s: + +```rust +impl Source for [KVS] where KVS: Source { + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + for kv in self { + kv.visit(&mut visitor)?; + } + + Ok(()) } } +``` -impl<'kvs> Key<'kvs> { - /// Get a `Key` from a borrowed string. - pub fn from_str(key: &'kvs (impl AsRef + ?Sized)) -> Self { - Key { - inner: key.as_ref(), - } +When `std` is available, `Source` is implemented for some standard collections too: + +```rust +#[cfg(feature = "std")] +impl Source for Box where KVS: Source { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + (**self).visit(visitor) } +} - /// Get a borrowed string from a `Key`. - pub fn as_str(&self) -> &str { - &self.inner +#[cfg(feature = "std")] +impl Source for Arc where KVS: Source { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + (**self).visit(visitor) } } -impl<'kvs> AsRef for Key<'kvs> { - fn as_ref(&self) -> &str { - self.as_str() +#[cfg(feature = "std")] +impl Source for Rc where KVS: Source { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + (**self).visit(visitor) } } #[cfg(feature = "std")] -impl<'kvs> Borrow for Key<'kvs> { - fn borrow(&self) -> &str { - self.as_str() +impl Source for Vec where KVS: Source { + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + self.as_slice().visit(visitor) } } -impl<'kvs> Serialize for Key<'kvs> { - fn serialize(&self, serializer: S) -> Result +#[cfg(feature = "std")] +impl Source for collections::BTreeMap +where + K: Borrow + Ord, + V: Visit, +{ + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + for (k, v) in self { + visitor.visit_pair(k.to_key(), v.to_value())?; + } + + Ok(()) + } + + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> where - S: Serializer, + Q: Borrow, { - serializer.serialize_str(self.inner) + let key = key.to_key(); + collections::BTreeMap::get(self, key.as_ref()).map(Visit::to_value) } } -impl<'kvs> Display for Key<'kvs> { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - self.inner.fmt(f) +#[cfg(feature = "std")] +impl Source for collections::HashMap +where + K: Borrow + Eq + Hash, + V: Visit, +{ + fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + for (k, v) in self { + visitor.visit_pair(k.to_key(), v.to_value())?; + } + + Ok(()) } -} -impl<'kvs> Debug for Key<'kvs> { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - self.inner.fmt(f) + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: Borrow, + { + let key = key.to_key(); + collections::HashMap::get(self, key.as_ref()).map(Visit::to_value) } } ``` -Other standard implementations could be added for any `K: Borrow` in the same fashion. - -#### Ownership - -The `Key` type can either borrow or own its inner value. +The `BTreeMap` and `HashMap` implementations provide more efficient implementations of `Source::get`. -#### Thread-safety +### `Record` and `RecordBuilder` -The `Key` type is probably `Send` + `Sync`, but that's not guaranteed. +Structured key-value pairs can be set on a `RecordBuilder`: -### `source::Visitor` +```rust +impl<'a> RecordBuilder<'a> { + /// Set key values + pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a> { + self.record.kvs = kvs; + self + } +} +``` -The `Visitor` trait used by `Source` can visit a single key-value pair: +These key-value pairs can then be accessed on the built `Record`: ```rust -pub trait Visitor<'kvs> { - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; +#[derive(Clone, Debug)] +pub struct Record<'a> { + ... + + kvs: ErasedSource<'a>, } -impl<'a, 'kvs, T: ?Sized> Visitor<'kvs> for &'a mut T -where - T: Visitor<'kvs>, -{ - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { - (*self).visit_pair(k, v) +impl<'a> Record<'a> { + /// The key value pairs attached to this record. + /// + /// Pairs aren't guaranteed to be unique (the same key may be repeated with different values). + pub fn key_values(&self) -> ErasedSource { + self.kvs.clone() } } ``` -A `Visitor` may serialize the keys and values as it sees them. It may also do other work, like sorting or de-duplicating them. Operations that involve ordering keys will probably require allocations. - -#### Implementors +## The `log!` macros -There aren't any public implementors of `Visitor` in the `log` crate, but the `Source::try_for_each` and `Source::serialize_as_map` methods use the trait internally. +The `log!` macro will initially support a fairly spartan syntax for capturing structured data. The current `log!` macro looks like this: -Other crates that use key-value pairs will implement `Visitor`. +```rust +log!(); +``` -#### Object safety +This RFC proposes an additional semi-colon-separated part of the macro for capturing key-value pairs: -The `Visitor` trait is object-safe. +```rust +log!( ; ) +``` -### `Source` +The `;` and structured values are optional. If they're not present then the behaviour of the `log!` macro is the same as it is today. -The `Source` trait is a bit like `Serialize`. It gives us a way to inspect some arbitrary collection of key-value pairs using a visitor pattern: +As an example, this is what a `log!` statement containing structured key-value pairs could look like: ```rust -pub trait Source { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error>; - - ... -} - -impl<'a, T: ?Sized> Source for &'a T -where - T: Source, -{ - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { - (*self).visit(visitor) - } -} +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + correlation = correlation_id, + user = user +); ``` -`Source` doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. +There's a *big* design space around the syntax for capturing log records we could explore, especially when you consider procedural macros. The syntax proposed here for the `log!` macro is not designed to be really ergonomic. It's designed to be *ok*, and to encourage an exploration of the design space by offering a consistent base that other macros could build off. -#### Ownership +Having said that, there are a few unintrusive quality-of-life features that make the `log!` macros nicer to use with structured data. -The `Source` trait is probably the point where having some way to convert from a borrowed to an owned variant would make the most sense. +### Expansion -We could add a method to `Source` that allowed it to be converted into an owned variant with a default implementation: +Styructured key-value pairs in the `log!` macro expand to statements that borrow from their environment. ```rust -pub trait Source { - fn to_owned(&self) -> OwnedSource { - OwnedSource::serialized(self) - } -} +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + correlation = correlation_id, + user = user +); ``` -The `OwnedSource` could then encapsulte some sharable `dyn Source + Send + Sync`: +Will expand to something like: ```rust -#[derive(Clone)] -pub struct OwnedSource(Arc); +{ + let lvl = log::Level::Info; -impl OwnedSource { - fn new(impl Into>) -> Self { - OwnedSource(source.into()) - } + if lvl <= ::STATIC_MAX_LEVEL && lvl <= ::max_level() { + let correlation &correlation_id; + let user = &user; - fn serialize(impl Source) -> Self { - // Serialize the `Source` to something like - // `Vec<(String, OwnedValue)>` - // where `OwnedValue` is like `serde_json::Value` - ... + let kvs: &[(&str, &dyn::key_values::Visit)] = + &[("correlation", &correlation), ("user", &user)]; + + ::__private_api_log( + ::std::fmt::Arguments::new_v1( + &["This is the rendered ", ". It is not structured"], + &match (&"message",) { + (arg0,) => [::std::fmt::ArgumentV1::new(arg0, ::std::fmt::Display::fmt)], + }, + ), + lvl, + &("bin", "mod", "mod.rs", 13u32), + &kvs, + ); } -} +}; ``` -Other implementations of `Source` would be encouraged to override the `to_owned` method if they could provide a more efficient implementation. As an example, if there's a `Source` that is already wrapped up in an `Arc` then it can implement `to_owned` by just cloning itself. - -#### Adapters - -Some useful adapters exist as provided methods on the `Source` trait. They're similar to adapters on the standard `Iterator` trait: +# Drawbacks, rationale, and alternatives +[drawbacks]: #drawbacks -```rust -pub trait Source { - ... +Structured logging is a non-trivial feature to support. It adds complexity and overhead to the `log` crate. - /// Erase this `Source` so it can be used without - /// requiring generic type parameters. - fn erase(&self) -> ErasedSource - where - Self: Sized, - { - ErasedSource::erased(self) - } +## The `Debug + Serialize` blanket implementation of `Visit` - /// An adapter to borrow self. - fn by_ref(&self) -> &Self { - self - } +Making sure the `Visit` trait doesn't drop any implementations when the blanket implementation from `kv_serde` replaces the concrete ones is subtle and nonstandard. We have to be especially careful of references and generics. Any mistakes made here can result in dependencies that become uncompilable depending on Cargo features with no workaround besides removing that impl. Using a macro to define the small fixed set, and keeping all impls local to a single module, could help catch these cases. - /// Chain two `Source`s together. - fn chain(self, other: KVS) -> Chained - where - Self: Sized, - { - Chained(self, other) - } +It's also possibly surprising that the way the `Visit` trait is implemented in the ecosystem is through an entirely unrelated combination of `serde` and `std` traits. At least it's surprising on the surface. For libraries that define loggable types, they just implement some standard traits for serialization without involving `log` at all. These are traits they should be considering anyway. For consumers of the `log!` macro, they are mostly going to capture structured values for types they didn't produce, so having `serde` as the answer to _how can I log a `Url`, or a `Uuid`?_ sounds reasonable. It also means libraries defining types like `Url` and `Uuid` don't have yet another public serialization trait to implement. - /// Find the value for a given key. - /// - /// If the key is present multiple times, this method will - /// return the *last* value for the given key. - /// - /// The default implementation will scan all key-value pairs. - /// Implementors are encouraged provide a more efficient version - /// if they can. Standard collections like `BTreeMap` and `HashMap` - /// will do an indexed lookup instead of a scan. - fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> - where - Q: Borrow, - { - struct Get<'k, 'v>(Key<'k>, Option>); +If a library provides a datatype that you'd reasonably want to log, but it doesn't implement `serde::Serialize` then adding support for that type isn't just beneficial to you, but to anyone else that might want to serialize that type. - impl<'k, 'kvs> Visitor<'kvs> for Get<'k, 'kvs> { - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { - if k == self.0 { - self.1 = Some(v); - } +The real question for `serde` is whether or not depending on it as the general serialization framework in `log` creates the potential for some kind of ecosystem dichotomy if an alternative framework becomes popular where half the ecosystem uses `serde` and the other half uses something else that's incompatible. In that case `log` might not reasonably be able to support both without breakage if it goes down this path. The options for mitigating this in the design now is by either require all loggable types implement `Visit` explicitly, or just requiring callers opt in to `serde` support at the callsite in `log!`. - Ok(()) - } - } +### Require all loggable types implement `Visit` - let mut visitor = Get(key.to_key(), None); - let _ = self.visit(&mut visitor); +We could entirely punt on `serde` and just provide an API for simple values that implement the simple `Visit` trait. That avoids the potential serialization dichotomy in `log` altogether. - visitor.1 - } +The problem here is that any pervasive public API has the chance to create rifts in the ecosystem. By creating a new fundamental API for logging via the `Visit` trait we're just expanding the potential for dichotomies. - /// Apply a function to each key-value pair. - fn try_for_each(self, f: F) -> Result<(), Error> - where - Self: Sized, - F: FnMut(Key, Value) -> Result<(), E>, - E: Into, - { - struct ForEach(F, std::marker::PhantomData); +It also means we need to re-invent `serde`'s support for complex datastructures, the datatypes that implement its traits, and the formats that support it. We'll effectively turn `log` into a serialization framework of its own, and have to introduce arbitrary limitations on the kinds of values that can be logged. - impl<'kvs, F, E> Visitor<'kvs> for ForEach - where - F: FnMut(Key, Value) -> Result<(), E>, - E: Into, - { - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { - (self.0)(k, v).map_err(Into::into) - } - } +### Require callers opt in to `serde` support - self.visit(&mut ForEach(f, Default::default())) - } +We could avoid a potential serialization dichotomy by requiring callers opt in to `serde` support. That way if a new framework came along it could be naturally supported in the same way. There are a few ways callers could opt in to `serde` in the `log!` macros. The specifics aren't really important, but it could look something like this: - /// Serialize the key-value pairs as a map. - #[cfg(feature = "kv_serde")] - fn serialize_as_map(self) -> SerializeAsMap - where - Self: Sized, - { - SerializeAsMap(self) - } +```rust +use log::log_serde; - /// Serialize the key-value pairs as a map. - #[cfg(feature = "kv_serde")] - fn serialize_as_seq(self) -> SerializeAsSeq - where - Self: Sized, - { - SerializeAsSeq(self) - } -} +info!("A message"; user = log_serde!(user)); ``` -- `by_ref` to get a reference to a `Source` within a method chain. -- `chain` to concatenate one source with another. This is useful for composing implementations of `Log` together for contextual logging. -- `get` to try find the value associated with a key. -- `try_for_each` to try execute some closure over all key-value pairs. This is a convenient way to do something with each key-value pair without having to create and implement a `Visitor`. -- `serialize_as_map` to get a serializable map. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. -- `serialize_as_seq` to get a serializable sequence of tuples. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. +That way an alternative framework could be supported as: -None of these methods are required for the core API. They're helpful tools for working with key-value pairs with minimal machinery. Even if we don't necessarily include them right away it's worth having an API that can support them later without breakage. +```rust +use log::log_other_framework; -#### Object safety +info!("A message"; user = log_other_framework!(user)); +``` -`Source` is not object-safe because of the provided adapter methods not being object-safe. The only required method, `visit`, is safe though, so an object-safe version of `Source` that forwards this method can be reasonably written. +The problem with this approach is that it puts extra barriers in front of users that want to log. Instead of enabling crate features once and then logging structured values, each log statement needs to know how it can capture values. It also passes the burden of dealing with dichotomies onto every consumer of `log`. It seems like a reasonable idea from the perspective of the `log` crate, but is more hostile to end-users. -```rust -/// An erased `Source`. -#[derive(Clone)] -pub struct ErasedSource<'a>(&'a dyn ErasedSourceBridge); +There are substantially more end-users of the `log` crate calling the `log!` macros than there are frameworks and sinks that need to interact with its API so it's worth prioritizing end-user experience. Anything that requires end-users to opt-in to the most common scenarios isn't ideal. -impl<'a> ErasedSource<'a> { - /// Capture a `Source` and erase its concrete type. - pub fn new(kvs: &'a impl Source) -> Self { - ErasedSource(kvs) - } -} +# Prior art +[prior-art]: #prior-art -impl<'a> Default for ErasedSource<'a> { - fn default() -> Self { - ErasedSource(&(&[] as &[(&str, &dyn Visit)])) - } -} +Structured logging is a paradigm that's supported by logging frameworks in many language ecosystems. -impl<'a> Source for ErasedSource<'a> { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { - self.0.erased_visit(visitor) - } +## Rust - fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> - where - Q: Borrow, - { - let key = key.to_key(); - self.0.erased_get(key.as_ref()) - } -} +The `slog` library is a structured logging framework for Rust. Its API predates a stable `serde` crate so it defines its own traits that are similar to `serde::Serialize`. A log record consists of a rendered message and bag of structured key-value pairs. `slog` goes further than this RFC proposes by requiring callers of its `log!` macros to state whether key-values are owned or borrowed by the record, and whether the data is safe to share across threads. -/// A trait that erases a `Source` so it can be stored -/// in a `Record` without requiring any generic parameters. -trait ErasedSourceBridge { - fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; - fn erased_get<'kvs>(&'kvs self, key: &str) -> Option>; -} +This RFC proposes an API that's inspired by `slog`, but doesn't directly support distinguishing between owned or borrowed key-value pairs. Everything is borrowed. That means the only way to send a `Record` to another thread is to serialize it into a different type. -impl ErasedSourceBridge for KVS -where - KVS: Source + ?Sized, -{ - fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { - self.visit(visitor) - } +## Go - fn erased_get<'kvs>(&'kvs self, key: &str) -> Option> { - self.get(key) - } -} -``` +The `logrus` library is a structured logging framework for Go. It uses a similar separation of the textual log message from structured key-value pairs that this API proposes. -#### Implementors +## .NET -A `Source` with a single pair is implemented for a tuple of a key and value: +The C# community has mostly standardised around using message templates for packaging a log message with structured key-value pairs. Instead of logging a rendered message and separate bag of structured data, the log record contains a template that allows key-value pairs to be interpolated from the same bag of structured data. It avoids duplicating the same information multiple times. -```rust -impl Source for (K, V) -where - K: Borrow, - V: Visit, -{ - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { - visitor.visit_pair(self.0.to_key(), self.1.to_value()) - } -} -``` +Supporting something like message templates in Rust using the `log!` macros would probably require procedural macros. A macro like that could be built on top of the API proposed by this RFC. -A `Source` with multiple pairs is implemented for arrays of `Source`s: +# Unresolved questions +[unresolved-questions]: #unresolved-questions -```rust -impl Source for [KVS] where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { - for kv in self { - kv.visit(&mut visitor)?; - } +# Appendix - Ok(()) - } -} -``` +## Public API -When `std` is available, `Source` is implemented for some standard collections too: +For context, ignoring the `log!` macros, this is roughly the additional public API this RFC proposes to support structured logging: ```rust -#[cfg(feature = "std")] -impl Source for Box where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { - (**self).visit(visitor) - } +impl<'a> RecordBuilder<'a> { + /// Set the key-value pairs on a log record. + pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a>; } -#[cfg(feature = "std")] -impl Source for Arc where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { - (**self).visit(visitor) - } -} +impl<'a> Record<'a> { + /// Get the key-value pairs. + pub fn key_values(&self) -> ErasedSource; -#[cfg(feature = "std")] -impl Source for Rc where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { - (**self).visit(visitor) - } + /// Get a builder that's preconfigured from this record. + pub fn to_builder(&self) -> RecordBuilder; } -#[cfg(feature = "std")] -impl Source for Vec where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { - self.as_slice().visit(visitor) - } -} +pub mod kv { + pub mod source { + pub use kv::Error; -#[cfg(feature = "std")] -impl Source for collections::BTreeMap -where - K: Borrow + Ord, - V: Visit, -{ - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { - for (k, v) in self { - visitor.visit_pair(k.to_key(), v.to_value())?; + /// A source for key-value pairs. + pub trait Source { + /// Serialize the key value pairs. + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; + + /// Erase this `Source` so it can be used without + /// requiring generic type parameters. + fn erase(&self) -> ErasedSource + where + Self: Sized {} + + /// Find the value for a given key. + /// + /// If the key is present multiple times, this method will + /// return the *last* value for the given key. + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: Borrow {} + + /// An adapter to borrow self. + fn by_ref(&self) -> &Self {} + + /// Chain two `Source`s together. + fn chain(self, other: KVS) -> Chain + where + Self: Sized {} + + /// Apply a function to each key-value pair. + fn try_for_each(self, f: F) -> Result<(), Error> + where + Self: Sized, + F: FnMut(Key, Value) -> Result<(), E>, + E: Into {} + + /// Serialize the key-value pairs as a map. + fn serialize_as_map(self) -> SerializeAsMap + where + Self: Sized {} + + /// Serialize the key-value pairs as a sequence of tuples. + fn serialize_as_seq(self) -> SerializeAsSeq + where + Self: Sized {} } - Ok(()) - } + /// A visitor for a set of key-value pairs. + /// + /// The visitor is driven by an implementation of `Source`. + /// The visitor expects keys and values that satisfy a given lifetime. + pub trait Visitor<'kvs> { + /// Visit a single key-value pair. + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; + } - fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> - where - Q: Borrow, - { - let key = key.to_key(); - collections::BTreeMap::get(self, key.as_ref()).map(Visit::to_value) - } -} + /// An erased `Source`. + pub struct ErasedSource<'a> {} -#[cfg(feature = "std")] -impl Source for collections::HashMap -where - K: Borrow + Eq + Hash, - V: Visit, -{ - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { - for (k, v) in self { - visitor.visit_pair(k.to_key(), v.to_value())?; + impl<'a> ErasedSource<'a> { + /// Capture a `Source` and erase its concrete type. + pub fn new(kvs: &'a impl Source) -> Self {} } - Ok(()) - } + impl<'a> Clone for ErasedSource<'a> {} + impl<'a> Default for ErasedSource<'a> {} + impl<'a> Source for ErasedSource<'a> {} - fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> - where - Q: Borrow, - { - let key = key.to_key(); - collections::HashMap::get(self, key.as_ref()).map(Visit::to_value) - } -} -``` + /// A `Source` adapter that visits key-value pairs + /// in sequence. + /// + /// This is the result of calling `chain` on a `Source`. + pub struct Chain {} -The `BTreeMap` and `HashMap` implementations provide more efficient implementations of `Source::get`. + impl Source for Chain + where + A: Source, + B: Source {} -## The `log!` macros + /// A `Source` adapter that can be serialized as + /// a map using `serde`. + /// + /// This is the result of calling `serialize_as_map` on + /// a `Source`. + pub struct SerializeAsMap {} -The `log!` macro will initially support a fairly spartan syntax for capturing structured data. The current `log!` macro looks like this: + impl Serialize for SerializeAsMap + where + KVS: Source {} + + /// A `Source` adapter that can be serialized as + /// a sequence of tuples using `serde`. + /// + /// This is the result of calling `serialize_as_seq` on + /// a `Source`. + pub struct SerializeAsSeq {} + + impl Serialize for SerializeAsSeq + where + KVS: Source {} -```rust -log!(); -``` + impl Source for (K, V) + where + K: Borrow, + V: kv::value::Visit {} -This RFC proposes an additional semi-colon-separated part of the macro for capturing key-value pairs: + impl Source for [KVS] + where + KVS: Source {} -```rust -log!( ; ) -``` + #[cfg(feature = "std")] + impl Source for Box where KVS: Source {} + #[cfg(feature = "std")] + impl Source for Arc where KVS: Source {} + #[cfg(feature = "std")] + impl Source for Rc where KVS: Source {} -The `;` and structured values are optional. If they're not present then the behaviour of the `log!` macro is the same as it is today. + #[cfg(feature = "std")] + impl Source for Vec + where + KVS: Source {} -As an example, this is what a `log!` statement containing structured key-value pairs could look like: + #[cfg(feature = "std")] + impl Source for BTreeMap + where + K: Borrow + Ord, + V: kv::value::Visit {} -```rust -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - correlation = correlation_id, - user = user -); -``` + #[cfg(feature = "std")] + impl Source for HashMap + where + K: Borrow + Eq + Hash, + V: kv::value::Visit {} -There's a *big* design space around the syntax for capturing log records we could explore, especially when you consider procedural macros. The syntax proposed here for the `log!` macro is not designed to be really ergonomic. It's designed to be *ok*, and to encourage an exploration of the design space by offering a consistent base that other macros could build off. + /// The key in a key-value pair. + pub struct Key<'kvs> {} -Having said that, there are a few unintrusive quality-of-life features that make the `log!` macros nicer to use with structured data. + /// The value in a key-value pair. + pub use kv::value::Value; + } -### Expansion + pub mod value { + pub use kv::Error; -Styructured key-value pairs in the `log!` macro expand to statements that borrow from their environment. + /// An arbitrary structured value. + pub struct Value<'v> { + /// Create a new borrowed value. + pub fn new(v: &'v impl Visit) -> Self {} -```rust -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - correlation = correlation_id, - user = user -); -``` + /// Create a new borrowed value from an arbitrary type. + pub fn any(&'v T, fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self {} -Will expand to something like: + /// Visit the value with the given serializer. + pub fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> {} + } -```rust -{ - let lvl = log::Level::Info; + impl<'v> Debug for Value<'v> {} + impl<'v> Display for Value<'v> {} - if lvl <= ::STATIC_MAX_LEVEL && lvl <= ::max_level() { - let correlation &correlation_id; - let user = &user; + /// A serializer for primitive values. + pub trait Visitor { + /// Visit an arbitrary value. + fn visit_any(&mut self, v: Value) -> Result<(), Error>; - let kvs: &[(&str, &dyn::key_values::Visit)] = - &[("correlation", &correlation), ("user", &user)]; + /// Visit a signed integer. + fn visit_i64(&mut self, v: i64) -> Result<(), Error> {} - ::__private_api_log( - ::std::fmt::Arguments::new_v1( - &["This is the rendered ", ". It is not structured"], - &match (&"message",) { - (arg0,) => [::std::fmt::ArgumentV1::new(arg0, ::std::fmt::Display::fmt)], - }, - ), - lvl, - &("bin", "mod", "mod.rs", 13u32), - &kvs, - ); - } -}; -``` + /// Visit an unsigned integer. + fn visit_u64(&mut self, v: u64) -> Result<(), Error> {} -# Drawbacks, rationale, and alternatives -[drawbacks]: #drawbacks + /// Visit a floating point number. + fn visit_f64(&mut self, v: f64) -> Result<(), Error> {} -Structured logging is a non-trivial feature to support. + /// Visit a boolean. + fn visit_bool(&mut self, v: bool) -> Result<(), Error> {} -## The `Debug + Serialize` blanket implementation of `Visit` + /// Visit a single character. + fn visit_char(&mut self, v: char) -> Result<(), Error> {} -Making sure the `Visit` trait doesn't drop any implementations when the blanket implementation from `kv_serde` replaces the concrete ones is subtle and nonstandard. We have to be especially careful of references and generics. Any mistakes made here can result in dependencies that become uncompilable depending on Cargo features with no workaround besides removing that impl. Using a macro to define the small fixed set, and keeping all impls local to a single module, could help catch these cases. + /// Visit a UTF8 string. + fn visit_str(&mut self, v: &str) -> Result<(), Error> {} -It's also possibly surprising that the way the `Visit` trait is implemented in the ecosystem is through an entirely unrelated combination of `serde` and `std` traits. At least it's surprising on the surface. For libraries that define loggable types, they just implement some standard traits for serialization without involving `log` at all. These are traits they should be considering anyway. For consumers of the `log!` macro, they are mostly going to capture structured values for types they didn't produce, so having `serde` as the answer to _how can I log a `Url`, or a `Uuid`?_ sounds reasonable. It also means libraries defining types like `Url` and `Uuid` don't have yet another public serialization trait to implement. + /// Visit a raw byte buffer. + fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> {} -If a library provides a datatype that you'd reasonably want to log, but it doesn't implement `serde::Serialize` then adding support for that type isn't just beneficial to you, but to anyone else that might want to serialize that type. + /// Visit an empty value. + fn visit_none(&mut self) -> Result<(), Error> {} -The real question for `serde` is whether or not depending on it as the general serialization framework in `log` creates the potential for some kind of ecosystem dichotomy if an alternative framework becomes popular where half the ecosystem uses `serde` and the other half uses something else that's incompatible. In that case `log` might not reasonably be able to support both without breakage if it goes down this path. The options for mitigating this in the design now is by either require all loggable types implement `Visit` explicitly, or just requiring callers opt in to `serde` support at the callsite in `log!`. + /// Visit standard arguments. + fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error> {} + } -### Require all loggable types implement `Visit` + impl<'a, T: ?Sized> Visitor for &'a mut T + where + T: Visitor {} -We could entirely punt on `serde` and just provide an API for simple values that implement the simple `Visit` trait. That avoids the potential serialization dichotomy in `log` altogether. + /// Covnert a type into a value. + /// + /// ** This trait can't be implemented manually ** + pub trait Visit: private::Sealed { + /// Visit this value. + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; -The problem here is that any pervasive public API has the chance to create rifts in the ecosystem. By creating a new fundamental API for logging via the `Visit` trait we're just expanding the potential for dichotomies. + /// Convert a reference to this value into an erased `Value`. + fn to_value(&self) -> Value + where + Self: Sized, + { + Value::new(self) + } + } -It also means we need to re-invent `serde`'s support for complex datastructures, the datatypes that implement its traits, and the formats that support it. We'll effectively turn `log` into a serialization framework of its own, and have to introduce arbitrary limitations on the kinds of values that can be logged. + #[cfg(not(feature = "kv_serde"))] + impl Visit for u8 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for u16 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for u32 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for u64 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for u128 {} -### Require callers opt in to `serde` support + #[cfg(not(feature = "kv_serde"))] + impl Visit for i8 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for i16 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for i32 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for i64 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for i128 {} -We could avoid a potential serialization dichotomy by requiring callers opt in to `serde` support. That way if a new framework came along it could be naturally supported in the same way. There are a few ways callers could opt in to `serde` in the `log!` macros. The specifics aren't really important, but it could look something like this: + #[cfg(not(feature = "kv_serde"))] + impl Visit for f32 {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for f64 {} -```rust -use log::log_serde; + #[cfg(not(feature = "kv_serde"))] + impl Visit for char {} + #[cfg(not(feature = "kv_serde"))] + impl Visit for bool {} -info!("A message"; user = log_serde!(user)); -``` + #[cfg(not(feature = "kv_serde"))] + impl Visit for Option + where + T: Visit {} -That way an alternative framework could be supported as: + #[cfg(all(not(feature = "kv_serde"), feature = "std"))] + impl Visit for Box + where + T: Visit {} -```rust -use log::log_other_framework; + #[cfg(not(feature = "kv_serde"))] + impl<'a> Visit for &'a str {} + #[cfg(all(not(feature = "kv_serde"), feature = "std"))] + impl Visit for String {} -info!("A message"; user = log_other_framework!(user)); -``` + #[cfg(not(feature = "kv_serde"))] + impl<'a> Visit for &'a [u8] {} + #[cfg(all(not(feature = "kv_serde"), feature = "std"))] + impl Visit for Vec {} -The problem with this approach is that it puts extra barriers in front of users that want to log. Instead of enabling crate features once and then logging structured values, each log statement needs to know how it can capture values. It also passes the burden of dealing with dichotomies onto every consumer of `log`. It seems like a reasonable idea from the perspective of the `log` crate, but is more hostile to end-users. + #[cfg(not(feature = "kv_serde"))] + impl<'a, T> Visit for &'a T + where + T: Visit {} -# Prior art -[prior-art]: #prior-art + #[cfg(feature = "kv_serde")] + impl Visit for T + where + T: Debug + Serialize {} + } -Structured logging is a paradigm that's supported by logging frameworks in many language ecosystems. + pub use source::Source; -## Rust + /// An error encountered while visiting key-value pairs. + pub struct Error {} -The `slog` library is a structured logging framework for Rust. Its API predates a stable `serde` crate so it defines its own traits that are similar to `serde::Serialize`. A log record consists of a rendered message and bag of structured key-value pairs. `slog` goes further than this RFC proposes by requiring callers of its `log!` macros to state whether key-values are owned or borrowed by the record, and whether the data is safe to share across threads. + impl Error { + /// Create an error from a static message. + pub fn msg(msg: &'static str) -> Self {} -This RFC proposes an API that's inspired by `slog`, but doesn't directly support distinguishing between owned or borrowed key-value pairs. Everything is borrowed. That means the only way to send a `Record` to another thread is to serialize it into a different type. + /// Get a reference to a standard error. + #[cfg(feature = "std")] + pub fn as_error(&self) -> &(dyn std::error::Error + Send + Sync + 'static) {} -## Go + /// Convert into a standard error. + #[cfg(feature = "std")] + pub fn into_error(self) -> Box {} -The `logrus` library is a structured logging framework for Go. It uses a similar separation of the textual log message from structured key-value pairs that this API proposes. + /// Convert into a `serde` error. + #[cfg(feature = "kv_serde")] + pub fn into_serde(self) -> E + where + E: serde::ser::Error {} + } -## .NET + #[cfg(not(feature = "std"))] + impl From for Error {} -The C# community has mostly standardised around using message templates for packaging a log message with structured key-value pairs. Instead of logging a rendered message and separate bag of structured data, the log record contains a template that allows key-value pairs to be interpolated from the same bag of structured data. It avoids duplicating the same information multiple times. + #[cfg(feature = "std")] + impl From for Error + where + E: std::error::Error {} -Supporting something like message templates in Rust using the `log!` macros would probably require procedural macros. A macro like that could be built on top of the API proposed by this RFC. + #[cfg(feature = "std")] + impl From for Box {} -# Unresolved questions -[unresolved-questions]: #unresolved-questions + #[cfg(feature = "std")] + impl AsRef for Error {} +} +``` From 4d7587baf3c209c0c77ef88272c876aed234669e Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Sun, 11 Nov 2018 17:21:42 +1000 Subject: [PATCH 04/20] more tweaks to wording --- rfcs/0000-structured-logging.md | 303 ++++++++++++++++---------------- 1 file changed, 155 insertions(+), 148 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 1f3b095f0..ca6d52096 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -24,11 +24,11 @@ The API is heavily inspired by the `slog` logging framework. - [Implications for dependents](#implications-for-dependents) - [Cargo features](#cargo-features) - [Key-values API](#key-values-api) - - [`Visit`](#Visit) - - [`Value`](#value) - - [`Key`](#key) - - [`value::Visitor`](#valuevisitor) - [`Error`](#error) + - [`value::Visit`](#valueVisit) + - [`value::Visitor`](#valuevisitor) + - [`Value`](#valuevalue) + - [`Key`](#key) - [`Source`](#source) - [`source::Visitor`](#sourcevisitor) - [`Record` and `RecordBuilder`](#record-and-recordbuilder) @@ -89,7 +89,7 @@ Having a way to capture additional metadata is good for human-centric formats. H Why add structured logging support to the `log` crate when libraries like `slog` already exist and support it? `log` needs to support structured logging to make the experience of using `slog` and other logging tools in the Rust ecosystem more compatible. -On the surface there doesn't seem to be a lot of difference between `log` and `slog`, so why not just deprecate one in favour of the other? Conceptually, `log` and `slog` are different libraries that fill different use-cases, even if there's some overlap. +On the surface there doesn't seem to be a lot of difference between `log` and `slog`, so why not just deprecate one in favour of the other? Conceptually, `log` and `slog` are different libraries that fill different roles, even if there's some overlap. `slog` is a logging _framework_. It offers all the fundamental tools needed out-of-the-box to capture log records, define and implement the composable pieces of a logging pipeline, and pass them through that pipeline to an eventual destination. It has conventions and trade-offs baked into the design of its API. Loggers are treated explicitly as values in data structures and as arguments, and callers can control whether to pass owned or borrowed data. @@ -206,7 +206,7 @@ info!( ); ``` -If you come across a data type in the Rust ecosystem that you can't log, then try looking for a `serde` feature on the crate that defines it. If there isn't one already then adding it will be useful not just for you, but for anyone that might want to serialize those types for other reasons. +If you come across a data type in the Rust ecosystem that you can't log, then add the `kv_serde` feature to `log` and try looking for a `serde` feature on the crate that defines it. If there isn't one already then adding it will be useful not just for you, but for anyone that might want to serialize those types for other reasons. ## Supporting key-value pairs in `Log` implementations @@ -621,7 +621,7 @@ To make it possible to carry any arbitrary `S::Error` type, where we don't know ### `value::Visit` -The `Visit` trait can be treated like a lightweight subset of `serde::Serialize` that can interoperate with `serde` without necessarily depending on it: +The `Visit` trait can be treated like a lightweight subset of `serde::Serialize` that can interoperate with `serde`, without necessarily depending on it: ```rust /// A type that can be converted into a borrowed value. @@ -672,6 +672,17 @@ With default features, the types that implement `Visit` are a subset of `T: Debu -------------------------------------- ``` +The full set of standard types that implement `Visit` are: + +- Standard formats: `Arguments` +- Primitives: `bool`, `char` +- Unsigned integers: `u8`, `u16`, `u32`, `u64`, `u128` +- Signed integers: `i8`, `i16`, `i32`, `i64`, `i128` +- Strings: `&str`, `String` +- Bytes: `&[u8]`, `Vec` +- Paths: `Path`, `PathBuf` +- Special types: `Option` and `()`. + Enabling the `kv_serde` feature expands the set of types that implement `Visit` from this subset to all `T: Debug + Serialize`. #### Object safety @@ -982,6 +993,137 @@ We don't necessarily need a macro to make new implementations accessible for new In a future Rust with specialization we might be able to avoid all the machinery needed to keep the manual impls consistent with the blanket one, and allow consumers to implement `Visit` without needing `serde`. The specifics of specialization are still up in the air though. Under the proposed _always applicable_ rule, manual implementations like `impl Visit for Option where T: Visit` wouldn't be allowed. The ` where specialize(T: Visit)` scheme might make it possible though, although this would probably be a breaking change in any case. +### `value::Visitor` + +A visitor for a `Visit` that can interogate its structure: + +```rust +/// A serializer for primitive values. +pub trait Visitor { + /// Visit an arbitrary value. + /// + /// Depending on crate features there are a few things + /// you can do with a value. You can: + /// + /// - format it using `Debug`. + /// - serialize it using `serde`. + fn visit_any(&mut self, v: Value) -> Result<(), Error>; + + /// Visit a signed integer. + fn visit_i64(&mut self, v: i64) -> Result<(), Error> { + self.visit_any(v.to_value()) + } + + /// Visit an unsigned integer. + fn visit_u64(&mut self, v: u64) -> Result<(), Error> { + self.visit_any(v.to_value()) + } + + /// Visit a 128bit signed integer. + #[cfg(feature = "i128")] + fn visit_i128(&mut self, v: i128) -> Result<(), Error> { + self.visit_any(v.to_value()) + } + + /// Visit a 128bit unsigned integer. + #[cfg(feature = "i128")] + fn visit_u128(&mut self, v: u128) -> Result<(), Error> { + self.visit_any(v.to_value()) + } + + /// Visit a floating point number. + fn visit_f64(&mut self, v: f64) -> Result<(), Error> { + self.visit_any(v.to_value()) + } + + /// Visit a boolean. + fn visit_bool(&mut self, v: bool) -> Result<(), Error> { + self.visit_any(v.to_value()) + } + + /// Visit a single character. + fn visit_char(&mut self, v: char) -> Result<(), Error> { + let mut b = [0; 4]; + self.visit_str(&*v.encode_utf8(&mut b)) + } + + /// Visit a UTF8 string. + fn visit_str(&mut self, v: &str) -> Result<(), Error> { + self.visit_any((&v).to_value()) + } + + /// Visit a raw byte buffer. + fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { + self.visit_any((&v).to_value()) + } + + /// Visit standard arguments. + fn visit_none(&mut self) -> Result<(), Error> { + self.visit_any(().to_value()) + } + + /// Visit standard arguments. + fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error> { + self.visit_any(v.to_value()) + } +} + +impl<'a, T: ?Sized> Visitor for &'a mut T +where + T: Visitor, +{ + fn visit_any(&mut self, v: Value) -> Result<(), Error> { + (**self).visit_any(v) + } + + fn visit_i64(&mut self, v: i64) -> Result<(), Error> { + (**self).visit_i64(v) + } + + fn visit_u64(&mut self, v: u64) -> Result<(), Error> { + (**self).visit_u64(v) + } + + #[cfg(feature = "i128")] + fn visit_i128(&mut self, v: i128) -> Result<(), Error> { + (**self).visit_i128(v) + } + + #[cfg(feature = "i128")] + fn visit_u128(&mut self, v: u128) -> Result<(), Error> { + (**self).visit_u128(v) + } + + fn visit_f64(&mut self, v: f64) -> Result<(), Error> { + (**self).visit_f64(v) + } + + fn visit_bool(&mut self, v: bool) -> Result<(), Error> { + (**self).visit_bool(v) + } + + fn visit_char(&mut self, v: char) -> Result<(), Error> { + (**self).visit_char(v) + } + + fn visit_str(&mut self, v: &str) -> Result<(), Error> { + (**self).visit_str(v) + } + + fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { + (**self).visit_bytes(v) + } + + fn visit_none(&mut self) -> Result<(), Error> { + (**self).visit_none() + } + + fn visit_fmt(&mut self, args: &fmt::Arguments) -> Result<(), Error> { + (**self).visit_fmt(args) + } +} +``` + ### `Value` A `Value` is an erased container for a `Visit`, with a potentially short-lived lifetime: @@ -1121,7 +1263,7 @@ impl<'v> fmt::Display for Value<'v> { #### Serialization -When the `kv_serde` feature is enabled, `Value` implements the `Serialize` trait by forwarding to its inner value: +When the `kv_serde` feature is enabled, `Value` implements the `serde::Serialize` trait by forwarding to its inner value: ```rust impl<'v> Serialize for Value<'v> { @@ -1172,137 +1314,6 @@ The `Value` type borrows from its inner value. The `Value` type doesn't try to guarantee that values are `Send` or `Sync`, and doesn't offer any way of retaining that information when erasing. -### `value::Visitor` - -A visitor for a `Value` that can interogate its structure: - -```rust -/// A serializer for primitive values. -pub trait Visitor { - /// Visit an arbitrary value. - /// - /// Depending on crate features there are a few things - /// you can do with a value. You can: - /// - /// - format it using `Debug`. - /// - serialize it using `serde`. - fn visit_any(&mut self, v: Value) -> Result<(), Error>; - - /// Visit a signed integer. - fn visit_i64(&mut self, v: i64) -> Result<(), Error> { - self.visit_any(v.to_value()) - } - - /// Visit an unsigned integer. - fn visit_u64(&mut self, v: u64) -> Result<(), Error> { - self.visit_any(v.to_value()) - } - - /// Visit a 128bit signed integer. - #[cfg(feature = "i128")] - fn visit_i128(&mut self, v: i128) -> Result<(), Error> { - self.visit_any(v.to_value()) - } - - /// Visit a 128bit unsigned integer. - #[cfg(feature = "i128")] - fn visit_u128(&mut self, v: u128) -> Result<(), Error> { - self.visit_any(v.to_value()) - } - - /// Visit a floating point number. - fn visit_f64(&mut self, v: f64) -> Result<(), Error> { - self.visit_any(v.to_value()) - } - - /// Visit a boolean. - fn visit_bool(&mut self, v: bool) -> Result<(), Error> { - self.visit_any(v.to_value()) - } - - /// Visit a single character. - fn visit_char(&mut self, v: char) -> Result<(), Error> { - let mut b = [0; 4]; - self.visit_str(&*v.encode_utf8(&mut b)) - } - - /// Visit a UTF8 string. - fn visit_str(&mut self, v: &str) -> Result<(), Error> { - self.visit_any((&v).to_value()) - } - - /// Visit a raw byte buffer. - fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { - self.visit_any((&v).to_value()) - } - - /// Visit standard arguments. - fn visit_none(&mut self) -> Result<(), Error> { - self.visit_any(().to_value()) - } - - /// Visit standard arguments. - fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error> { - self.visit_any(v.to_value()) - } -} - -impl<'a, T: ?Sized> Visitor for &'a mut T -where - T: Visitor, -{ - fn visit_any(&mut self, v: Value) -> Result<(), Error> { - (**self).visit_any(v) - } - - fn visit_i64(&mut self, v: i64) -> Result<(), Error> { - (**self).visit_i64(v) - } - - fn visit_u64(&mut self, v: u64) -> Result<(), Error> { - (**self).visit_u64(v) - } - - #[cfg(feature = "i128")] - fn visit_i128(&mut self, v: i128) -> Result<(), Error> { - (**self).visit_i128(v) - } - - #[cfg(feature = "i128")] - fn visit_u128(&mut self, v: u128) -> Result<(), Error> { - (**self).visit_u128(v) - } - - fn visit_f64(&mut self, v: f64) -> Result<(), Error> { - (**self).visit_f64(v) - } - - fn visit_bool(&mut self, v: bool) -> Result<(), Error> { - (**self).visit_bool(v) - } - - fn visit_char(&mut self, v: char) -> Result<(), Error> { - (**self).visit_char(v) - } - - fn visit_str(&mut self, v: &str) -> Result<(), Error> { - (**self).visit_str(v) - } - - fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { - (**self).visit_bytes(v) - } - - fn visit_none(&mut self) -> Result<(), Error> { - (**self).visit_none() - } - - fn visit_fmt(&mut self, args: &fmt::Arguments) -> Result<(), Error> { - (**self).visit_fmt(args) - } -} -``` - ### `Key` A `Key` is a short-lived structure that can be represented as a UTF-8 string. This might be possible without allocating, or it might require a destination to write into: @@ -1404,9 +1415,7 @@ A `Visitor` may serialize the keys and values as it sees them. It may also do ot #### Implementors -There aren't any public implementors of `Visitor` in the `log` crate, but the `Source::try_for_each` and `Source::serialize_as_map` methods use the trait internally. - -Other crates that use key-value pairs will implement `Visitor`. +There aren't any public implementors of `Visitor` in the `log` crate. Other crates that use key-value pairs will implement `Visitor`. #### Object safety @@ -1414,13 +1423,11 @@ The `Visitor` trait is object-safe. ### `Source` -The `Source` trait is a bit like `Serialize`. It gives us a way to inspect some arbitrary collection of key-value pairs using a visitor pattern: +The `Source` trait is a bit like `std::iter::Iterator`. It gives us a way to inspect some arbitrary collection of key-value pairs using an object-safe visitor pattern: ```rust pub trait Source { fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error>; - - ... } impl<'a, T: ?Sized> Source for &'a T @@ -1469,7 +1476,7 @@ impl OwnedSource { } ``` -Other implementations of `Source` would be encouraged to override the `to_owned` method if they could provide a more efficient implementation. As an example, if there's a `Source` that is already wrapped up in an `Arc` then it can implement `to_owned` by just cloning itself. +Other implementations of `Source` are encouraged to override the `to_owned` method if they could provide a more efficient implementation. As an example, if there's a `Source` that is already wrapped up in an `Arc` then it can implement `to_owned` by just cloning itself. #### Adapters @@ -1642,7 +1649,7 @@ where #### Implementors -A `Source` with a single pair is implemented for a tuple of a key and value: +A `Source` containing a single key-value pair is implemented for a tuple of a key and value: ```rust impl Source for (K, V) @@ -1879,7 +1886,7 @@ The problem here is that any pervasive public API has the chance to create rifts It also means we need to re-invent `serde`'s support for complex datastructures, the datatypes that implement its traits, and the formats that support it. We'll effectively turn `log` into a serialization framework of its own, and have to introduce arbitrary limitations on the kinds of values that can be logged. -### Require callers opt in to `serde` support +### Require callers opt in to `serde` support at the callsite We could avoid a potential serialization dichotomy by requiring callers opt in to `serde` support. That way if a new framework came along it could be naturally supported in the same way. There are a few ways callers could opt in to `serde` in the `log!` macros. The specifics aren't really important, but it could look something like this: From d3df3679177ac8cc40aa5cf0662342d0a4eab784 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Mon, 12 Nov 2018 08:23:03 +1000 Subject: [PATCH 05/20] fix a broken link --- rfcs/0000-structured-logging.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index ca6d52096..e1645bd61 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -27,7 +27,7 @@ The API is heavily inspired by the `slog` logging framework. - [`Error`](#error) - [`value::Visit`](#valueVisit) - [`value::Visitor`](#valuevisitor) - - [`Value`](#valuevalue) + - [`Value`](#value) - [`Key`](#key) - [`Source`](#source) - [`source::Visitor`](#sourcevisitor) @@ -1872,6 +1872,8 @@ Structured logging is a non-trivial feature to support. It adds complexity and o Making sure the `Visit` trait doesn't drop any implementations when the blanket implementation from `kv_serde` replaces the concrete ones is subtle and nonstandard. We have to be especially careful of references and generics. Any mistakes made here can result in dependencies that become uncompilable depending on Cargo features with no workaround besides removing that impl. Using a macro to define the small fixed set, and keeping all impls local to a single module, could help catch these cases. +Another problem is documentation. It's not really easy to show in `rustdoc` how different crate features change the public API. Making it obvious how the bounds on `Visit` change might be tricky. + It's also possibly surprising that the way the `Visit` trait is implemented in the ecosystem is through an entirely unrelated combination of `serde` and `std` traits. At least it's surprising on the surface. For libraries that define loggable types, they just implement some standard traits for serialization without involving `log` at all. These are traits they should be considering anyway. For consumers of the `log!` macro, they are mostly going to capture structured values for types they didn't produce, so having `serde` as the answer to _how can I log a `Url`, or a `Uuid`?_ sounds reasonable. It also means libraries defining types like `Url` and `Uuid` don't have yet another public serialization trait to implement. If a library provides a datatype that you'd reasonably want to log, but it doesn't implement `serde::Serialize` then adding support for that type isn't just beneficial to you, but to anyone else that might want to serialize that type. From f5008efc9abf9b3d98c3d690ee015afabc8bd681 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Mon, 12 Nov 2018 11:35:36 +1000 Subject: [PATCH 06/20] add example of implementing multiple log traits --- rfcs/0000-structured-logging.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index e1645bd61..188e88c2a 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -1878,6 +1878,24 @@ It's also possibly surprising that the way the `Visit` trait is implemented in t If a library provides a datatype that you'd reasonably want to log, but it doesn't implement `serde::Serialize` then adding support for that type isn't just beneficial to you, but to anyone else that might want to serialize that type. +The degenerate case the `Debug + Serialize` implementation tries to avoid is one where a library needs to implement several very similar serialization-esc traits in order to be loggable in different frameworks: + +```rust +struct Url { .. } + +#[cfg(feature = "serde")] +impl serde::Serialize for Url { .. } + +#[cfg(feature = "log")] +impl log::kv::value::Visit for Url { .. } + +#[cfg(feature = "slog")] +impl slog::Value for Url { .. } + +#[cfg(feature = "log-framework-x")] +impl log_framework_x::Serialize for Url { .. } +``` + The real question for `serde` is whether or not depending on it as the general serialization framework in `log` creates the potential for some kind of ecosystem dichotomy if an alternative framework becomes popular where half the ecosystem uses `serde` and the other half uses something else that's incompatible. In that case `log` might not reasonably be able to support both without breakage if it goes down this path. The options for mitigating this in the design now is by either require all loggable types implement `Visit` explicitly, or just requiring callers opt in to `serde` support at the callsite in `log!`. ### Require all loggable types implement `Visit` From 64d3f9ded0bd1b8142611f836632afe04498d1f7 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Mon, 12 Nov 2018 12:25:24 +1000 Subject: [PATCH 07/20] more wording tidyups --- rfcs/0000-structured-logging.md | 222 +++++++++++++------------------- 1 file changed, 88 insertions(+), 134 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 188e88c2a..85c2a65de 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -134,7 +134,7 @@ info!( ### What can be logged? -Using default Cargo features, a closed set of types from the standard library are supported as structured values: +Using default Cargo features, a fixed set of types from the standard library are supported as structured values: - Standard formats: `Arguments` - Primitives: `bool`, `char` @@ -142,8 +142,10 @@ Using default Cargo features, a closed set of types from the standard library ar - Signed integers: `i8`, `i16`, `i32`, `i64`, `i128` - Strings: `&str`, `String` - Bytes: `&[u8]`, `Vec` -- Paths: `Path`, `PathBuf` -- Special types: `Option` and `()`. +- Paths: `&Path`, `PathBuf` +- Special types: `Option`, `&T`, and `()`. + +Using other Cargo features, any `T: std::fmt::Debug + serde::Serialize` can be logged, but we'll come back to that later. In the example from before, `correlation_id` and `user` can be used as structured values if they're in that set of concrete types: @@ -159,7 +161,7 @@ info!( ); ``` -What if the `correlation_id` is a `uuid::Uuid` instead of a string? What if the `user` is some other datastructure containing an id along with some other metadata? Only being able to log a few types from the standard library is a bit limiting. To make logging other values possible, the `kv_serde` Cargo feature expands the set of loggable values above to also include any other type that implements both `std::fmt::Debug` and `serde::Serialize`: +What if the `correlation_id` is a `uuid::Uuid` instead of a string? What if the `user` is some other datastructure containing an id and other metadata? Only being able to log a few primitive types from the standard library is a bit limiting. To make logging complex types and values from other libraries in the ecosystem possible, the `kv_serde` Cargo feature expands the set of loggable values above to also include any type that implements both `std::fmt::Debug` and `serde::Serialize`: ```toml [dependencies.log] @@ -210,7 +212,7 @@ If you come across a data type in the Rust ecosystem that you can't log, then ad ## Supporting key-value pairs in `Log` implementations -Capturing structured logs is only half the story. Implementors of the `Log` trait also need to be able to work with any key-value pairs associated with a log record. Key-value pairs are accessible on a log record through the `key_values` method: +Capturing structured logs is only half the story. Implementors of the `Log` trait also need to be able to work with any key-value pairs associated with a log record. Key-value pairs are accessible on a log record through the `Record::key_values` method: ```rust impl Record { @@ -235,7 +237,7 @@ Each key-value pair, shown as `$key: $value`, can be formatted from the `Source` ```rust use log::kv::Source; -fn write_pretty(w: impl Write, r: &Record) -> io::Result<()> { +fn log_record(w: impl Write, r: &Record) -> io::Result<()> { // Write the first line of the log record ... @@ -248,7 +250,9 @@ fn write_pretty(w: impl Write, r: &Record) -> io::Result<()> { } ``` -In the above example, the `try_for_each` method iterates over each key-value pair and writes them to the terminal. Now take the following json format: +In the above example, the `Source::try_for_each` method iterates over each key-value pair in the `Source` and writes them to the terminal. + +Let's look at a structured example. Take the following json format: ```json { @@ -262,24 +266,24 @@ In the above example, the `try_for_each` method iterates over each key-value pai } ``` -Defining a serializable structure based on a log record for this format could be done using `serde_derive`, and then written using `serde_json`. This requires the `kv_serde` feature: +A `Source` can be serialized as a map using `serde`. This requires the `kv_serde` feature: ```toml [dependencies.log] features = ["kv_serde"] ``` -The structured key-value pairs can then be naturally serialized as a map: +Defining a serializable structure based on a log record for this format could then be done using `serde_derive`, and then written using `serde_json`: ```rust use log::kv::Source; -fn write_json(w: impl Write, r: &Record) -> io::Result<()> { +fn log_record(w: impl Write, r: &Record) -> io::Result<()> { let r = SerializeRecord { lvl: r.level(), ts: epoch_millis(), msg: r.args().to_string(), - props: r.key_values().serialize_as_map(), + kvs: r.key_values().serialize_as_map(), }; serde_json::to_writer(w, &r)?; @@ -293,17 +297,15 @@ struct SerializeRecord { ts: u64, msg: String, #[serde(flatten)] - props: KVS, + kvs: KVS, } ``` -This time, instead of using the `try_for_each` method, we use `serialize_as_map` to get an adapter that will serialize each key-value pair as an entry in a map. - -The crate that produces log records might not be the same crate that consumes them. A producer can depend on the `kv_serde` feature to log more types, and a consumer will always be able to handle them, even if they don't depend on the `kv_serde` feature. +This time, instead of using the `Source::try_for_each` method, we use the `Source::serialize_as_map` method to get an adapter that will serialize each key-value pair as an entry in a `serde` map. ## Integrating log frameworks with `log` -The `Source` trait describes some container for structured key-value pairs. Other log frameworks that want to integrate with the `log` crate should build `Record`s that contain some implementation of `Source` based on their own structured logging. +The `Source` trait we saw previously describes some container for structured key-value pairs that can be iterated. Other log frameworks that want to integrate with the `log` crate should build `Record`s that contain some implementation of `Source` based on their own structured logging. The previous section demonstrated some of the methods available on `Source` like `Source::try_for_each` and `Source::serialize_as_map`. Both of those methods are provided on top of a single required `Source::visit` method. The `Source` trait itself looks something like this: @@ -315,7 +317,7 @@ trait Source { } ``` -where `source::Visitor` is another trait that accepts individual key-value pairs: +where `Visitor` is another trait that accepts individual key-value pairs: ```rust trait Visitor<'kvs> { @@ -323,35 +325,32 @@ trait Visitor<'kvs> { } ``` -The following example wraps up a `BTreeMap` and implements the `Source` trait for it: +As an example, let's say our log framework captures its key-value pairs in a `BTreeMap`: ```rust -use log::kv::source::{self, Source}; - -struct MySource { +struct KeyValues { data: BTreeMap, } +``` -impl Source for MySource { +The `Source` trait could be implemented for `KeyValues` like this: + +```rust +use log::kv::source::{self, Source}; + +impl Source for KeyValues { fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { self.data.visit(visitor) } } ``` -The implementation is pretty trivial because `BTreeMap` happens to already implement the `Source` trait. Now let's assume `BTreeMap` didn't implement `Source`. A manual implementation iterating through the map and converting the `(String, serde_json::Value)` pairs into types that can be visited could look like this: +This implementation is pretty trivial because `BTreeMap` happens to already implement the `Source` trait itself. Let's assume `BTreeMap` didn't implement `Source`. A manual implementation iterating through the map and converting the `(&String, &serde_json::Value)` pairs into types that can be visited could look like this: ```rust -use log::kv::{ - source::{self, Source}, - value, -}; - -struct MySource { - data: BTreeMap, -} +use log::kv::source::{self, Source}; -impl Source for MySource { +impl Source for KeyValues { fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { for (k, v) in self.data { visitor.visit_pair(source::Key::new(k), source::Value::new(v)) @@ -360,7 +359,7 @@ impl Source for MySource { } ``` -The `Key::new` method accepts any `T: Borrow`. The `Value::new` accepts any `T: std::fmt::Debug + serde::Serialize`. Values that can't implement `Debug + Serialize` can still be visited using the `source::Value::any` method. This method lets us provide an inline function that will visit the value: +The `Key::new` method accepts any `T: Borrow`. The `Value::new` accepts [a value that can be logged](#what-can-be-logged). Values that can't satisfy the requirements for logging on their own can still be supported using the `Value::any` method. This method lets us provide an inline function that will visit the value without implementing any traits: ```rust use log::kv::{ @@ -368,20 +367,24 @@ use log::kv::{ value, }; -struct MySource { +// We don't want to depend on `serde` +// but still want to be able to log `MyValue`s +#[derive(Debug)] +struct MyValue { .. } + +struct KeyValues { data: BTreeMap, } -impl Source for MySource { +impl Source for KeyValues { fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { for (k, v) in self.data { let key = source::Key::new(k); - // Let's assume `MyValue` doesn't implement `Serialize` - // Instead it implements `Display`. + // The `Value::any` method takes a value and a + // function that lets us visit it let value = source::Value::any(v: &MyValue, |v: &MyValue, visitor: &mut dyn value::Visitor| { - // Let's assume `MyValue` implements `Display` - visitor.visit_fmt(format_args!("{}", v)) + visitor.visit_fmt(format_args!("{:?}", v)) }); visitor.visit_pair(key, value) @@ -389,7 +392,7 @@ impl Source for MySource { } ``` -The `value::Visitor` trait is similar to `serde::Serializer`, but only supports a few common types: +The `value::Visitor` trait is similar to `serde::Serializer`, but only supports simple structures: ```rust trait Visitor { @@ -407,7 +410,7 @@ trait Visitor { } ``` -A `Source` doesn't have to just contain key-value pairs directly like `BTreeMap` though. It could also act like an adapter, like we have for iterators in the standard library. As another example, the following `Source` doesn't store any key-value pairs of its own, it will sort and de-duplicate pairs read from another source by first reading them into a map before forwarding them on: +A `Source` doesn't have to just contain key-value pairs directly like `BTreeMap` though. It could act like an adapter that changes its pairs before emitting them, like we have for iterators in the standard library. As another example, the following `Source` doesn't store any key-value pairs of its own, instead it will sort and de-duplicate pairs read from another source by first reading them into a map before forwarding them on: ```rust use log::kv::source::{self, Source, Visitor}; @@ -449,7 +452,7 @@ where ## Writing your own `value::Visitor` -Consumers of key-value pairs can visit structured values without needing a `serde::Serializer`. Instead they can implement a `value::Visitor`. A `Visitor` can always visit any structured value by formatting it using its `Debug` implementation: +Consumers of key-value pairs can visit structured values without needing a `serde::Serializer`, or needing to depend on `serde` at all. Instead they can implement a `value::Visitor`. A `Visitor` can always visit any structured value by formatting it using its `Debug` implementation: ```rust use log::kv::value::{self, Value}; @@ -621,7 +624,7 @@ To make it possible to carry any arbitrary `S::Error` type, where we don't know ### `value::Visit` -The `Visit` trait can be treated like a lightweight subset of `serde::Serialize` that can interoperate with `serde`, without necessarily depending on it: +The `Visit` trait can be treated like a lightweight subset of `serde::Serialize` that can interoperate with `serde`, without necessarily depending on it. It can't be implemented manually: ```rust /// A type that can be converted into a borrowed value. @@ -647,14 +650,25 @@ mod private { } ``` -We'll look at the `Visitor` trait shortly. It's like `serde::Serializer`. +We'll look at the `Visitor` trait in more detail later. -`Visit` is the trait bound that structured values need to satisfy before they can be logged. The trait can't be implemented outside of the `log` crate, because it uses blanket implementations depending on Cargo features. If a crate defines a datastructure that users might want to log, instead of trying to implement `Visit`, it should implement the `serde::Serialize` and `std::fmt::Debug` traits. This means that `Visit` can piggyback off `serde::Serialize` as the pervasive public dependency, so that `Visit` itself doesn't need to be one. +`Visit` is the trait bound that structured values need to satisfy before they can be logged. The trait can't be implemented outside of the `log` crate, because it uses blanket implementations depending on Cargo features. If a crate defines a datastructure that users might want to log, instead of trying to implement `Visit`, it should implement the `serde::Serialize` and `std::fmt::Debug` traits. It shouldn't need to depend on the `log` crate at all. This means that `Visit` can piggyback off `serde::Serialize` as the pervasive public dependency, so that `Visit` itself doesn't need to be one. The trait bounds on `private::Sealed` ensure that any generic `T: Visit` carries some additional traits that are needed for the blanket implementation of `Serialize`. As an example, any `Option` can also be treated as `Option` and therefore implement `Serialize` itself. The `Visit` trait is responsible for a lot of type system mischief. With default features, the types that implement `Visit` are a subset of `T: Debug + Serialize`: +- Standard formats: `Arguments` +- Primitives: `bool`, `char` +- Unsigned integers: `u8`, `u16`, `u32`, `u64`, `u128` +- Signed integers: `i8`, `i16`, `i32`, `i64`, `i128` +- Strings: `&str`, `String` +- Bytes: `&[u8]`, `Vec` +- Paths: `&Path`, `PathBuf` +- Special types: `Option`, `&T`, and `()`. + +Enabling the `kv_serde` feature expands the set of types that implement `Visit` from this subset to all `T: Debug + Serialize`. + ``` -------- feature = "kv_serde" -------- | | @@ -663,8 +677,12 @@ With default features, the types that implement `Visit` are a subset of `T: Debu | | | - not(feature = "kv_serde") - | | | | | -| | u8, i8, &str, &[u8], bool | | -| | etc... | | +| | u8, u16, u32, u64, u128 | | +| | i8, i16, i32, i64, i128 | | +| | bool, char, &str, String | | +| | &[u8], Vec | | +| | &Path, PathBuf, Arguments | | +| | Option, &T, () | | | | | | | ----------------------------- | | | @@ -672,24 +690,11 @@ With default features, the types that implement `Visit` are a subset of `T: Debu -------------------------------------- ``` -The full set of standard types that implement `Visit` are: - -- Standard formats: `Arguments` -- Primitives: `bool`, `char` -- Unsigned integers: `u8`, `u16`, `u32`, `u64`, `u128` -- Signed integers: `i8`, `i16`, `i32`, `i64`, `i128` -- Strings: `&str`, `String` -- Bytes: `&[u8]`, `Vec` -- Paths: `Path`, `PathBuf` -- Special types: `Option` and `()`. - -Enabling the `kv_serde` feature expands the set of types that implement `Visit` from this subset to all `T: Debug + Serialize`. - #### Object safety -The `Visit` trait is not object-safe, but has a simple object-safe wrapper used by `Value`. +The `Visit` trait is not object-safe, but has a simple object-safe wrapper used by `Value`. We'll look at `Value` in more detail later. -#### Without `serde` +#### `Visit` without the `kv_serde` feature Without the `kv_serde` feature, the `Visit` trait is implemented for a fixed set of fundamental types from the standard library: @@ -864,7 +869,7 @@ impl Visit for PathBuf { } ``` -#### With `serde` +#### `Visit` with the `kv_serde` feature With the `kv_serde` feature, the `Visit` trait is implemented for any type that is `Debug + Serialize`: @@ -1070,58 +1075,7 @@ pub trait Visitor { impl<'a, T: ?Sized> Visitor for &'a mut T where - T: Visitor, -{ - fn visit_any(&mut self, v: Value) -> Result<(), Error> { - (**self).visit_any(v) - } - - fn visit_i64(&mut self, v: i64) -> Result<(), Error> { - (**self).visit_i64(v) - } - - fn visit_u64(&mut self, v: u64) -> Result<(), Error> { - (**self).visit_u64(v) - } - - #[cfg(feature = "i128")] - fn visit_i128(&mut self, v: i128) -> Result<(), Error> { - (**self).visit_i128(v) - } - - #[cfg(feature = "i128")] - fn visit_u128(&mut self, v: u128) -> Result<(), Error> { - (**self).visit_u128(v) - } - - fn visit_f64(&mut self, v: f64) -> Result<(), Error> { - (**self).visit_f64(v) - } - - fn visit_bool(&mut self, v: bool) -> Result<(), Error> { - (**self).visit_bool(v) - } - - fn visit_char(&mut self, v: char) -> Result<(), Error> { - (**self).visit_char(v) - } - - fn visit_str(&mut self, v: &str) -> Result<(), Error> { - (**self).visit_str(v) - } - - fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { - (**self).visit_bytes(v) - } - - fn visit_none(&mut self) -> Result<(), Error> { - (**self).visit_none() - } - - fn visit_fmt(&mut self, args: &fmt::Arguments) -> Result<(), Error> { - (**self).visit_fmt(args) - } -} + T: Visitor { } ``` ### `Value` @@ -1165,7 +1119,7 @@ impl<'v> Value<'v> { #### `ErasedVisit` -The `ErasedVisit` trait is an object-safe wrapper for the `Visit` trait. `Visit` itself isn't technically object-safe because it needs the non-object-safe `serde::Serialize` as a supertrait to carry in generic contexts: +The `ErasedVisit` trait is a private object-safe wrapper for the `Visit` trait. `Visit` itself isn't technically object-safe because it needs the non-object-safe `serde::Serialize` as a supertrait to carry in generic contexts: ```rust #[cfg(not(feature = "kv_serde"))] @@ -1190,7 +1144,7 @@ where #### `Any` -Other logging frameworks that want to integrate with `log` might not want to pull in a `serde` dependency, and so they couldn't implement the `Visit` trait. The `Any` type uses some `std::fmt` inspired black-magic to allow values that don't implement the `Visit` trait to be erased in a `Value`. It does this by taking a borrowed value along with a function pointer that looks like `Visit::visit`: +Other logging frameworks that want to integrate with `log` might not want to pull in a `serde` dependency, and so they couldn't implement the `Visit` trait. The `Any` type is a private container that uses some `std::fmt` inspired black-magic to allow values that don't implement the `Visit` trait to be erased in a `Value`. It does this by taking a borrowed value along with a function pointer that looks like `Visit::visit`: ```rust struct Void { @@ -1403,12 +1357,7 @@ pub trait Visitor<'kvs> { impl<'a, 'kvs, T: ?Sized> Visitor<'kvs> for &'a mut T where - T: Visitor<'kvs>, -{ - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { - (*self).visit_pair(k, v) - } -} + T: Visitor<'kvs> { } ``` A `Visitor` may serialize the keys and values as it sees them. It may also do other work, like sorting or de-duplicating them. Operations that involve ordering keys will probably require allocations. @@ -1432,12 +1381,7 @@ pub trait Source { impl<'a, T: ?Sized> Source for &'a T where - T: Source, -{ - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { - (*self).visit(visitor) - } -} + T: Source { } ``` `Source` doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. @@ -1592,7 +1536,7 @@ None of these methods are required for the core API. They're helpful tools for w #### Object safety -`Source` is not object-safe because of the provided adapter methods not being object-safe. The only required method, `visit`, is safe though, so an object-safe version of `Source` that forwards this method can be reasonably written. +`Source` is not object-safe because of the provided adapter methods not being object-safe. The only required method, `visit`, is safe though, so an object-safe version of `Source` that forwards this method can be reasonably written in a similar way to the object-safe `ErasedVisit`: ```rust /// An erased `Source`. @@ -1870,6 +1814,10 @@ Structured logging is a non-trivial feature to support. It adds complexity and o ## The `Debug + Serialize` blanket implementation of `Visit` +### Drawbacks + +The main drawbacks of the feature-gated `Debug + Serialize` approach are that it's non-standard, which makes it harder to communicate. + Making sure the `Visit` trait doesn't drop any implementations when the blanket implementation from `kv_serde` replaces the concrete ones is subtle and nonstandard. We have to be especially careful of references and generics. Any mistakes made here can result in dependencies that become uncompilable depending on Cargo features with no workaround besides removing that impl. Using a macro to define the small fixed set, and keeping all impls local to a single module, could help catch these cases. Another problem is documentation. It's not really easy to show in `rustdoc` how different crate features change the public API. Making it obvious how the bounds on `Visit` change might be tricky. @@ -1898,15 +1846,21 @@ impl log_framework_x::Serialize for Url { .. } The real question for `serde` is whether or not depending on it as the general serialization framework in `log` creates the potential for some kind of ecosystem dichotomy if an alternative framework becomes popular where half the ecosystem uses `serde` and the other half uses something else that's incompatible. In that case `log` might not reasonably be able to support both without breakage if it goes down this path. The options for mitigating this in the design now is by either require all loggable types implement `Visit` explicitly, or just requiring callers opt in to `serde` support at the callsite in `log!`. -### Require all loggable types implement `Visit` +### Alternatives + +#### Require all loggable types implement `Visit` -We could entirely punt on `serde` and just provide an API for simple values that implement the simple `Visit` trait. That avoids the potential serialization dichotomy in `log` altogether. +We could entirely punt on `serde` and the non-standard implementations of `Visit`, and just provide an API for values that manually implement the `Visit` trait. That avoids the potential serialization dichotomy in `log` altogether. -The problem here is that any pervasive public API has the chance to create rifts in the ecosystem. By creating a new fundamental API for logging via the `Visit` trait we're just expanding the potential for dichotomies. +The problem here is that any pervasive public API has the chance to create rifts in the ecosystem. By creating a new fundamental API for logging via the `Visit` trait we're just expanding the potential for dichotomies. It also makes `log` another participant in the grab-bag of serialization traits that types in the ecosystem need to implement. It also means we need to re-invent `serde`'s support for complex datastructures, the datatypes that implement its traits, and the formats that support it. We'll effectively turn `log` into a serialization framework of its own, and have to introduce arbitrary limitations on the kinds of values that can be logged. -### Require callers opt in to `serde` support at the callsite +#### Attempt to deprecate other log-specific serialization traits in favour of `Visit` + +The case where types in the ecosystem like `Url` need to implement a random grab-bag of serialization traits to be compatible with every framework could be avoided by encouraging other frameworks to use `log`'s `Visit` instead of defining their own. We might get to a point where frameworks decide to standardize on a particular API, but that decision should be made naturally instead of being forced onto the ecosystem. The approach this RFC takes makes it possible to standardise, but doesn't depend on it. This also doesn't solve the issue of making `log` into a pervasive public dependency in the first place. + +#### Require callers opt in to `serde` support at the callsite We could avoid a potential serialization dichotomy by requiring callers opt in to `serde` support. That way if a new framework came along it could be naturally supported in the same way. There are a few ways callers could opt in to `serde` in the `log!` macros. The specifics aren't really important, but it could look something like this: From 2779a898f05af1a87c2a69484404e9e2876e8cff Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Tue, 29 Jan 2019 14:20:57 +1000 Subject: [PATCH 08/20] WIP: work on new API design --- rfcs/0000-structured-logging.md | 2335 +++++++++++++++---------------- 1 file changed, 1139 insertions(+), 1196 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 85c2a65de..c2f2d9e17 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -3,9 +3,9 @@ Add support for structured logging to the `log` crate in both `std` and `no_std` environments, allowing log records to carry typed data beyond a textual message. This document serves as an introduction to what structured logging is all about, and as an RFC for an implementation in the `log` crate. -`log` will provide a lightweight fundamental serialization API out-of-the-box that allows a fixed set of common types from the standard library to be logged as structured values. Using optional Cargo features, that set can be expanded to support anything that implements `serde::Serialize + std::fmt::Debug`. It doesn't turn `log` into a pervasive public dependency to support structured logging for types outside the standard library. +`log` will provide an API for capturing structured data that's agnostic of the underlying serialization framework, whether that's `std::fmt`, `serde`, or `sval`. -The API is heavily inspired by the `slog` logging framework. +The API is heavily inspired by `slog` and `tokio-trace`. > NOTE: Code in this RFC uses recent language features like `impl Trait`, but can be implemented without them. @@ -25,19 +25,16 @@ The API is heavily inspired by the `slog` logging framework. - [Cargo features](#cargo-features) - [Key-values API](#key-values-api) - [`Error`](#error) - - [`value::Visit`](#valueVisit) - - [`value::Visitor`](#valuevisitor) - [`Value`](#value) + - [`ToValue`](#tovalue) - [`Key`](#key) + - [`ToKey`](#tokey) - [`Source`](#source) - - [`source::Visitor`](#sourcevisitor) + - [`Visitor`](#visitor) - [`Record` and `RecordBuilder`](#record-and-recordbuilder) - [The `log!` macros](#the-log-macros) - [Drawbacks, rationale, and alternatives](#drawbacks-rationale-and-alternatives) - [Prior art](#prior-art) - - [Rust](#rust) - - [Go](#go) - - [.NET](#net) - [Unresolved questions](#unresolved-questions) - [Appendix](#appendix) - [Public API](#public-api) @@ -89,17 +86,19 @@ Having a way to capture additional metadata is good for human-centric formats. H Why add structured logging support to the `log` crate when libraries like `slog` already exist and support it? `log` needs to support structured logging to make the experience of using `slog` and other logging tools in the Rust ecosystem more compatible. -On the surface there doesn't seem to be a lot of difference between `log` and `slog`, so why not just deprecate one in favour of the other? Conceptually, `log` and `slog` are different libraries that fill different roles, even if there's some overlap. +On the surface there doesn't seem to be a lot of difference between `log` and `slog`, so why not just deprecate one in favor of the other? Conceptually, `log` and `slog` are different libraries that fill different roles, even if there's some overlap. -`slog` is a logging _framework_. It offers all the fundamental tools needed out-of-the-box to capture log records, define and implement the composable pieces of a logging pipeline, and pass them through that pipeline to an eventual destination. It has conventions and trade-offs baked into the design of its API. Loggers are treated explicitly as values in data structures and as arguments, and callers can control whether to pass owned or borrowed data. +`slog` is a logging _framework_. It offers all the fundamental tools needed out-of-the-box to capture log records, define and implement the pieces of a logging pipeline, and pass them through that pipeline to an eventual destination. It has conventions and trade-offs baked into the design of its API. Loggers are treated explicitly as values in data structures and as arguments, and callers can control whether to pass owned or borrowed data. -`log` is a logging _facade_. It's only concerned with a standard, minimal API for capturing log records, and surfacing those records to some consumer. The tools provided by `log` are only those that are fundamental to the operation of the `log!` macro. From `log`'s point of view, a logging framework like `slog` is a black-box implementation of the `Log` trait. In this role, the `Log` trait can act as a common entrypoint for capturing log records. That means the `Record` type can act as a common container for describing a log record. `log` has its own set of trade-offs baked into the design of its API. The `log!` macro assumes a single, global entrypoint, and all data in a log record is borrowed from the callsite. +`log` is a logging _facade_. It's only concerned with a standard, minimal API for capturing log records, and surfacing those records to some consumer. The tools provided by `log` are only those that are fundamental to the operation of the `log!` macro. From `log`'s point of view, a logging framework like `slog` is a black-box implementation of the `Log` trait. In this role, the `Log` trait can act as a common entry-point for capturing log records. That means the `Record` type can act as a common container for describing a log record. `log` has its own set of trade-offs baked into the design of its API. The `log!` macro assumes a single, global entry-point, and all data in a log record is borrowed from the call-site. -A healthy logging ecosystem needs both `log` and frameworks like `slog`. As a standard API, `log` can support a diverse but cohesive ecosystem of logging tools in Rust by acting as the glue between libraries, frameworks, and applications. A lot of libraries already depend on it. In order to really fulfil this role though, `log` needs to support structured logging so that libraries and their consumers can take advantage of it in a framework-agnostic way. +A healthy logging ecosystem needs both `log` and frameworks like `slog`. As a standard API, `log` can support a diverse but cohesive ecosystem of logging tools in Rust by acting as the glue between libraries, frameworks, and applications. A lot of libraries already depend on it. In order to really fulfill this role though, `log` needs to support structured logging so that libraries and their consumers can take advantage of it in a framework-agnostic way. # Guide-level explanation [guide-level-explanation]: #guide-level-explanation +This section introduces the new structured logging API through a tour of how structured values can be captured and consumed. + ## Logging structured key-value pairs Structured logging is supported in `log` by allowing typed key-value pairs to be associated with a log record. A `;` separates structured key-value pairs from values that are replaced into the message: @@ -134,7 +133,58 @@ info!( ### What can be logged? -Using default Cargo features, a fixed set of types from the standard library are supported as structured values: +A type can be logged if it implements the `ToValue` trait: + +```rust +pub trait ToValue { + fn to_Value(&self) -> Value; +} +``` + +where `Value` is a special container for structured data: + +```rust +pub struct Value<'v>(_); + +// A value can always be debugged +impl<'v> Debug for Value<'v> { + .. +} +``` + +We'll look at `Value` in more detail later. For now, we can think of it as a container that normalizes capturing and emitting the structure of values. + +In the example from before: + +```rust +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + correlation = correlation_id, + user +); +``` + +the `correlation_id` and `user` fields can be used as structured values if they implement the `ToValue` trait: + +``` +info!( + "This is the rendered {message}. It is not structured", + message = "message"; + ^^^^^^^^^^^^^^^^^^^ + impl Display + + correlation = correlation_id, + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + impl ToValue + + user + ^^^^ + impl ToValue +); +``` + +Within `log` itself, a fixed set of primitive types from the standard library implement the `ToValue` trait: - Standard formats: `Arguments` - Primitives: `bool`, `char` @@ -145,70 +195,135 @@ Using default Cargo features, a fixed set of types from the standard library are - Paths: `&Path`, `PathBuf` - Special types: `Option`, `&T`, and `()`. -Using other Cargo features, any `T: std::fmt::Debug + serde::Serialize` can be logged, but we'll come back to that later. +Each of these types implements `ToValue` in a way that retains their typing. Using `u8` as an example: + +```rust +impl ToValue for u8 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.u64(*v as u64)) + } +} +``` -In the example from before, `correlation_id` and `user` can be used as structured values if they're in that set of concrete types: +The `Value::from_any` method accepts any type, `&T`, and an ad-hoc function that tells the `Value` what its structure is: ```rust -let user = "a user id"; -let correlation_id = "some correlation id"; +impl<'v> Value<'v> { + pub fn from_any(v: &'v T, from: fn(FromAny, &T) -> Result<(), Error>) -> Self { + .. + } +} -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - correlation = correlation_id, - user -); +pub struct FromAny(_); + +impl FromAny { + pub fn debug(v: impl Debug) -> Result<(), Error> { + .. + } + + fn u64(v: u64) -> Result<(), Error> { + .. + } +} +``` + +This machinery is very similar to the internals of `std::fmt`. + +Only being able to log primitive types from the standard library is a bit limiting though. What if `correlation_id` is a `uuid::Uuid`, and `user` is a struct, `User`, with fields? + +#### Implementing `ToValue` for a simple value + +`uuid::Uuid` could implement the `ToValue` trait directly by capturing its structure as a debuggable format: + +```rust +impl ToValue for Uuid { + fn to_value(&self) -> Value { + Value::from_any(self, |from, uuid| from.debug(uuid.to_hyphenated())) + } +} ``` -What if the `correlation_id` is a `uuid::Uuid` instead of a string? What if the `user` is some other datastructure containing an id and other metadata? Only being able to log a few primitive types from the standard library is a bit limiting. To make logging complex types and values from other libraries in the ecosystem possible, the `kv_serde` Cargo feature expands the set of loggable values above to also include any type that implements both `std::fmt::Debug` and `serde::Serialize`: +There's some subtlety in this implementation. The actual value whose structure is captured is not the `&'v Uuid`, it's the owned `ToHyphenated<'v>` structure. This is why `Value::from_any` uses a separate function for capturing the structure of its values. It lets us capture a borrowed `Uuid` with the right lifetime `'v`, but materialize an owned `ToHyphenated` with the structure we want. + +#### Implementing `ToValue` for a complex value + +A structure like `User` is a bit different. It could be represented using `Debug`, but then the contents of its fields would be lost in an opaque and unstructured string. It would be better represented as a map of key-value pairs. However, complex values like maps and sequences aren't directly supported in `log`. They're offloaded to serialization frameworks like `serde` and `sval` that are capable of handling them effectively. + +Fundamental serialization frameworks do have direct integration with `log`'s `Value` type through Cargo features. Let's use `sval` as an example. It's a serialization framework that's built specifically for structured logging. Adding the `kv_sval` feature to `log` will enable its integration: ```toml [dependencies.log] -features = ["kv_serde"] +features = ["kv_sval"] +``` + +The `User` type can then derive `sval`'s `Value` trait and implement `log`'s `ToValue` trait in terms of `sval`: + +```rust +#[derive(Debug, Value)] +struct User { + name: String, +} + +impl ToValue for User { + fn to_value(&self) -> Value { + Value::from_sval(self) + } +} +``` + +Using `serde` instead of `sval` is a similar story: -[dependencies.uuid] -features = ["serde"] +```toml +[dependencies.log] +features = ["kv_serde"] ``` ```rust #[derive(Debug, Serialize)] struct User { - id: Uuid, - .. + name: String, +} + +impl ToValue for User { + fn to_value(&self) -> Value { + Value::from_serde(self) + } } +``` + +#### Capturing values without implementing `ToValue` + +Instead of implementing `ToValue` on types throughout the ecosystem at all, callers of the `log!` macros could instead create ad-hoc `Value`s from their data: -let user = User { id, .. }; -let correlation_id = Uuid::new_v4(); +```rust +use log::key_values::Value; info!( "This is the rendered {message}. It is not structured", message = "message"; - correlation = correlation_id, - user + correlation = Value::from_serde(correlation_id), + user = Value::from_sval(user), ); ``` -So the effective trait bounds for structured values are `Debug + Serialize`: +In this example, neither `correlation_id` nor `user` need to implement any traits from `log`: ``` info!( "This is the rendered {message}. It is not structured", message = "message"; - ^^^^^^^^^ - Display - correlation = correlation_id, - ^^^^^^^^^^^^^^ - Debug + Serialize + correlation = Value::from_serde(correlation_id), + ^^^^^^^^^^^^^^ + impl serde::Serialize + Debug - user - ^^^^ - Debug + Serialize + user = Value::from_sval(user), + ^^^^ + impl sval::Value + Debug ); ``` -If you come across a data type in the Rust ecosystem that you can't log, then add the `kv_serde` feature to `log` and try looking for a `serde` feature on the crate that defines it. If there isn't one already then adding it will be useful not just for you, but for anyone that might want to serialize those types for other reasons. +Having to decorate every value in every `log!` macro is not ideal for users of the `log` crate, but it does open the door for alternative implementations of the `log!` macros to be more opinionated about what kinds of structured values they'll accept by default. ## Supporting key-value pairs in `Log` implementations @@ -220,7 +335,49 @@ impl Record { } ``` -where `Source` is a trait for iterating over the individual key-value pairs. +where `Source` is a trait for iterating over the individual key-value pairs: + +```rust +pub trait Source { + // Get the value for a given key + fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> + where + Q: ToKey, + { + .. + } + + // Run a function for each key-value pair + fn try_for_each(self, f: F) -> Result<(), Error> + where + Self: Sized, + F: FnMut(Key, Value) -> Result<(), E>, + E: Into, + { + .. + } + + // Serialize the source as a map of key-value pairs + fn as_map(self) -> AsMap + where + Self: Sized, + { + .. + } + + // Serialize the source as a sequence of key-value tuples + fn as_seq(self) -> AsSeq + where + Self: Sized, + { + .. + } + + // Other methods we'll look at later +} +``` + +### Writing key-value pairs as text To demonstrate how to work with a `Source`, let's take the terminal log format from before: @@ -232,14 +389,14 @@ correlation: 123 took: 18 ``` -Each key-value pair, shown as `$key: $value`, can be formatted from the `Source` using the `std::fmt` machinery: +Each key-value pair, shown as a `$key: $value` line, can be formatted from the `Source` using the `std::fmt` machinery: ```rust -use log::kv::Source; +use log::key_values::Source; fn log_record(w: impl Write, r: &Record) -> io::Result<()> { // Write the first line of the log record - ... + .. // Write each key-value pair on a new line record @@ -252,7 +409,9 @@ fn log_record(w: impl Write, r: &Record) -> io::Result<()> { In the above example, the `Source::try_for_each` method iterates over each key-value pair in the `Source` and writes them to the terminal. -Let's look at a structured example. Take the following json format: +### Writing key-value pairs as JSON + +Let's look at a structured example. Take the following JSON map: ```json { @@ -266,24 +425,24 @@ Let's look at a structured example. Take the following json format: } ``` -A `Source` can be serialized as a map using `serde`. This requires the `kv_serde` feature: +A `Source` can be serialized as a map using a serialization framework like `serde` or `sval`. Using `serde` for this example requires the `kv_serde` feature: ```toml [dependencies.log] features = ["kv_serde"] ``` -Defining a serializable structure based on a log record for this format could then be done using `serde_derive`, and then written using `serde_json`: +Defining a serializable structure based on a log record for the previous JSON map could then be done using `serde_derive`, and then written using `serde_json`: ```rust -use log::kv::Source; +use log::key_values::Source; fn log_record(w: impl Write, r: &Record) -> io::Result<()> { let r = SerializeRecord { lvl: r.level(), ts: epoch_millis(), msg: r.args().to_string(), - kvs: r.key_values().serialize_as_map(), + kvs: r.key_values().as_map(), }; serde_json::to_writer(w, &r)?; @@ -301,13 +460,13 @@ struct SerializeRecord { } ``` -This time, instead of using the `Source::try_for_each` method, we use the `Source::serialize_as_map` method to get an adapter that will serialize each key-value pair as an entry in a `serde` map. +This time, instead of using the `Source::try_for_each` method, we use the `Source::as_map` method to get an adapter that implements `serde::Serialize` by serializing each key-value pair as an entry in a `serde` map. ## Integrating log frameworks with `log` -The `Source` trait we saw previously describes some container for structured key-value pairs that can be iterated. Other log frameworks that want to integrate with the `log` crate should build `Record`s that contain some implementation of `Source` based on their own structured logging. +The `Source` trait we saw previously describes some container for structured key-value pairs that can be iterated through. Other log frameworks that want to integrate with the `log` crate should build `Record`s that contain some implementation of `Source` based on their own structured logging. -The previous section demonstrated some of the methods available on `Source` like `Source::try_for_each` and `Source::serialize_as_map`. Both of those methods are provided on top of a single required `Source::visit` method. The `Source` trait itself looks something like this: +The previous section demonstrated some of the methods available on `Source` like `Source::try_for_each` and `Source::as_map`. Both of those methods are provided on top of a required lower-level `Source::visit` method, which looks something like this: ```rust trait Source { @@ -325,7 +484,9 @@ trait Visitor<'kvs> { } ``` -As an example, let's say our log framework captures its key-value pairs in a `BTreeMap`: +where `Key` is a container for a string and `Value` is the container for structured data we saw previously. The lifetime `'kvs` is threaded from the original borrow of the `Source` through to the `Key`s and `Value`s that a `Visitor` sees. That allows visitors to work with key-value pairs that can live for longer than a single call to `Visitor::visit_pair`. + +Let's implement a `Source`. As an example, let's say our log framework captures its key-value pairs in a `BTreeMap`: ```rust struct KeyValues { @@ -336,84 +497,23 @@ struct KeyValues { The `Source` trait could be implemented for `KeyValues` like this: ```rust -use log::kv::source::{self, Source}; - -impl Source for KeyValues { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { - self.data.visit(visitor) - } -} -``` - -This implementation is pretty trivial because `BTreeMap` happens to already implement the `Source` trait itself. Let's assume `BTreeMap` didn't implement `Source`. A manual implementation iterating through the map and converting the `(&String, &serde_json::Value)` pairs into types that can be visited could look like this: - -```rust -use log::kv::source::{self, Source}; +use log::key_values::source::{self, Source}; impl Source for KeyValues { fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { for (k, v) in self.data { - visitor.visit_pair(source::Key::new(k), source::Value::new(v)) + visitor.visit_pair(source::Key::from_str(k), source::Value::from_serde(v)) } } } ``` -The `Key::new` method accepts any `T: Borrow`. The `Value::new` accepts [a value that can be logged](#what-can-be-logged). Values that can't satisfy the requirements for logging on their own can still be supported using the `Value::any` method. This method lets us provide an inline function that will visit the value without implementing any traits: - -```rust -use log::kv::{ - source::{self, Source}, - value, -}; - -// We don't want to depend on `serde` -// but still want to be able to log `MyValue`s -#[derive(Debug)] -struct MyValue { .. } - -struct KeyValues { - data: BTreeMap, -} - -impl Source for KeyValues { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { - for (k, v) in self.data { - let key = source::Key::new(k); - - // The `Value::any` method takes a value and a - // function that lets us visit it - let value = source::Value::any(v: &MyValue, |v: &MyValue, visitor: &mut dyn value::Visitor| { - visitor.visit_fmt(format_args!("{:?}", v)) - }); - - visitor.visit_pair(key, value) - } -} -``` - -The `value::Visitor` trait is similar to `serde::Serializer`, but only supports simple structures: - -```rust -trait Visitor { - fn visit_i64(&mut self, v: i64) -> Result<(), Error>; - fn visit_u64(&mut self, v: u64) -> Result<(), Error>; - fn visit_i128(&mut self, v: i128) -> Result<(), Error>; - fn visit_u128(&mut self, v: u128) -> Result<(), Error>; - fn visit_f64(&mut self, v: f64) -> Result<(), Error>; - fn visit_bool(&mut self, v: bool) -> Result<(), Error>; - fn visit_char(&mut self, v: char) -> Result<(), Error>; - fn visit_str(&mut self, v: &str) -> Result<(), Error>; - fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error>; - fn visit_none(&mut self) -> Result<(), Error>; - fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error>' -} -``` +The `Key::from_str` method accepts any `T: Borrow`. The `Value::from_serde` accepts any `T: serde::Serialize + Debug`. A `Source` doesn't have to just contain key-value pairs directly like `BTreeMap` though. It could act like an adapter that changes its pairs before emitting them, like we have for iterators in the standard library. As another example, the following `Source` doesn't store any key-value pairs of its own, instead it will sort and de-duplicate pairs read from another source by first reading them into a map before forwarding them on: ```rust -use log::kv::source::{self, Source, Visitor}; +use log::key_values::source::{self, Source, Visitor}; pub struct SortRetainLast(KVS); @@ -450,32 +550,17 @@ where } ``` -## Writing your own `value::Visitor` +## How producers and consumers of structured values interact -Consumers of key-value pairs can visit structured values without needing a `serde::Serializer`, or needing to depend on `serde` at all. Instead they can implement a `value::Visitor`. A `Visitor` can always visit any structured value by formatting it using its `Debug` implementation: +The previous sections demonstrated some of the APIs for capturing and consuming structured data on log records. The `ToValue` trait and `Value::from_any` methods capture values into a common `Value` container. The `Source` trait allows these `Value`s to be consumed using `std::fmt`, `sval` or `serde`. -```rust -use log::kv::value::{self, Value}; - -struct WriteVisitor(W); - -impl value::Visitor for WriteVisitor -where - W: Write, -{ - fn visit_any(&mut self, v: Value) -> Result<(), value::Error> { - write!(&mut self.0, "{:?}", v)?; - - Ok(()) - } -} -``` - -There are other methods besides `visit_any` that can be implemented. By default they all forward to `visit_any`. +Values captured from any one supported framework can be represented by any other. That means a value can be captured in terms of `sval` and consumed in terms of `serde`, with its underlying structure retained. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation +This section details the nuts-and-bolts of the structured logging API. + ## Design considerations ### Don't break anything @@ -484,59 +569,38 @@ Allow structured logging to be added in the current `0.4.x` series of `log`. Thi ### Don't create a public dependency -Don't create a new serialization framework that causes `log` to become a public dependency of any library that wants their data types to be loggable. Logging is a truly cross-cutting concern, so if `log` was a public dependency it would probably be at least as pervasive as `serde` is now. +Don't create a new serialization API that requires `log` to become a public dependency of any library that wants their data types to be logged. Logging is a truly cross-cutting concern, so if `log` was a public dependency it would become even more difficult to develop its API without compounding churn. Any traits that are expected to be publicly implemented should be narrowly scoped to make backwards compatibility easier. ### Support arbitrary producers and arbitrary consumers -Provide an API that's suitable for two independent logging frameworks to integrate through if they want. - -### Prioritize end-users of `log!` - -There are far more consumers of the `log!` macros that don't need to worry about the internals of the `log` crate than there are log frameworks and sinks that do so it makes sense to prioritize `log!` ergonomics. +Provide an API that's suitable for two independent logging frameworks to integrate through if they want. Producers of structured data and consumers of structured data should be able to use different serialization frameworks opaquely and still get good results. As an example, a caller of `info!` should be able to log a map that implements `sval::Value`, and the implementor of the receiving `Log` trait should be able to format that map using `serde::Serialize`. ### Object safety `log` is already designed to be object-safe so this new structured logging API needs to be object-safe too. +### Enable the next round of `log` development + +Once structured logging is available, there will be a lot of new ways to hold `log` and new concepts to try out, such as a procedural-macro-based `log!` implementation and explicit treatment of `std::error::Error`. The APIs introduced by this RFC should enable others to build these features in external crates, and look at integrating that work back into `log` in the future. + ## Cargo features Structured logging will be supported in either `std` or `no_std` contexts by default. ```toml [features] -kv_serde = ["std", "serde", "erased-serde"] -i128 = [] +std = [] +kv_sval = ["sval"] +kv_serde = ["std", "serde", "erased-serde", "sval"] ``` -### `kv_serde` +### `kv_sval` and `kv_serde` -Using default features, structured logging will be supported by `log` in `no_std` environments for a fixed set of types from the standard library. Using the `kv_serde` feature, any type that implements `Debug + Serialize` can be logged, and its potentially complex structure will be retained. +Using default features, implementors of the `Log` trait will be able to format structured data (in the form of `Value`s) using the `std::fmt` machinery. -### `i128` +`sval` is a new serialization framework that's specifically designed with structured logging in mind. It's `no_std` and object-safe, but isn't stable and requires `rustc` `1.31.0`. Using the `kv_sval` feature, any `Value` will also implement `sval::Value` so its underlying structure will be visible to consumers of structured data using `sval::Stream`s. -Add support for 128bit numbers without bumping `log`'s current minimally supported version of `rustc`. - -## Implications for dependents - -Dependents of `log` will notice the following: - -### Default crate features - -The API that's available with default features doesn't add any extra dependencies to the `log` crate, and shouldn't impact compile times or artifact size much. - -### After opting in to `kv_serde` - -In `no_std` environments (which is the default for `log`): - -- `serde` will enter the `Cargo.lock` if it wasn't there already. This will impact compile-times. -- Artifact size of `log` will increase. - -In `std` environments (which is common when using `env_logger` and other crates that implement `Log`): - -- `serde` and `erased-serde` will enter the `Cargo.lock` if it wasn't there already. This will impact compile-times. -- Artifact size of `log` will increase. - -In either case, `serde` will become a public dependency of the `log` crate, so any breaking changes to `serde` will result in breaking changes to `log`. +Using the `kv_serde` feature, any `Value` will also implement `serde::Serialize` so its underlying structure will be visible to consumers of structured data using `serde::Serializer`s. ## Key-values API @@ -547,791 +611,646 @@ Just about the only things you can do with a structured value are format it or s ```rust pub struct Error(Inner); -enum Inner { - Static(&'static str), - #[cfg(feature = "std")] - Owned(String), -} - impl Error { pub fn msg(msg: &'static str) -> Self { Error(Inner::Static(msg)) } +} - #[cfg(feature = "std")] - pub fn as_error(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { - &self.0 +impl Debug for Error { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + self.0.fmt(f) } +} - #[cfg(feature = "std")] - pub fn into_error(self) -> Box { - Box::new(self.0) +impl Display for Error { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + self.0.fmt(f) } +} - #[cfg(feature = "kv_serde")] - pub fn into_serde(self) -> E - where - E: serde::ser::Error, - { - E::custom(self) - } +enum Inner { + Static(&'static str), + #[cfg(feature = "std")] + Owned(String), } -#[cfg(feature = "std")] -impl From for Error -where - E: std::error::Error, -{ - fn from(err: E) -> Self { - Error(Inner::Owned(err.to_string())) +impl Debug for Inner { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self { + Inner::Static(msg) => msg.fmt(f), + #[cfg(feature = "std")] + Inner::Owned(ref msg) => msg.fmt(f), + } } } -#[cfg(feature = "std")] -impl From for Box { - fn from(err: Error) -> Self { - err.into_error() +impl Display for Inner { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self { + Inner::Static(msg) => msg.fmt(f), + #[cfg(feature = "std")] + Inner::Owned(ref msg) => msg.fmt(f), + } } } -#[cfg(feature = "std")] -impl From for io::Error { - fn from(err: Error) -> Self { +impl From for Error { + #[cfg(feature = "std")] + fn from(err: fmt::Error) -> Self { + Self::custom(err) + } + #[cfg(not(feature = "std"))] + fn from(_: fmt::Error) -> Self { + Self::msg("formatting failed") } } -impl AsRef for Error { - fn as_ref(&self) -> &(dyn std::error::Error + Send + Sync + 'static) { - self.as_error() +impl From for fmt::Error { + fn from(_: Error) -> Self { + Self } } -#[cfg(feature = "std")] -impl std::error::Error for Inner { - fn description(&self) -> &str { - match self { - Inner::Static(msg) => msg, - Inner::Owned(msg) => msg, +#[cfg(feature = "kv_sval")] +mod sval_support { + use super::*; + + impl From for Error { + fn from(err: sval::Error) -> Self { + Self::from_sval(err) } } -} -``` -There's no really universal way to handle errors in a logging pipeline. Knowing that some error occurred, and knowing where, should be enough for implementations of `Log` to decide how to handle it. The `Error` type doesn't try to be a general-purpose error management tool, it tries to make it easy to early-return with other errors. + #[cfg(not(feature = "std"))] + impl Error { + pub(crate) fn from_sval(err: sval::Error) -> Self { + Error::msg("sval streaming failed") + } -To make it possible to carry any arbitrary `S::Error` type, where we don't know how long the value can live for and whether it's `Send` or `Sync`, without extra work, the `Error` type does not attempt to store the error value itself. It just converts it into a `String`. + pub fn into_sval(self) -> sval::Error { + sval::Error::msg("streaming failed") + } + } -### `value::Visit` + #[cfg(feature = "std")] + impl Error { + pub(crate) fn from_sval(err: sval::Error) -> Self { + Error::custom(err) + } -The `Visit` trait can be treated like a lightweight subset of `serde::Serialize` that can interoperate with `serde`, without necessarily depending on it. It can't be implemented manually: + pub fn into_sval(self) -> sval::Error { + self.into() + } + } +} -```rust -/// A type that can be converted into a borrowed value. -pub trait Visit: private::Sealed { - /// Visit this value. - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; +#[cfg(feature = "kv_serde")] +mod serde_support { + impl Error { + /// Convert into `serde`. + pub fn into_serde(self) -> E + where + E: serde::ser::Error, + { + E::custom(self) + } + } - /// Convert a reference to this value into an erased `Value`. - fn to_value(&self) -> Value - where - Self: Sized, - { - Value::new(self) + impl Error { + #[cfg(not(feature = "std"))] + pub(crate) fn from_serde(err: impl serde::ser::Error) -> Self { + Self::msg("serde serialization failed") + } + + #[cfg(feature = "std")] + pub(crate) fn from_serde(err: impl serde::ser::Error) -> Self { + Self::custom(err) + } } } -mod private { - #[cfg(not(feature = "kv_serde"))] - pub trait Sealed: Debug {} - - #[cfg(feature = "kv_serde")] - pub trait Sealed: Debug + Serialize {} -} -``` - -We'll look at the `Visitor` trait in more detail later. +#[cfg(feature = "std")] +mod std_support { + impl Error { + /// Create an error for a formattable value. + pub fn custom(err: impl fmt::Display) -> Self { + Error(Inner::Owned(err.to_string())) + } + } -`Visit` is the trait bound that structured values need to satisfy before they can be logged. The trait can't be implemented outside of the `log` crate, because it uses blanket implementations depending on Cargo features. If a crate defines a datastructure that users might want to log, instead of trying to implement `Visit`, it should implement the `serde::Serialize` and `std::fmt::Debug` traits. It shouldn't need to depend on the `log` crate at all. This means that `Visit` can piggyback off `serde::Serialize` as the pervasive public dependency, so that `Visit` itself doesn't need to be one. + impl From for Error { + fn from(err: io::Error) -> Self { + Error::custom(err) + } + } -The trait bounds on `private::Sealed` ensure that any generic `T: Visit` carries some additional traits that are needed for the blanket implementation of `Serialize`. As an example, any `Option` can also be treated as `Option` and therefore implement `Serialize` itself. The `Visit` trait is responsible for a lot of type system mischief. + impl From for io::Error { + fn from(err: Error) -> Self { + io::Error::new(io::ErrorKind::Other, err) + } + } -With default features, the types that implement `Visit` are a subset of `T: Debug + Serialize`: + impl error::Error for Error { + fn description(&self) -> &str { + self.0.description() + } -- Standard formats: `Arguments` -- Primitives: `bool`, `char` -- Unsigned integers: `u8`, `u16`, `u32`, `u64`, `u128` -- Signed integers: `i8`, `i16`, `i32`, `i64`, `i128` -- Strings: `&str`, `String` -- Bytes: `&[u8]`, `Vec` -- Paths: `&Path`, `PathBuf` -- Special types: `Option`, `&T`, and `()`. + fn cause(&self) -> Option<&dyn error::Error> { + self.0.cause() + } + } -Enabling the `kv_serde` feature expands the set of types that implement `Visit` from this subset to all `T: Debug + Serialize`. - -``` --------- feature = "kv_serde" -------- -| | -| T: Debug + Serialize | -| | -| | -| - not(feature = "kv_serde") - | -| | | | -| | u8, u16, u32, u64, u128 | | -| | i8, i16, i32, i64, i128 | | -| | bool, char, &str, String | | -| | &[u8], Vec | | -| | &Path, PathBuf, Arguments | | -| | Option, &T, () | | -| | | | -| ----------------------------- | -| | -| | --------------------------------------- + impl error::Error for Inner { + fn description(&self) -> &str { + match self { + Inner::Static(msg) => msg, + Inner::Owned(msg) => msg, + } + } + } +} ``` -#### Object safety +There's no really universal way to handle errors in a logging pipeline. Knowing that some error occurred, and knowing where, should be enough for implementations of `Log` to decide how to handle it. The `Error` type doesn't try to be a general-purpose error management tool, it tries to make it easy to early-return with other errors. -The `Visit` trait is not object-safe, but has a simple object-safe wrapper used by `Value`. We'll look at `Value` in more detail later. +To make it possible to carry any arbitrary `S::Error` type, where we don't know how long the value can live for and whether it's `Send` or `Sync`, without extra work, the `Error` type does not attempt to store the error value itself. It just converts it into a `String`. -#### `Visit` without the `kv_serde` feature +### `Value` -Without the `kv_serde` feature, the `Visit` trait is implemented for a fixed set of fundamental types from the standard library: +A `Value` is an erased container for some type whose structure can be visited, with a potentially short-lived lifetime: ```rust -impl Visit for u8 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u64(*self as u64) - } -} +pub struct Value<'v>(Inner<'v>); -impl Visit for u16 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u64(*self as u64) +impl<'v> Value<'v> { + pub fn from_any(v: &'v T, from: FromAnyFn) -> Self { + Value(Inner::new(v, from)) } } +``` -impl Visit for u32 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u64(*self as u64) - } -} +The inner type is an erased reference to some data and a function that captures its structure: -impl Visit for u64 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u64(*self) - } +```rust +struct Void { + _priv: (), + _oibit_remover: PhantomData<*mut dyn Fn()>, } -impl Visit for i8 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_i64(*self as i64) - } +#[derive(Clone, Copy)] +struct Inner<'a> { + data: &'a Void, + from: FromAnyFn, } -impl Visit for i16 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_i64(*self as i64) - } -} +type FromAnyFn = fn(FromAny, &T) -> Result<(), Error>; -impl Visit for i32 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_i64(*self as i64) +impl<'a> Inner<'a> { + fn new(data: &'a T, from: FromAnyFn) -> Self { + unsafe { + Inner { + data: mem::transmute::<&'a T, &'a Void>(data), + from: mem::transmute::, FromAnyFn>(from), + } + } } -} -impl Visit for i64 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_i64(*self) + fn visit(&self, backend: &mut dyn Backend) -> Result<(), Error> { + (self.from)(FromAny(backend), self.data) } } +``` + +The `FromAny` type is like a visitor that accepts values with a particular structure, but doesn't require those values satisfy any lifetime constraints: + +```rust +/// A builder for a value. +/// +/// An instance of this type is passed to the `Value::from_any` method. +pub struct FromAny<'a>(&'a mut dyn Backend); -impl Visit for f32 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_f64(*self as f64) +// NOTE: These methods aren't public. They're used by implementations of +// ToValue for standard library types. +impl<'a> FromAny<'a> { + fn value(self, v: Value) -> Result<(), Error> { + v.0.visit(self.0) } -} -impl Visit for f64 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_f64(*self) + fn u64(self, v: u64) -> Result<(), Error> { + self.0.u64(v) } -} -impl Visit for char { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_char(*self) + fn i64(self, v: i64) -> Result<(), Error> { + self.0.i64(v) + } + + fn f64(self, v: f64) -> Result<(), Error> { + self.0.f64(v) } -} -impl Visit for bool { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_bool(*self) + fn bool(self, v: bool) -> Result<(), Error> { + self.0.bool(v) } -} -impl Visit for () { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_none() + fn char(self, v: char) -> Result<(), Error> { + self.0.char(v) } -} -#[cfg(feature = "i128")] -impl Visit for u128 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u128(*self) + fn none(self) -> Result<(), Error> { + self.0.none() } -} -#[cfg(feature = "i128")] -impl Visit for i128 { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_i128(*self) + fn str(self, v: &str) -> Result<(), Error> { + self.0.str(v) } } +``` -impl Visit for Option -where - T: Visit, -{ - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - match self { - Some(v) => v.visit(visitor), - None => visitor.visit_none(), +Internal implementations of `ToValue` for standard library primitives use the private methods on `Backend` to retain their structure, without having to commit to any machinery in the public API. Each serialization framework supported by `log` provides an internal implementation of `Backend`, so there's one for `std::fmt`, one for `sval`, and one for `serde`. + +Each backend adds constructor methods to `Value` that allows it to capture values that satisfy the traits expected from that backend: + +```rust +mod fmt { + impl<'v> Value<'v> { + pub fn from_debug(v: &'v impl Debug) -> Self { + Self::from_any(v, |from, v| from.debug(v)) } } -} -impl<'a> Visit for fmt::Arguments<'a> { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_fmt(self) + impl<'a> FromAny<'a> { + pub fn debug(self, v: impl Debug) -> Result<(), Error> { + .. + } } -} -impl<'a> Visit for &'a str { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_str(self) + impl<'v> fmt::Debug for Value<'v> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + .. + } } -} -impl<'a> Visit for &'a [u8] { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_bytes(self) + impl<'v> fmt::Display for Value<'v> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + .. + } } } -impl<'v> Visit for Value<'v> { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - self.visit(visitor) +#[cfg(feature = "kv_sval")] +mod sval { + impl<'v> Value<'v> { + pub fn from_sval(v: &'v (impl sval::Value + Debug)) -> Self { + Self::from_any(v, |from, v| from.sval(v)) + } } -} -#[cfg(feature = "std")] -impl Visit for Box -where - T: Visit -{ - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - (**self).visit(visitor) + impl<'a> FromAny<'a> { + pub fn sval(self, v: impl sval::Value + Debug) -> Result<(), Error> { + .. + } } -} -#[cfg(feature = "std")] -impl<'a> Visit for &'a Path { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - match self.to_str() { - Some(s) => visitor.visit_str(s), - None => visitor.visit_fmt(&format_args!("{:?}", self)), + impl<'v> sval::Value for Value<'v> { + fn stream(&self, stream: &mut sval::value::Stream) -> Result<(), sval::Error> { + .. } } } -#[cfg(feature = "std")] -impl Visit for String { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_str(&*self) +#[cfg(feature = "kv_serde")] +mod serde { + impl<'v> Value<'v> { + pub fn from_serde(v: &'v (impl Serialize + Debug)) -> Self { + Self::from_any(v, |from, v| from.serde(v)) + } } -} -#[cfg(feature = "std")] -impl Visit for Vec { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_bytes(&*self) + impl<'a> FromAny<'a> { + pub fn serde(self, v: impl Serialize + Debug) -> Result<(), Error> { + .. + } } -} -#[cfg(feature = "std")] -impl Visit for PathBuf { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - self.as_path().visit(visitor) + impl<'v> serde::Serialize for Value<'v> { + fn serialize(&self, serializer: S) -> Result + where + S: serde::Serializer, + { + .. + } } } ``` -#### `Visit` with the `kv_serde` feature +The implementation details of `Backend` aren't really important because they aren't public and could be implemented in a couple of ways, but a working reference is provided in the appendix. -With the `kv_serde` feature, the `Visit` trait is implemented for any type that is `Debug + Serialize`: +#### A minimal initial API + +An initial implementation of `Value` could support just the `std::fmt` machinery: ```rust -#[cfg(feature = "kv_serde")] -impl Visit for T -where - T: Debug + Serialize {} +pub struct Value<'v>(_); + +impl<'v> Value<'v> { + pub fn from_debug(v: &'v impl Debug) -> Self { + .. + } +} + +impl<'v> Debug for Value<'v> { + .. +} + +impl<'v> Display for Value<'v> { + .. +} ``` -#### Ensuring the fixed set is a subset of the blanket implementation +Structured serialization frameworks could then be introduced without breakage. This could either be done in terms of the `FromAny` machinery shown previously, by exposing a serialization contract directly, or both. -Changing trait implementations based on Cargo features is a dangerous game. Cargo features are additive, so any observable changes to trait implementations must also be purely additive, otherwise you can end up with libraries that can't compile if a feature is active. This can be very subtle when references and generics are involved. +#### Adding another supported framework -When the `kv_serde` feature is active, the implementaiton of `Visit` changes from a fixed set to an open one. We have to guarantee that the open set is a superset of the fixed one. That means any valid `T: Visit` without the `kv_serde` feature remains a valid `T: Visit` with the `kv_serde` feature. +Adding optional support for another serialization framework like `serde` or `sval` can be done by implementing the `serde::Serialize` or `sval::Value` traits, and adding constructor methods to `Value` that allow it to capture implementations of those traits. A side-effect of pushing all supported serialization frameworks through the one type is that all supported frameworks will have to provide bridging support for all other supported frameworks. This makes the barrier for new frameworks raise exponentially and discourages supporting too many, but also protects the end-user experience from degrading when multiple frameworks are used in a single logging pipeline. -There are a few ways we could achieve this, depending on the quality of the docs we want to produce. +The `sval` library is designed as a compatible extension for structured logging, and might be the first serialization framework to consider supporting (it comes along with `serde` support). -For more readable documentation at the risk of incorrectly implementing `Visit`, we can use a private trait like `EnsureVisit: Visit` that is implemented alongside the concrete `Visit` trait regardless of any blanket implementations of `Visit`: +#### Ownership -```rust -// The blanket implemention of `Visit` when `kv_serde` is enabled -#[cfg(feature = "kv_serde")] -impl Visit for T where T: Debug + Serialize {} +The `Value` type borrows from its inner value. -/// This trait is a private implementation detail for testing. -/// -/// All it does is make sure that our set of concrete types -/// that implement `Visit` always implement the `Visit` trait, -/// regardless of crate features and blanket implementations. -trait EnsureVisit: Visit {} +#### Thread-safety -// Ensure any reference to a `Visit` implements `Visit` -impl<'a, T> EnsureVisit for &'a T where T: Visit {} +The `Value` type doesn't try to guarantee that values are `Send` or `Sync`, and doesn't offer any way of retaining that information when erasing. -// These impl blocks always exists -impl EnsureVisit for Option where T: Visit {} -// This impl block only exists if the `kv_serde` isn't active -#[cfg(not(feature = "kv_serde"))] -impl private::Sealed for Option where T: Visit {} -#[cfg(not(feature = "kv_serde"))] -impl Visit for Option where T: Visit { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { +### `ToValue` - } +The `ToValue` trait represents a type that can be converted into a `Value`: + +```rust +pub trait ToValue { + fn to_value(&self) -> Value; } ``` -In the above example, we can ensure that `Option` always implements the `Visit` trait, whether it's done manually or as part of a blanket implementation. All types that implement `Visit` manually with any `#[cfg]` _must_ also always implement `EnsureVisit` manually (with no `#[cfg]`) with the exact same type bounds. It's pretty subtle, but the subtlety can be localized to a single module within the `log` crate so it can be managed. +It's the trait bound that values passed as structured data to the `log!` macros need to satisfy. -Using a trait for this type checking means the `impl Visit for Option` and `impl EnsureVisit for Option` can be wrapped up in a macro so that we never miss adding them. The below macro is an example of a (not very pretty) one that can add the needed implementations of `EnsureVisit` along with the regular `Visit`: +#### Implementors -```rust -macro_rules! impl_to_value { - () => {}; - ( - impl: { $($params:tt)* } - where: { $($where:tt)* } - $ty:ty: { $($serialize:tt)* } - $($rest:tt)* - ) => { - impl<$($params)*> EnsureVisit for $ty - where - $($where)* {} - - #[cfg(not(feature = "kv_serde"))] - impl<$($params)*> private::Sealed for $ty - where - $($where)* {} +`ToValue` is implemented for fundamental primitive types from the standard library: - #[cfg(not(feature = "kv_serde"))] - impl<$($params)*> Visit for $ty - where - $($where)* - { - $($serialize)* - } - - impl_to_value!($($rest)*); - }; - ( - impl: { $($params:tt)* } - $ty:ty: { $($serialize:tt)* } - $($rest:tt)* - ) => { - impl_to_value! { - impl: {$($params)*} where: {} $ty: { $($serialize)* } $($rest)* - } - }; - ( - $ty:ty: { $($serialize:tt)* } - $($rest:tt)* - ) => { - impl_to_value! { - impl: {} where: {} $ty: { $($serialize)* } $($rest)* - } +```rust +impl<'v> ToValue for Value<'v> { + fn to_value(&self) -> Value { + Value(self.0) } } -// Ensure any reference to a `Visit` is also `Visit` -impl<'a, T> EnsureVisit for &'a T where T: Visit {} - -impl_to_value! { - u8: { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - visitor.visit_u64(*self as u64) - } +impl<'a, T> ToValue for &'a T +where + T: ToValue, +{ + fn to_value(&self) -> Value { + (**self).to_value() } +} - impl: { T: Visit } Option: { - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - match self { - Some(v) => v.to_value().visit(visitor), - None => visitor.visit_none(), - } - } +impl ToValue for () { + fn to_value(&self) -> Value { + Value::from_any(self, |from, _| from.none()) } - - ... } -``` - -We don't necessarily need a macro to make new implementations accessible for new contributors safely though. -##### What about specialization? - -In a future Rust with specialization we might be able to avoid all the machinery needed to keep the manual impls consistent with the blanket one, and allow consumers to implement `Visit` without needing `serde`. The specifics of specialization are still up in the air though. Under the proposed _always applicable_ rule, manual implementations like `impl Visit for Option where T: Visit` wouldn't be allowed. The ` where specialize(T: Visit)` scheme might make it possible though, although this would probably be a breaking change in any case. - -### `value::Visitor` - -A visitor for a `Visit` that can interogate its structure: - -```rust -/// A serializer for primitive values. -pub trait Visitor { - /// Visit an arbitrary value. - /// - /// Depending on crate features there are a few things - /// you can do with a value. You can: - /// - /// - format it using `Debug`. - /// - serialize it using `serde`. - fn visit_any(&mut self, v: Value) -> Result<(), Error>; +impl ToValue for u8 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.u64(*v as u64)) + } +} - /// Visit a signed integer. - fn visit_i64(&mut self, v: i64) -> Result<(), Error> { - self.visit_any(v.to_value()) +impl ToValue for u16 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.u64(*v as u64)) } +} - /// Visit an unsigned integer. - fn visit_u64(&mut self, v: u64) -> Result<(), Error> { - self.visit_any(v.to_value()) +impl ToValue for u32 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.u64(*v as u64)) } +} - /// Visit a 128bit signed integer. - #[cfg(feature = "i128")] - fn visit_i128(&mut self, v: i128) -> Result<(), Error> { - self.visit_any(v.to_value()) +impl ToValue for u64 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.u64(*v)) } +} - /// Visit a 128bit unsigned integer. - #[cfg(feature = "i128")] - fn visit_u128(&mut self, v: u128) -> Result<(), Error> { - self.visit_any(v.to_value()) +impl ToValue for i8 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.i64(*v as i64)) } +} - /// Visit a floating point number. - fn visit_f64(&mut self, v: f64) -> Result<(), Error> { - self.visit_any(v.to_value()) +impl ToValue for i16 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.i64(*v as i64)) } +} - /// Visit a boolean. - fn visit_bool(&mut self, v: bool) -> Result<(), Error> { - self.visit_any(v.to_value()) +impl ToValue for i32 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.i64(*v as i64)) } +} - /// Visit a single character. - fn visit_char(&mut self, v: char) -> Result<(), Error> { - let mut b = [0; 4]; - self.visit_str(&*v.encode_utf8(&mut b)) +impl ToValue for i64 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.i64(*v)) } +} - /// Visit a UTF8 string. - fn visit_str(&mut self, v: &str) -> Result<(), Error> { - self.visit_any((&v).to_value()) +impl ToValue for f32 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.f64(*v as f64)) } +} - /// Visit a raw byte buffer. - fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> { - self.visit_any((&v).to_value()) +impl ToValue for f64 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.f64(*v)) } +} - /// Visit standard arguments. - fn visit_none(&mut self) -> Result<(), Error> { - self.visit_any(().to_value()) +impl ToValue for bool { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.bool(*v)) } +} - /// Visit standard arguments. - fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error> { - self.visit_any(v.to_value()) +impl ToValue for char { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.char(*v)) } } -impl<'a, T: ?Sized> Visitor for &'a mut T +impl ToValue for Option where - T: Visitor { } + T: ToValue, +{ + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| match v { + Some(ref v) => from.value(v.to_value()), + None => from.none(), + }) + } +} + +impl<'a> ToValue for &'a str { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.str(*v)) + } +} ``` -### `Value` +### `Key` -A `Value` is an erased container for a `Visit`, with a potentially short-lived lifetime: +A `Key` is a short-lived structure that can be represented as a UTF-8 string. This might be possible without allocating, or it might require a destination to write into: ```rust -/// The value in a key-value pair. -pub struct Value<'v>(ValueInner<'v>); +pub struct Key<'k>(Inner<'k>); -enum ValueInner<'v> { - Erased(&'v dyn ErasedVisit), - Any(Any<'v>), -} - -impl<'v> Value<'v> { - /// Create a value. - pub fn new(v: &'v impl Visit) -> Self { - Value(ValueInner::Erased(v)) +impl<'k> Key<'k> { + /// Create a key from a borrowed string and optional index. + pub fn from_str(key: &'k (impl Borrow + ?Sized)) -> Self { + Key(Inner::Borrowed(key.borrow())) } - /// Create a value from an anonymous type. - /// - /// The value must be provided with a compatible visit method. - pub fn any(v: &'v T, visit: fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self - where - T: 'static, - { - Value(ValueInner::Any(Any::new(v, visit))) - } - - /// Visit the contents of this value with a visitor. - pub fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { + pub fn as_str(&self) -> &str { match self.0 { - ValueInner::Erased(v) => v.erased_visit(visitor), - ValueInner::Any(ref v) => v.visit(visitor), + Inner::Borrowed(k) => k, + #[cfg(feature = "std")] + Inner::Owned(ref k) => &*k, } } } -``` - -#### `ErasedVisit` - -The `ErasedVisit` trait is a private object-safe wrapper for the `Visit` trait. `Visit` itself isn't technically object-safe because it needs the non-object-safe `serde::Serialize` as a supertrait to carry in generic contexts: -```rust -#[cfg(not(feature = "kv_serde"))] -trait ErasedVisit: fmt::Debug { - fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; +impl<'k> AsRef for Key<'k> { + fn as_ref(&self) -> &str { + self.as_str() + } } -#[cfg(feature = "kv_serde")] -trait ErasedVisit: fmt::Debug + erased_serde::Serialize { - fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; +impl<'k> Borrow for Key<'k> { + fn borrow(&self) -> &str { + self.as_str() + } } -impl ErasedVisit for T -where - T: Visit, -{ - fn erased_visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - self.visit(visitor) +impl<'k> From<&'k str> for Key<'k> { + fn from(k: &'k str) -> Self { + Key::from_str(k, None) } } -``` -#### `Any` +impl<'k> PartialEq for Key<'k> { + fn eq(&self, other: &Self) -> bool { + self.as_str().eq(other.as_str()) + } +} -Other logging frameworks that want to integrate with `log` might not want to pull in a `serde` dependency, and so they couldn't implement the `Visit` trait. The `Any` type is a private container that uses some `std::fmt` inspired black-magic to allow values that don't implement the `Visit` trait to be erased in a `Value`. It does this by taking a borrowed value along with a function pointer that looks like `Visit::visit`: +impl<'k> Eq for Key<'k> {} -```rust -struct Void { - _priv: (), - _oibit_remover: PhantomData<*mut dyn Fn()>, +impl<'k> PartialOrd for Key<'k> { + fn partial_cmp(&self, other: &Self) -> Option { + self.as_str().partial_cmp(other.as_str()) + } } -struct Any<'a> { - data: &'a Void, - visit: fn(&Void, &mut dyn Visitor) -> Result<(), Error>, +impl<'k> Ord for Key<'k> { + fn cmp(&self, other: &Self) -> Ordering { + self.as_str().cmp(other.as_str()) + } } -impl<'a> Any<'a> { - fn new(data: &'a T, visit: fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self +impl<'k> Hash for Key<'k> { + fn hash(&self, state: &mut H) where - T: 'static, + H: Hasher, { - unsafe { - Any { - data: mem::transmute::<&'a T, &'a Void>(data), - visit: mem::transmute::< - fn(&T, &mut dyn Visitor) -> Result<(), Error>, - fn(&Void, &mut dyn Visitor) -> Result<(), Error>> - (visit), - } - } - } - - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> { - (self.visit)(self.data, visitor) + self.as_str().hash(state) } } -``` - -There's some scary code in `Any`, which is really just something like an ad-hoc trait object. - -#### Formatting -`Value` always implements `Debug` and `Display` by forwarding to its inner value: - -```rust -impl<'v> fmt::Debug for Value<'v> { +impl<'k> fmt::Debug for Key<'k> { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self.0 { - ValueInner::Erased(v) => v.fmt(f), - ValueInner::Any(ref v) => { - struct ValueFmt<'a, 'b>(&'a mut fmt::Formatter<'b>); - - impl<'a, 'b> Visitor for ValueFmt<'a, 'b> { - fn visit_any(&mut self, v: Value) -> Result<(), Error> { - write!(self.0, "{:?}", v)?; - - Ok(()) - } - } - - let mut visitor = ValueFmt(f); - v.visit(&mut visitor).map_err(|_| fmt::Error) - } - } + self.as_str().fmt(f) } } -impl<'v> fmt::Display for Value<'v> { +impl<'k> fmt::Display for Key<'k> { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "{:?}", self) + self.as_str().fmt(f) } } -``` - -#### Serialization - -When the `kv_serde` feature is enabled, `Value` implements the `serde::Serialize` trait by forwarding to its inner value: - -```rust -impl<'v> Serialize for Value<'v> { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - match self.0 { - ValueInner::Erased(v) => { - erased_serde::serialize(v, serializer) - }, - ValueInner::Any(ref v) => { - struct ErasedVisitSerde { - serializer: Option, - ok: Option, - } - impl Visitor for ErasedVisitSerde - where - S: Serializer, - { - fn visit_any(&mut self, v: Value) -> Result<(), Error> { - let ok = v.serialize(self.serializer.take().expect("missing serializer"))?; - self.ok = Some(ok); - - Ok(()) - } - } - - let mut visitor = ErasedVisitSerde { - serializer: Some(serializer), - ok: None, - }; - - v.visit(&mut visitor).map_err(|e| e.into_serde())?; - Ok(visitor.ok.expect("missing return value")) - }, - } - } +enum Inner<'k> { + Borrowed(&'k str), + #[cfg(feature = "std")] + Owned(String), } -``` - -#### Ownership - -The `Value` type borrows from its inner value. - -#### Thread-safety - -The `Value` type doesn't try to guarantee that values are `Send` or `Sync`, and doesn't offer any way of retaining that information when erasing. -### `Key` - -A `Key` is a short-lived structure that can be represented as a UTF-8 string. This might be possible without allocating, or it might require a destination to write into: - -```rust -/// A key in a key-value pair. -/// -/// The key can be treated like `&str`. -#[derive(PartialEq, Eq, PartialOrd, Ord, Hash)] -pub struct Key<'kvs> { - inner: &'kvs str, -} +#[cfg(feature = "std")] +mod std_support { + use super::*; -impl<'kvs> Borrow for Key<'kvs> { - fn to_key(&self) -> Key { - Key { inner: self.inner } + impl<'k> Key<'k> { + /// Create a key from an owned string and optional index. + pub fn from_owned(key: impl Into) -> Self { + Key(Inner::Owned(key.into())) + } } -} -impl<'kvs> Key<'kvs> { - /// Get a `Key` from a borrowed string. - pub fn from_str(key: &'kvs (impl AsRef + ?Sized)) -> Self { - Key { - inner: key.as_ref(), + impl ToKey for String { + fn to_key(&self) -> Key { + Key::from_str(self, None) } } - /// Get a borrowed string from a `Key`. - pub fn as_str(&self) -> &str { - &self.inner + impl<'k> From for Key<'k> { + fn from(k: String) -> Self { + Key::from_owned(k, None) + } } } -impl<'kvs> AsRef for Key<'kvs> { - fn as_ref(&self) -> &str { - self.as_str() - } -} +#[cfg(feature = "kv_sval")] +mod sval_support { + use super::*; -#[cfg(feature = "std")] -impl<'kvs> Borrow for Key<'kvs> { - fn borrow(&self) -> &str { - self.as_str() - } -} + use sval::value::{self, Value}; -impl<'kvs> Serialize for Key<'kvs> { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - serializer.serialize_str(self.inner) + impl<'k> Value for Key<'k> { + fn stream(&self, stream: &mut value::Stream) -> Result<(), value::Error> { + self.as_str().stream(stream) + } } } -impl<'kvs> Display for Key<'kvs> { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - self.inner.fmt(f) - } -} +#[cfg(feature = "kv_serde")] +mod serde_support { + use super::*; + + use serde::{Serialize, Serializer}; -impl<'kvs> Debug for Key<'kvs> { - fn fmt(&self, f: &mut Formatter) -> fmt::Result { - self.inner.fmt(f) + impl<'k> Serialize for Key<'k> { + fn serialize(&self, serializer: S) -> Result + where + S: Serializer, + { + self.as_str().serialize(serializer) + } } } ``` @@ -1346,92 +1265,59 @@ The `Key` type can either borrow or own its inner value. The `Key` type is probably `Send` + `Sync`, but that's not guaranteed. -### `source::Visitor` - -The `Visitor` trait used by `Source` can visit a single key-value pair: - -```rust -pub trait Visitor<'kvs> { - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; -} - -impl<'a, 'kvs, T: ?Sized> Visitor<'kvs> for &'a mut T -where - T: Visitor<'kvs> { } -``` +#### Extensibility -A `Visitor` may serialize the keys and values as it sees them. It may also do other work, like sorting or de-duplicating them. Operations that involve ordering keys will probably require allocations. +Future enhancements that could be made to `Key` that haven't been otherwise considered by this RFC. -#### Implementors +##### Adding an index to keys -There aren't any public implementors of `Visitor` in the `log` crate. Other crates that use key-value pairs will implement `Visitor`. +The `Key` type could be extended to hold an optional index into a source. This could be used to retrieve a specific key-value pair more efficiently than scanning. -#### Object safety +### `ToKey` -The `Visitor` trait is object-safe. - -### `Source` - -The `Source` trait is a bit like `std::iter::Iterator`. It gives us a way to inspect some arbitrary collection of key-value pairs using an object-safe visitor pattern: +The `ToKey` trait represents a type that can be converted into a `Key`: ```rust -pub trait Source { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error>; +pub trait ToKey { + fn to_key(&self) -> Key; } - -impl<'a, T: ?Sized> Source for &'a T -where - T: Source { } ``` -`Source` doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. - -#### Ownership - -The `Source` trait is probably the point where having some way to convert from a borrowed to an owned variant would make the most sense. +#### Implementors -We could add a method to `Source` that allowed it to be converted into an owned variant with a default implementation: +The `ToKey` trait is implemented for common string containers in the standard library: ```rust -pub trait Source { - fn to_owned(&self) -> OwnedSource { - OwnedSource::serialized(self) +impl<'a, T: ?Sized> ToKey for &'a T +where + T: ToKey, +{ + fn to_key(&self) -> Key { + (**self).to_key() } } -``` - -The `OwnedSource` could then encapsulte some sharable `dyn Source + Send + Sync`: -```rust -#[derive(Clone)] -pub struct OwnedSource(Arc); - -impl OwnedSource { - fn new(impl Into>) -> Self { - OwnedSource(source.into()) +impl ToKey for str { + fn to_key(&self) -> Key { + Key::from_str(self, None) } +} - fn serialize(impl Source) -> Self { - // Serialize the `Source` to something like - // `Vec<(String, OwnedValue)>` - // where `OwnedValue` is like `serde_json::Value` - ... +impl<'k> ToKey for Key<'k> { + fn to_key(&self) -> Key { + Key::from_str(self) } } ``` -Other implementations of `Source` are encouraged to override the `to_owned` method if they could provide a more efficient implementation. As an example, if there's a `Source` that is already wrapped up in an `Arc` then it can implement `to_owned` by just cloning itself. - -#### Adapters +### `Source` -Some useful adapters exist as provided methods on the `Source` trait. They're similar to adapters on the standard `Iterator` trait: +The `Source` trait is a bit like `std::iter::Iterator`. It gives us a way to inspect some arbitrary collection of key-value pairs using an object-safe visitor pattern: ```rust pub trait Source { - ... + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>; - /// Erase this `Source` so it can be used without - /// requiring generic type parameters. fn erase(&self) -> ErasedSource where Self: Sized, @@ -1439,31 +1325,9 @@ pub trait Source { ErasedSource::erased(self) } - /// An adapter to borrow self. - fn by_ref(&self) -> &Self { - self - } - - /// Chain two `Source`s together. - fn chain(self, other: KVS) -> Chained - where - Self: Sized, - { - Chained(self, other) - } - - /// Find the value for a given key. - /// - /// If the key is present multiple times, this method will - /// return the *last* value for the given key. - /// - /// The default implementation will scan all key-value pairs. - /// Implementors are encouraged provide a more efficient version - /// if they can. Standard collections like `BTreeMap` and `HashMap` - /// will do an indexed lookup instead of a scan. fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> where - Q: Borrow, + Q: ToKey, { struct Get<'k, 'v>(Key<'k>, Option>); @@ -1483,14 +1347,24 @@ pub trait Source { visitor.1 } - /// Apply a function to each key-value pair. + fn by_ref(&self) -> &Self { + self + } + + fn chain(self, other: KVS) -> Chained + where + Self: Sized, + { + Chained(self, other) + } + fn try_for_each(self, f: F) -> Result<(), Error> where Self: Sized, F: FnMut(Key, Value) -> Result<(), E>, E: Into, { - struct ForEach(F, std::marker::PhantomData); + struct ForEach(F, PhantomData); impl<'kvs, F, E> Visitor<'kvs> for ForEach where @@ -1502,35 +1376,60 @@ pub trait Source { } } - self.visit(&mut ForEach(f, Default::default())) + let mut for_each = ForEach(f, Default::default()); + self.visit(&mut for_each) } - /// Serialize the key-value pairs as a map. - #[cfg(feature = "kv_serde")] - fn serialize_as_map(self) -> SerializeAsMap + #[cfg(any(feature = "kv_serde", feature = "kv_sval"))] + fn as_map(self) -> AsMap where Self: Sized, { - SerializeAsMap(self) + AsMap(self) } - /// Serialize the key-value pairs as a map. - #[cfg(feature = "kv_serde")] - fn serialize_as_seq(self) -> SerializeAsSeq + #[cfg(any(feature = "kv_serde", feature = "kv_sval"))] + fn as_seq(self) -> AsSeq + where + Self: Sized, + { + AsSeq(self) + } +} +``` + +`Source` doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. + +#### A minimal initial API + +An initial implementation of `Source` could be provided with just the `visit` and `erase` methods: + +```rust +pub trait Source { + /// Serialize the key value pairs. + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>; + + /// Erase this `Source` so it can be used without + /// requiring generic type parameters. + fn erase(&self) -> ErasedSource where Self: Sized, { - SerializeAsSeq(self) + ErasedSource::erased(self) } } ``` +#### Adapters + +Some useful adapters exist as provided methods on the `Source` trait. They're similar to adapters on the standard `Iterator` trait: + - `by_ref` to get a reference to a `Source` within a method chain. - `chain` to concatenate one source with another. This is useful for composing implementations of `Log` together for contextual logging. - `get` to try find the value associated with a key. - `try_for_each` to try execute some closure over all key-value pairs. This is a convenient way to do something with each key-value pair without having to create and implement a `Visitor`. -- `serialize_as_map` to get a serializable map. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. -- `serialize_as_seq` to get a serializable sequence of tuples. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. +- `as_map` to get a serializable map. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. +- `as_seq` to get a serializable sequence of tuples. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. None of these methods are required for the core API. They're helpful tools for working with key-value pairs with minimal machinery. Even if we don't necessarily include them right away it's worth having an API that can support them later without breakage. @@ -1539,42 +1438,35 @@ None of these methods are required for the core API. They're helpful tools for w `Source` is not object-safe because of the provided adapter methods not being object-safe. The only required method, `visit`, is safe though, so an object-safe version of `Source` that forwards this method can be reasonably written in a similar way to the object-safe `ErasedVisit`: ```rust -/// An erased `Source`. -#[derive(Clone)] +#[derive(Clone, Copy)] pub struct ErasedSource<'a>(&'a dyn ErasedSourceBridge); impl<'a> ErasedSource<'a> { - /// Capture a `Source` and erase its concrete type. - pub fn new(kvs: &'a impl Source) -> Self { + pub fn erased(kvs: &'a impl Source) -> Self { ErasedSource(kvs) } -} -impl<'a> Default for ErasedSource<'a> { - fn default() -> Self { - ErasedSource(&(&[] as &[(&str, &dyn Visit)])) + pub fn empty() -> Self { + ErasedSource(&(&[] as &[(&str, Value)])) } } impl<'a> Source for ErasedSource<'a> { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> { self.0.erased_visit(visitor) } fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> where - Q: Borrow, + Q: ToKey, { - let key = key.to_key(); - self.0.erased_get(key.as_ref()) + self.0.erased_get(key.to_key()) } } -/// A trait that erases a `Source` so it can be stored -/// in a `Record` without requiring any generic parameters. trait ErasedSourceBridge { fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; - fn erased_get<'kvs>(&'kvs self, key: &str) -> Option>; + fn erased_get<'kvs>(&'kvs self, key: Key) -> Option>; } impl ErasedSourceBridge for KVS @@ -1585,7 +1477,7 @@ where self.visit(visitor) } - fn erased_get<'kvs>(&'kvs self, key: &str) -> Option> { + fn erased_get<'kvs>(&'kvs self, key: Key) -> Option> { self.get(key) } } @@ -1598,10 +1490,11 @@ A `Source` containing a single key-value pair is implemented for a tuple of a ke ```rust impl Source for (K, V) where - K: Borrow, - V: Visit, + K: ToKey, + V: ToValue, { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> + { visitor.visit_pair(self.0.to_key(), self.1.to_value()) } } @@ -1611,9 +1504,9 @@ A `Source` with multiple pairs is implemented for arrays of `Source`s: ```rust impl Source for [KVS] where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> { for kv in self { - kv.visit(&mut visitor)?; + kv.visit(visitor)?; } Ok(()) @@ -1624,43 +1517,39 @@ impl Source for [KVS] where KVS: Source { When `std` is available, `Source` is implemented for some standard collections too: ```rust -#[cfg(feature = "std")] impl Source for Box where KVS: Source { fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { (**self).visit(visitor) } } -#[cfg(feature = "std")] impl Source for Arc where KVS: Source { fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { (**self).visit(visitor) } } -#[cfg(feature = "std")] impl Source for Rc where KVS: Source { fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { (**self).visit(visitor) } } -#[cfg(feature = "std")] impl Source for Vec where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { self.as_slice().visit(visitor) } } -#[cfg(feature = "std")] -impl Source for collections::BTreeMap +impl Source for BTreeMap where K: Borrow + Ord, - V: Visit, + V: ToValue, { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> + { for (k, v) in self { - visitor.visit_pair(k.to_key(), v.to_value())?; + visitor.visit_pair(k.borrow().to_key(), v.to_value())?; } Ok(()) @@ -1668,22 +1557,21 @@ where fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> where - Q: Borrow, + Q: ToKey, { - let key = key.to_key(); - collections::BTreeMap::get(self, key.as_ref()).map(Visit::to_value) + BTreeMap::get(self, key.to_key().borrow()).map(|v| v.to_value()) } } -#[cfg(feature = "std")] -impl Source for collections::HashMap +impl Source for HashMap where K: Borrow + Eq + Hash, - V: Visit, + V: ToValue, { - fn visit<'kvs>(&'kvs self, visitor: &mut Visitor<'kvs>) -> Result<(), Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> + { for (k, v) in self { - visitor.visit_pair(k.to_key(), v.to_value())?; + visitor.visit_pair(k.borrow().to_key(), v.to_value())?; } Ok(()) @@ -1691,16 +1579,77 @@ where fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> where - Q: Borrow, + Q: ToKey, { - let key = key.to_key(); - collections::HashMap::get(self, key.as_ref()).map(Visit::to_value) + HashMap::get(self, key.to_key().borrow()).map(|v| v.to_value()) } } ``` The `BTreeMap` and `HashMap` implementations provide more efficient implementations of `Source::get`. +#### Extensibility + +Future enhancements that could be made to `Source` that haven't been otherwise considered by this RFC. + +##### Sending `Source`s between threads + +Before a record could be processed on a background thread it would need to be converted into some owned variant. The `Source` trait is the point where having some way to convert from a borrowed to an owned value would make the most sense because that's where the knowledge of the underlying key-value storage is. + +A new provided method could be added to the `Source` trait that allowed it to be converted into an owned variant that is `Send + Sync + 'static`: + +```rust +pub trait Source { + .. + + fn to_owned(&self) -> OwnedSource { + OwnedSource::collect(self) + } +} + +#[derive(Clone)] +pub struct OwnedSource(Arc); + +impl OwnedSource { + pub fn new(impl Into>) -> Self { + OwnedSource(source.into()) + } + + pub fn collect(impl Source) -> Self { + // Serialize the `Source` to something like + // `Vec<(String, OwnedValue)>` + // where `OwnedValue` is like `serde_json::Value` + .. + } +} +``` + +Other implementations of `Source` would be encouraged to override the `to_owned` method if they could provide a more efficient implementation. As an example, if there's a `Source` that is already wrapped up in an `Arc` then it can implement `to_owned` by just cloning itself. + +### `Visitor` + +The `Visitor` trait used by `Source` can visit a single key-value pair: + +```rust +pub trait Visitor<'kvs> { + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; +} + +impl<'a, 'kvs, T: ?Sized> Visitor<'kvs> for &'a mut T +where + T: Visitor<'kvs> { } +``` + +A `Visitor` may serialize the keys and values as it sees them. It may also do other work, like sorting or de-duplicating them. Operations that involve ordering keys will probably require allocations. + +#### Implementors + +There aren't any public implementors of `Visitor` in the `log` crate. Other crates that use key-value pairs will implement `Visitor`. + +#### Object safety + +The `Visitor` trait is object-safe. + ### `Record` and `RecordBuilder` Structured key-value pairs can be set on a `RecordBuilder`: @@ -1720,7 +1669,7 @@ These key-value pairs can then be accessed on the built `Record`: ```rust #[derive(Clone, Debug)] pub struct Record<'a> { - ... + .. kvs: ErasedSource<'a>, } @@ -1749,7 +1698,7 @@ This RFC proposes an additional semi-colon-separated part of the macro for captu log!( ; ) ``` -The `;` and structured values are optional. If they're not present then the behaviour of the `log!` macro is the same as it is today. +The `;` and structured values are optional. If they're not present then the behavior of the `log!` macro is the same as it is today. As an example, this is what a `log!` statement containing structured key-value pairs could look like: @@ -1764,11 +1713,11 @@ info!( There's a *big* design space around the syntax for capturing log records we could explore, especially when you consider procedural macros. The syntax proposed here for the `log!` macro is not designed to be really ergonomic. It's designed to be *ok*, and to encourage an exploration of the design space by offering a consistent base that other macros could build off. -Having said that, there are a few unintrusive quality-of-life features that make the `log!` macros nicer to use with structured data. +Having said that, there are a few nonintrusive quality-of-life features that make the `log!` macros nicer to use with structured data. ### Expansion -Styructured key-value pairs in the `log!` macro expand to statements that borrow from their environment. +Structured key-value pairs in the `log!` macro expand to statements that borrow from their environment. ```rust info!( @@ -1789,7 +1738,7 @@ Will expand to something like: let correlation &correlation_id; let user = &user; - let kvs: &[(&str, &dyn::key_values::Visit)] = + let kvs: &[(&str, &dyn::key_values::value::ToValue)] = &[("correlation", &correlation), ("user", &user)]; ::__private_api_log( @@ -1812,75 +1761,42 @@ Will expand to something like: Structured logging is a non-trivial feature to support. It adds complexity and overhead to the `log` crate. -## The `Debug + Serialize` blanket implementation of `Visit` - -### Drawbacks - -The main drawbacks of the feature-gated `Debug + Serialize` approach are that it's non-standard, which makes it harder to communicate. +## Internalizing `sval` and `serde` -Making sure the `Visit` trait doesn't drop any implementations when the blanket implementation from `kv_serde` replaces the concrete ones is subtle and nonstandard. We have to be especially careful of references and generics. Any mistakes made here can result in dependencies that become uncompilable depending on Cargo features with no workaround besides removing that impl. Using a macro to define the small fixed set, and keeping all impls local to a single module, could help catch these cases. +Values captured from any one supported framework can be represented by any other. That means a value can be captured in terms of `sval` and consumed in terms of `serde`, with its underlying structure retained. This is done through a one-to-one integration from each framework to each other framework. -Another problem is documentation. It's not really easy to show in `rustdoc` how different crate features change the public API. Making it obvious how the bounds on `Visit` change might be tricky. - -It's also possibly surprising that the way the `Visit` trait is implemented in the ecosystem is through an entirely unrelated combination of `serde` and `std` traits. At least it's surprising on the surface. For libraries that define loggable types, they just implement some standard traits for serialization without involving `log` at all. These are traits they should be considering anyway. For consumers of the `log!` macro, they are mostly going to capture structured values for types they didn't produce, so having `serde` as the answer to _how can I log a `Url`, or a `Uuid`?_ sounds reasonable. It also means libraries defining types like `Url` and `Uuid` don't have yet another public serialization trait to implement. - -If a library provides a datatype that you'd reasonably want to log, but it doesn't implement `serde::Serialize` then adding support for that type isn't just beneficial to you, but to anyone else that might want to serialize that type. - -The degenerate case the `Debug + Serialize` implementation tries to avoid is one where a library needs to implement several very similar serialization-esc traits in order to be loggable in different frameworks: - -```rust -struct Url { .. } - -#[cfg(feature = "serde")] -impl serde::Serialize for Url { .. } - -#[cfg(feature = "log")] -impl log::kv::value::Visit for Url { .. } +### Drawbacks -#[cfg(feature = "slog")] -impl slog::Value for Url { .. } +The one-to-one bridge between serialization frameworks within `log` makes the effort needed to support them increase exponentially with each addition, and discourages it from supporting more than a few. -#[cfg(feature = "log-framework-x")] -impl log_framework_x::Serialize for Url { .. } -``` +It also introduces direct coupling between `log` and these frameworks. For `sval` specifically, this is risky because it's not currently stable. Breaking changes are a possibility. -The real question for `serde` is whether or not depending on it as the general serialization framework in `log` creates the potential for some kind of ecosystem dichotomy if an alternative framework becomes popular where half the ecosystem uses `serde` and the other half uses something else that's incompatible. In that case `log` might not reasonably be able to support both without breakage if it goes down this path. The options for mitigating this in the design now is by either require all loggable types implement `Visit` explicitly, or just requiring callers opt in to `serde` support at the callsite in `log!`. +The mechanism suggested in this RFC for erasing values in `Value::from_any` relies on unsafe code. It's the same as what's used in `std::fmt`, but that machinery isn't directly exposed to callers outside of unstable features. ### Alternatives -#### Require all loggable types implement `Visit` - -We could entirely punt on `serde` and the non-standard implementations of `Visit`, and just provide an API for values that manually implement the `Visit` trait. That avoids the potential serialization dichotomy in `log` altogether. - -The problem here is that any pervasive public API has the chance to create rifts in the ecosystem. By creating a new fundamental API for logging via the `Visit` trait we're just expanding the potential for dichotomies. It also makes `log` another participant in the grab-bag of serialization traits that types in the ecosystem need to implement. - -It also means we need to re-invent `serde`'s support for complex datastructures, the datatypes that implement its traits, and the formats that support it. We'll effectively turn `log` into a serialization framework of its own, and have to introduce arbitrary limitations on the kinds of values that can be logged. - -#### Attempt to deprecate other log-specific serialization traits in favour of `Visit` - -The case where types in the ecosystem like `Url` need to implement a random grab-bag of serialization traits to be compatible with every framework could be avoided by encouraging other frameworks to use `log`'s `Visit` instead of defining their own. We might get to a point where frameworks decide to standardize on a particular API, but that decision should be made naturally instead of being forced onto the ecosystem. The approach this RFC takes makes it possible to standardise, but doesn't depend on it. This also doesn't solve the issue of making `log` into a pervasive public dependency in the first place. - -#### Require callers opt in to `serde` support at the callsite - -We could avoid a potential serialization dichotomy by requiring callers opt in to `serde` support. That way if a new framework came along it could be naturally supported in the same way. There are a few ways callers could opt in to `serde` in the `log!` macros. The specifics aren't really important, but it could look something like this: +Instead of internalizing a few serialization frameworks, `log` could provide a public common contract for them to conform to: ```rust -use log::log_serde; +// Instead of `Value::from_any` + `FromAny` -info!("A message"; user = log_serde!(user)); -``` - -That way an alternative framework could be supported as: +pub trait Visit { + fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; +} -```rust -use log::log_other_framework; +pub trait Visitor { + fn u64(&mut self, v: u64) -> Result<(), Error>; + fn i64(&mut self, v: i64) -> Result<(), Error>; -info!("A message"; user = log_other_framework!(user)); + .. +} ``` -The problem with this approach is that it puts extra barriers in front of users that want to log. Instead of enabling crate features once and then logging structured values, each log statement needs to know how it can capture values. It also passes the burden of dealing with dichotomies onto every consumer of `log`. It seems like a reasonable idea from the perspective of the `log` crate, but is more hostile to end-users. +This is fairly simple for primitive types like integers and strings, but becomes much more involved when dealing with complex values likes maps and sequences. A serialization framework needs to do more than just provide a contract, its API needs to work to support implementations on either side of that contract. Maintaining a useful serialization framework is a distraction for `log`. That's why the `sval` library was created; to manage the necessary complexity of building a serialization framework that's suitable for structured logging externally from the `log` crate. -There are substantially more end-users of the `log` crate calling the `log!` macros than there are frameworks and sinks that need to interact with its API so it's worth prioritizing end-user experience. Anything that requires end-users to opt-in to the most common scenarios isn't ideal. +So the public common serialization contract in `log` is effectively to integrate with one of a few fundamental frameworks. + +Within the `log` crate, internalizing fundamental serialization frameworks reduces the effort needed from building a complete framework down to shimming an existing framework. The effort of managing breaking changes in supported serialization frameworks isn't less than the effort of managing breaking changes in a common contract provided by `log`. The owner of that contract, whether it's `log` or `serde` or `sval`, has to consider the churn introduced by breakage. Serialization of structured values is a complex, necessary, but not primary feature of `log`, so if it should avoid owning that contract and the baggage that comes along with it if it can. # Prior art [prior-art]: #prior-art @@ -1899,7 +1815,7 @@ The `logrus` library is a structured logging framework for Go. It uses a similar ## .NET -The C# community has mostly standardised around using message templates for packaging a log message with structured key-value pairs. Instead of logging a rendered message and separate bag of structured data, the log record contains a template that allows key-value pairs to be interpolated from the same bag of structured data. It avoids duplicating the same information multiple times. +The C# community has mostly standardized around using message templates for packaging a log message with structured key-value pairs. Instead of logging a rendered message and separate bag of structured data, the log record contains a template that allows key-value pairs to be interpolated from the same bag of structured data. It avoids duplicating the same information multiple times. Supporting something like message templates in Rust using the `log!` macros would probably require procedural macros. A macro like that could be built on top of the API proposed by this RFC. @@ -1913,332 +1829,359 @@ Supporting something like message templates in Rust using the `log!` macros woul For context, ignoring the `log!` macros, this is roughly the additional public API this RFC proposes to support structured logging: ```rust -impl<'a> RecordBuilder<'a> { - /// Set the key-value pairs on a log record. - pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a>; -} -impl<'a> Record<'a> { - /// Get the key-value pairs. - pub fn key_values(&self) -> ErasedSource; +``` + +## Backends for `Value` - /// Get a builder that's preconfigured from this record. - pub fn to_builder(&self) -> RecordBuilder; +Each supported serialization framework supported by the `Value` type implements an internal `Backend` trait. The exact machinery isn't really important because it's not public. This reference implementation is an internal serialization contract that includes primitive types, and methods for specific frameworks depending on crate features: + +```rust +trait Backend: fmt::Backend + sval::Backend + serde::Backend { + fn u64(&mut self, v: u64) -> Result<(), Error>; + fn i64(&mut self, v: i64) -> Result<(), Error>; + fn f64(&mut self, v: f64) -> Result<(), Error>; + fn bool(&mut self, v: bool) -> Result<(), Error>; + fn char(&mut self, v: char) -> Result<(), Error>; + fn str(&mut self, v: &str) -> Result<(), Error>; + fn none(&mut self) -> Result<(), Error>; } +``` -pub mod kv { - pub mod source { - pub use kv::Error; +#### `std::fmt` backend - /// A source for key-value pairs. - pub trait Source { - /// Serialize the key value pairs. - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; +```rust +mod fmt { + impl<'v> value::Value<'v> { + pub fn from_debug(v: &'v impl fmt::Debug) -> Self { + Self::from_any(v, |from, v| from.debug(v)) + } + } - /// Erase this `Source` so it can be used without - /// requiring generic type parameters. - fn erase(&self) -> ErasedSource - where - Self: Sized {} + impl<'v> fmt::Debug for value::Value<'v> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + self.0.visit(&mut FmtBackend(f)).map_err(|_| fmt::Error) + } + } - /// Find the value for a given key. - /// - /// If the key is present multiple times, this method will - /// return the *last* value for the given key. - fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> - where - Q: Borrow {} + impl<'v> fmt::Display for value::Value<'v> { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!(f, "{:?}", self) + } + } - /// An adapter to borrow self. - fn by_ref(&self) -> &Self {} + impl<'a> value::FromAny<'a> { + pub fn debug(self, v: impl fmt::Debug) -> Result<(), value::Error> { + self.0.debug(&v) + } + } - /// Chain two `Source`s together. - fn chain(self, other: KVS) -> Chain - where - Self: Sized {} + pub(in crate::key_values::value) trait Backend { + fn debug(&mut self, v: &dyn Value) -> Result<(), value::Error>; + } - /// Apply a function to each key-value pair. - fn try_for_each(self, f: F) -> Result<(), Error> - where - Self: Sized, - F: FnMut(Key, Value) -> Result<(), E>, - E: Into {} + pub(in crate::key_values::value) use fmt::Debug as Value; - /// Serialize the key-value pairs as a map. - fn serialize_as_map(self) -> SerializeAsMap - where - Self: Sized {} + struct FmtBackend<'a, 'b>(&'a mut fmt::Formatter<'b>); - /// Serialize the key-value pairs as a sequence of tuples. - fn serialize_as_seq(self) -> SerializeAsSeq - where - Self: Sized {} + impl<'a, 'b> value::Backend for FmtBackend<'a, 'b> { + fn u64(&mut self, v: u64) -> Result<(), value::Error> { + self.debug(&v) } - /// A visitor for a set of key-value pairs. - /// - /// The visitor is driven by an implementation of `Source`. - /// The visitor expects keys and values that satisfy a given lifetime. - pub trait Visitor<'kvs> { - /// Visit a single key-value pair. - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; + fn i64(&mut self, v: i64) -> Result<(), value::Error> { + self.debug(&v) } - /// An erased `Source`. - pub struct ErasedSource<'a> {} + fn f64(&mut self, v: f64) -> Result<(), value::Error> { + self.debug(&v) + } - impl<'a> ErasedSource<'a> { - /// Capture a `Source` and erase its concrete type. - pub fn new(kvs: &'a impl Source) -> Self {} + fn bool(&mut self, v: bool) -> Result<(), value::Error> { + self.debug(&v) } - impl<'a> Clone for ErasedSource<'a> {} - impl<'a> Default for ErasedSource<'a> {} - impl<'a> Source for ErasedSource<'a> {} + fn char(&mut self, v: char) -> Result<(), value::Error> { + self.debug(&v) + } - /// A `Source` adapter that visits key-value pairs - /// in sequence. - /// - /// This is the result of calling `chain` on a `Source`. - pub struct Chain {} + fn none(&mut self) -> Result<(), value::Error> { + self.debug(&Option::None::<()>) + } - impl Source for Chain - where - A: Source, - B: Source {} + fn str(&mut self, v: &str) -> Result<(), value::Error> { + self.debug(&v) + } + } - /// A `Source` adapter that can be serialized as - /// a map using `serde`. - /// - /// This is the result of calling `serialize_as_map` on - /// a `Source`. - pub struct SerializeAsMap {} + impl<'a, 'b> Backend for FmtBackend<'a, 'b> { + fn debug(&mut self, v: &dyn fmt::Debug) -> Result<(), value::Error> { + write!(self.0, "{:?}", v)?; - impl Serialize for SerializeAsMap - where - KVS: Source {} + Ok(()) + } + } - /// A `Source` adapter that can be serialized as - /// a sequence of tuples using `serde`. - /// - /// This is the result of calling `serialize_as_seq` on - /// a `Source`. - pub struct SerializeAsSeq {} + #[cfg(feature = "kv_sval")] + impl<'a, 'b> value::sval::Backend for FmtBackend<'a, 'b> { + fn sval(&mut self, v: &dyn value::sval::Value) -> Result<(), value::Error> { + self.debug(&v) + } + } - impl Serialize for SerializeAsSeq - where - KVS: Source {} + #[cfg(feature = "kv_serde")] + impl<'a, 'b> value::serde::Backend for FmtBackend<'a, 'b> { + fn serde(&mut self, v: &dyn value::serde::Value) -> Result<(), value::Error> { + self.debug(&v) + } + } +} +``` - impl Source for (K, V) - where - K: Borrow, - V: kv::value::Visit {} +The `fmt::Backend` allows any `Value` to be formatted using `std::fmt`, which is exposed to consumers through the `Debug` and `Display` traits. - impl Source for [KVS] - where - KVS: Source {} +#### `sval` backend - #[cfg(feature = "std")] - impl Source for Box where KVS: Source {} - #[cfg(feature = "std")] - impl Source for Arc where KVS: Source {} - #[cfg(feature = "std")] - impl Source for Rc where KVS: Source {} +```rust +mod sval { + #[cfg(feature = "kv_sval")] + mod imp { + impl<'v> value::Value<'v> { + pub fn from_sval(v: &'v (impl sval::Value + fmt::Debug)) -> Self { + Self::from_any(v, |from, v| from.sval(v)) + } + } - #[cfg(feature = "std")] - impl Source for Vec - where - KVS: Source {} + impl<'v> sval::Value for value::Value<'v> { + fn stream(&self, stream: &mut sval::value::Stream) -> Result<(), sval::value::Error> { + self.0.visit(&mut SvalBackend(stream))?; - #[cfg(feature = "std")] - impl Source for BTreeMap - where - K: Borrow + Ord, - V: kv::value::Visit {} + Ok(()) + } + } - #[cfg(feature = "std")] - impl Source for HashMap - where - K: Borrow + Eq + Hash, - V: kv::value::Visit {} + impl<'a> value::FromAny<'a> { + pub fn sval(self, v: (impl sval::Value + fmt::Debug)) -> Result<(), value::Error> { + self.0.sval(&v) + } + } + + pub(in crate::key_values::value) trait Backend { + fn sval(&mut self, v: &dyn Value) -> Result<(), value::Error>; + } - /// The key in a key-value pair. - pub struct Key<'kvs> {} + pub(in crate::key_values::value) trait Value: sval::Value + fmt::Debug {} + impl Value for T where T: sval::Value + fmt::Debug {} - /// The value in a key-value pair. - pub use kv::value::Value; - } + struct SvalBackend<'a, 'b>(&'a mut sval::value::Stream<'b>); + + impl<'a, 'b> SvalBackend<'a, 'b> { + fn any(&mut self, v: impl sval::Value) -> Result<(), value::Error> { + self.0.any(v)?; + + Ok(()) + } + } + + impl<'a, 'b> value::Backend for SvalBackend<'a, 'b> { + fn u64(&mut self, v: u64) -> Result<(), value::Error> { + self.sval(&v) + } + + fn i64(&mut self, v: i64) -> Result<(), value::Error> { + self.sval(&v) + } + + fn f64(&mut self, v: f64) -> Result<(), value::Error> { + self.sval(&v) + } + + fn bool(&mut self, v: bool) -> Result<(), value::Error> { + self.sval(&v) + } - pub mod value { - pub use kv::Error; + fn char(&mut self, v: char) -> Result<(), value::Error> { + self.sval(&v) + } - /// An arbitrary structured value. - pub struct Value<'v> { - /// Create a new borrowed value. - pub fn new(v: &'v impl Visit) -> Self {} + fn none(&mut self) -> Result<(), value::Error> { + self.sval(&Option::None::<()>) + } - /// Create a new borrowed value from an arbitrary type. - pub fn any(&'v T, fn(&T, &mut dyn Visitor) -> Result<(), Error>) -> Self {} + fn str(&mut self, v: &str) -> Result<(), value::Error> { + self.sval(&v) + } + } - /// Visit the value with the given serializer. - pub fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error> {} + impl<'a, 'b> Backend for SvalBackend<'a, 'b> { + fn sval(&mut self, v: &dyn Value) -> Result<(), value::Error> { + self.any(v) + } } - impl<'v> Debug for Value<'v> {} - impl<'v> Display for Value<'v> {} + impl<'a, 'b> value::fmt::Backend for SvalBackend<'a, 'b> { + fn debug(&mut self, v: &dyn value::fmt::Value) -> Result<(), value::Error> { + self.any(format_args!("{:?}", v)) + } + } - /// A serializer for primitive values. - pub trait Visitor { - /// Visit an arbitrary value. - fn visit_any(&mut self, v: Value) -> Result<(), Error>; + #[cfg(feature = "kv_serde")] + impl<'a, 'b> value::serde::Backend for SvalBackend<'a, 'b> { + fn serde(&mut self, v: &dyn value::serde::Value) -> Result<(), value::Error> { + self.any(sval::serde::to_value(v)) + } + } + } - /// Visit a signed integer. - fn visit_i64(&mut self, v: i64) -> Result<(), Error> {} + #[cfg(not(feature = "kv_sval"))] + mod imp { + pub(in crate::key_values::value) trait Backend {} - /// Visit an unsigned integer. - fn visit_u64(&mut self, v: u64) -> Result<(), Error> {} + impl Backend for V where V: value::Backend {} + } - /// Visit a floating point number. - fn visit_f64(&mut self, v: f64) -> Result<(), Error> {} + pub(super) use self::imp::*; +} +``` - /// Visit a boolean. - fn visit_bool(&mut self, v: bool) -> Result<(), Error> {} +#### `serde` backend - /// Visit a single character. - fn visit_char(&mut self, v: char) -> Result<(), Error> {} +```rust +mod serde { + #[cfg(feature = "kv_serde")] + mod imp { + impl<'v> value::Value<'v> { + pub fn from_serde(v: &'v (impl serde::Serialize + fmt::Debug)) -> Self { + Self::from_any(v, |from, v| from.serde(v)) + } + } - /// Visit a UTF8 string. - fn visit_str(&mut self, v: &str) -> Result<(), Error> {} + impl<'v> serde::Serialize for value::Value<'v> { + fn serialize(&self, serializer: S) -> Result + where + S: serde::Serializer, + { + let mut visitor = SerdeBackend { + serializer: Some(serializer), + ok: None, + }; - /// Visit a raw byte buffer. - fn visit_bytes(&mut self, v: &[u8]) -> Result<(), Error> {} + self.0.visit(&mut visitor).map_err(value::Error::into_serde)?; - /// Visit an empty value. - fn visit_none(&mut self) -> Result<(), Error> {} + Ok(visitor.ok.expect("missing return value")) + } + } - /// Visit standard arguments. - fn visit_fmt(&mut self, v: &fmt::Arguments) -> Result<(), Error> {} + impl<'a> value::FromAny<'a> { + pub fn serde(self, v: (impl serde::Serialize + fmt::Debug)) -> Result<(), value::Error> { + self.0.serde(&v) + } } - impl<'a, T: ?Sized> Visitor for &'a mut T - where - T: Visitor {} + pub(in crate::key_values::value) trait Backend { + fn serde(&mut self, v: &dyn Value) -> Result<(), value::Error>; + } - /// Covnert a type into a value. - /// - /// ** This trait can't be implemented manually ** - pub trait Visit: private::Sealed { - /// Visit this value. - fn visit(&self, visitor: &mut dyn Visitor) -> Result<(), Error>; + pub(in crate::key_values::value) trait Value: erased_serde::Serialize + fmt::Debug {} + impl Value for T where T: serde::Serialize + fmt::Debug {} - /// Convert a reference to this value into an erased `Value`. - fn to_value(&self) -> Value + impl<'a> serde::Serialize for &'a dyn Value { + fn serialize(&self, serializer: S) -> Result where - Self: Sized, + S: serde::Serializer, { - Value::new(self) + erased_serde::serialize(*self, serializer) } } - #[cfg(not(feature = "kv_serde"))] - impl Visit for u8 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for u16 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for u32 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for u64 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for u128 {} - - #[cfg(not(feature = "kv_serde"))] - impl Visit for i8 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for i16 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for i32 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for i64 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for i128 {} - - #[cfg(not(feature = "kv_serde"))] - impl Visit for f32 {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for f64 {} - - #[cfg(not(feature = "kv_serde"))] - impl Visit for char {} - #[cfg(not(feature = "kv_serde"))] - impl Visit for bool {} - - #[cfg(not(feature = "kv_serde"))] - impl Visit for Option + struct SerdeBackend where - T: Visit {} + S: serde::Serializer, + { + serializer: Option, + ok: Option, + } - #[cfg(all(not(feature = "kv_serde"), feature = "std"))] - impl Visit for Box + impl SerdeBackend where - T: Visit {} - - #[cfg(not(feature = "kv_serde"))] - impl<'a> Visit for &'a str {} - #[cfg(all(not(feature = "kv_serde"), feature = "std"))] - impl Visit for String {} + S: serde::Serializer, + { + fn serialize(&mut self, v: impl erased_serde::Serialize) -> Result<(), value::Error> { + self.ok = Some(erased_serde::serialize(&v, self.serializer.take().expect("missing serializer")).map_err(value::Error::from_serde)?); - #[cfg(not(feature = "kv_serde"))] - impl<'a> Visit for &'a [u8] {} - #[cfg(all(not(feature = "kv_serde"), feature = "std"))] - impl Visit for Vec {} + Ok(()) + } + } - #[cfg(not(feature = "kv_serde"))] - impl<'a, T> Visit for &'a T + impl value::Backend for SerdeBackend where - T: Visit {} + S: serde::Serializer, + { + fn u64(&mut self, v: u64) -> Result<(), value::Error> { + self.serde(&v) + } - #[cfg(feature = "kv_serde")] - impl Visit for T - where - T: Debug + Serialize {} - } + fn i64(&mut self, v: i64) -> Result<(), value::Error> { + self.serde(&v) + } + + fn f64(&mut self, v: f64) -> Result<(), value::Error> { + self.serde(&v) + } - pub use source::Source; + fn bool(&mut self, v: bool) -> Result<(), value::Error> { + self.serde(&v) + } - /// An error encountered while visiting key-value pairs. - pub struct Error {} + fn char(&mut self, v: char) -> Result<(), value::Error> { + self.serde(&v) + } - impl Error { - /// Create an error from a static message. - pub fn msg(msg: &'static str) -> Self {} + fn none(&mut self) -> Result<(), value::Error> { + self.serde(&Option::None::<()>) + } - /// Get a reference to a standard error. - #[cfg(feature = "std")] - pub fn as_error(&self) -> &(dyn std::error::Error + Send + Sync + 'static) {} + fn str(&mut self, v: &str) -> Result<(), value::Error> { + self.serde(&v) + } + } - /// Convert into a standard error. - #[cfg(feature = "std")] - pub fn into_error(self) -> Box {} + impl Backend for SerdeBackend + where + S: serde::Serializer, + { + fn serde(&mut self, v: &dyn Value) -> Result<(), value::Error> { + self.serialize(v) + } + } - /// Convert into a `serde` error. - #[cfg(feature = "kv_serde")] - pub fn into_serde(self) -> E + impl value::fmt::Backend for SerdeBackend where - E: serde::ser::Error {} - } + S: serde::Serializer, + { + fn debug(&mut self, v: &dyn value::fmt::Value) -> Result<(), value::Error> { + self.serialize(format_args!("{:?}", v)) + } + } - #[cfg(not(feature = "std"))] - impl From for Error {} + #[cfg(feature = "kv_sval")] + impl value::sval::Backend for SerdeBackend + where + S: serde::Serializer, + { + fn sval(&mut self, v: &dyn value::sval::Value) -> Result<(), value::Error> { + self.serialize(sval::serde::to_serialize(v)) + } + } + } - #[cfg(feature = "std")] - impl From for Error - where - E: std::error::Error {} + #[cfg(not(feature = "kv_serde"))] + mod imp { + pub(in crate::key_values::value) trait Backend {} - #[cfg(feature = "std")] - impl From for Box {} + impl Backend for V where V: value::Backend {} + } - #[cfg(feature = "std")] - impl AsRef for Error {} + pub(super) use self::imp::*; } -``` +``` \ No newline at end of file From 87af21a2cfa6659b0423243b70657632792efdb1 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Tue, 29 Jan 2019 15:17:12 +1000 Subject: [PATCH 09/20] remove implementation details in favour of reference impl --- rfcs/0000-structured-logging.md | 854 ++++++-------------------------- 1 file changed, 139 insertions(+), 715 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index c2f2d9e17..f8ac1c9d4 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -18,10 +18,9 @@ The API is heavily inspired by `slog` and `tokio-trace`. - [Logging structured key-value pairs](#logging-structured-key-value-pairs) - [Supporting key-value pairs in `Log` implementations](#supporting-key-value-pairs-in-log-implementations) - [Integrating log frameworks with `log`](#integrating-log-frameworks-with-log) - - [Writing your own `value::Visitor`](#writing-your-own-valuevisitor) + - [How producers and consumers of structured values interact](#how-producers-and-consumers-of-structured-values-interact) - [Reference-level explanation](#reference-level-explanation) - [Design considerations](#design-considerations) - - [Implications for dependents](#implications-for-dependents) - [Cargo features](#cargo-features) - [Key-values API](#key-values-api) - [`Error`](#error) @@ -36,8 +35,6 @@ The API is heavily inspired by `slog` and `tokio-trace`. - [Drawbacks, rationale, and alternatives](#drawbacks-rationale-and-alternatives) - [Prior art](#prior-art) - [Unresolved questions](#unresolved-questions) -- [Appendix](#appendix) - - [Public API](#public-api) # Motivation [motivation]: #motivation @@ -604,6 +601,8 @@ Using the `kv_serde` feature, any `Value` will also implement `serde::Serialize` ## Key-values API +The following section details the public API for structured values in `log`, along with possible future extensions and minimal initial implementations. Actual implementation details are excluded for brevity unless they're particularly noteworthy. See the original comment on the [RFC issue](https://github.com/rust-lang-nursery/log/pull/296#issue-222687727) for a reference implementation. + ### `Error` Just about the only things you can do with a structured value are format it or serialize it. Serialization and writing might fail, so to allow errors to get carried back to callers there needs to be a general error type that they can early return with: @@ -613,95 +612,35 @@ pub struct Error(Inner); impl Error { pub fn msg(msg: &'static str) -> Self { - Error(Inner::Static(msg)) + .. } } impl Debug for Error { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - self.0.fmt(f) - } + .. } impl Display for Error { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - self.0.fmt(f) - } -} - -enum Inner { - Static(&'static str), - #[cfg(feature = "std")] - Owned(String), -} - -impl Debug for Inner { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self { - Inner::Static(msg) => msg.fmt(f), - #[cfg(feature = "std")] - Inner::Owned(ref msg) => msg.fmt(f), - } - } -} - -impl Display for Inner { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - match self { - Inner::Static(msg) => msg.fmt(f), - #[cfg(feature = "std")] - Inner::Owned(ref msg) => msg.fmt(f), - } - } + .. } impl From for Error { - #[cfg(feature = "std")] - fn from(err: fmt::Error) -> Self { - Self::custom(err) - } - - #[cfg(not(feature = "std"))] - fn from(_: fmt::Error) -> Self { - Self::msg("formatting failed") - } + .. } impl From for fmt::Error { - fn from(_: Error) -> Self { - Self - } + .. } #[cfg(feature = "kv_sval")] mod sval_support { - use super::*; - impl From for Error { - fn from(err: sval::Error) -> Self { - Self::from_sval(err) - } - } - - #[cfg(not(feature = "std"))] - impl Error { - pub(crate) fn from_sval(err: sval::Error) -> Self { - Error::msg("sval streaming failed") - } - - pub fn into_sval(self) -> sval::Error { - sval::Error::msg("streaming failed") - } + .. } - #[cfg(feature = "std")] impl Error { - pub(crate) fn from_sval(err: sval::Error) -> Self { - Error::custom(err) - } - pub fn into_sval(self) -> sval::Error { - self.into() + .. } } } @@ -709,24 +648,11 @@ mod sval_support { #[cfg(feature = "kv_serde")] mod serde_support { impl Error { - /// Convert into `serde`. pub fn into_serde(self) -> E where E: serde::ser::Error, { - E::custom(self) - } - } - - impl Error { - #[cfg(not(feature = "std"))] - pub(crate) fn from_serde(err: impl serde::ser::Error) -> Self { - Self::msg("serde serialization failed") - } - - #[cfg(feature = "std")] - pub(crate) fn from_serde(err: impl serde::ser::Error) -> Self { - Self::custom(err) + .. } } } @@ -734,41 +660,21 @@ mod serde_support { #[cfg(feature = "std")] mod std_support { impl Error { - /// Create an error for a formattable value. pub fn custom(err: impl fmt::Display) -> Self { - Error(Inner::Owned(err.to_string())) + .. } } impl From for Error { - fn from(err: io::Error) -> Self { - Error::custom(err) - } + .. } impl From for io::Error { - fn from(err: Error) -> Self { - io::Error::new(io::ErrorKind::Other, err) - } + .. } impl error::Error for Error { - fn description(&self) -> &str { - self.0.description() - } - - fn cause(&self) -> Option<&dyn error::Error> { - self.0.cause() - } - } - - impl error::Error for Inner { - fn description(&self) -> &str { - match self { - Inner::Static(msg) => msg, - Inner::Owned(msg) => msg, - } - } + .. } } ``` @@ -782,200 +688,170 @@ To make it possible to carry any arbitrary `S::Error` type, where we don't know A `Value` is an erased container for some type whose structure can be visited, with a potentially short-lived lifetime: ```rust -pub struct Value<'v>(Inner<'v>); +pub struct Value<'v>(_); impl<'v> Value<'v> { pub fn from_any(v: &'v T, from: FromAnyFn) -> Self { - Value(Inner::new(v, from)) + .. } -} -``` -The inner type is an erased reference to some data and a function that captures its structure: + pub fn from_debug(v: &'v impl Debug) -> Self { + Self::from_any(v, |from, v| from.debug(v)) + } -```rust -struct Void { - _priv: (), - _oibit_remover: PhantomData<*mut dyn Fn()>, + #[cfg(feature = "kv_sval")] + pub fn from_sval(v: &'v (impl sval::Value + Debug)) -> Self { + Self::from_any(v, |from, v| from.sval(v)) + } + + #[cfg(feature = "kv_serde")] + pub fn from_serde(v: &'v (impl serde::Serialize + Debug)) -> Self { + Self::from_any(v, |from, v| from.serde(v)) + } } -#[derive(Clone, Copy)] -struct Inner<'a> { - data: &'a Void, - from: FromAnyFn, +impl<'v> Debug for Value<'v> { + .. } -type FromAnyFn = fn(FromAny, &T) -> Result<(), Error>; +impl<'v> Display for Value<'v> { + .. +} -impl<'a> Inner<'a> { - fn new(data: &'a T, from: FromAnyFn) -> Self { - unsafe { - Inner { - data: mem::transmute::<&'a T, &'a Void>(data), - from: mem::transmute::, FromAnyFn>(from), - } - } - } +#[cfg(feature = "kv_sval")] +impl<'v> sval::Value for Value<'v> { + .. +} - fn visit(&self, backend: &mut dyn Backend) -> Result<(), Error> { - (self.from)(FromAny(backend), self.data) - } +#[cfg(feature = "kv_serde")] +impl<'v> serde::Serialize for Value<'v> { + .. } + +type FromAnyFn = fn(FromAny, &T) -> Result<(), Error>; ``` The `FromAny` type is like a visitor that accepts values with a particular structure, but doesn't require those values satisfy any lifetime constraints: ```rust -/// A builder for a value. -/// -/// An instance of this type is passed to the `Value::from_any` method. -pub struct FromAny<'a>(&'a mut dyn Backend); +pub struct FromAny<'a>(_); -// NOTE: These methods aren't public. They're used by implementations of -// ToValue for standard library types. impl<'a> FromAny<'a> { + pub fn debug(self, v: impl Debug) -> Result<(), Error> { + .. + } + + #[cfg(feature = "kv_sval")] + pub fn sval(self, v: impl sval::Value + Debug) -> Result<(), Error> { + .. + } + + #[cfg(feature = "kv_serde")] + pub fn serde(self, v: impl serde::Serialize + Debug) -> Result<(), Error> { + .. + } + fn value(self, v: Value) -> Result<(), Error> { - v.0.visit(self.0) + .. } fn u64(self, v: u64) -> Result<(), Error> { - self.0.u64(v) + .. } fn i64(self, v: i64) -> Result<(), Error> { - self.0.i64(v) + .. } fn f64(self, v: f64) -> Result<(), Error> { - self.0.f64(v) + .. } fn bool(self, v: bool) -> Result<(), Error> { - self.0.bool(v) + .. } fn char(self, v: char) -> Result<(), Error> { - self.0.char(v) + .. } fn none(self) -> Result<(), Error> { - self.0.none() + .. } fn str(self, v: &str) -> Result<(), Error> { - self.0.str(v) + .. } } ``` -Internal implementations of `ToValue` for standard library primitives use the private methods on `Backend` to retain their structure, without having to commit to any machinery in the public API. Each serialization framework supported by `log` provides an internal implementation of `Backend`, so there's one for `std::fmt`, one for `sval`, and one for `serde`. +#### A minimal initial API -Each backend adds constructor methods to `Value` that allows it to capture values that satisfy the traits expected from that backend: +An initial implementation of `Value` could support just the `std::fmt` machinery: ```rust -mod fmt { - impl<'v> Value<'v> { - pub fn from_debug(v: &'v impl Debug) -> Self { - Self::from_any(v, |from, v| from.debug(v)) - } - } - - impl<'a> FromAny<'a> { - pub fn debug(self, v: impl Debug) -> Result<(), Error> { - .. - } - } - - impl<'v> fmt::Debug for Value<'v> { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - .. - } - } +pub struct Value<'v>(_); - impl<'v> fmt::Display for Value<'v> { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - .. - } +impl<'v> Value<'v> { + pub fn from_debug(v: &'v impl Debug) -> Self { + .. } } -#[cfg(feature = "kv_sval")] -mod sval { - impl<'v> Value<'v> { - pub fn from_sval(v: &'v (impl sval::Value + Debug)) -> Self { - Self::from_any(v, |from, v| from.sval(v)) - } - } - - impl<'a> FromAny<'a> { - pub fn sval(self, v: impl sval::Value + Debug) -> Result<(), Error> { - .. - } - } - - impl<'v> sval::Value for Value<'v> { - fn stream(&self, stream: &mut sval::value::Stream) -> Result<(), sval::Error> { - .. - } - } +impl<'v> Debug for Value<'v> { + .. } -#[cfg(feature = "kv_serde")] -mod serde { - impl<'v> Value<'v> { - pub fn from_serde(v: &'v (impl Serialize + Debug)) -> Self { - Self::from_any(v, |from, v| from.serde(v)) - } - } - - impl<'a> FromAny<'a> { - pub fn serde(self, v: impl Serialize + Debug) -> Result<(), Error> { - .. - } - } - - impl<'v> serde::Serialize for Value<'v> { - fn serialize(&self, serializer: S) -> Result - where - S: serde::Serializer, - { - .. - } - } +impl<'v> Display for Value<'v> { + .. } ``` -The implementation details of `Backend` aren't really important because they aren't public and could be implemented in a couple of ways, but a working reference is provided in the appendix. +Structured serialization frameworks could then be introduced without breakage. This could either be done in terms of the `FromAny` machinery shown previously, by exposing a serialization contract directly, or both. -#### A minimal initial API +#### Erasing values in `Value::from_any` -An initial implementation of `Value` could support just the `std::fmt` machinery: +Internally, the `Value` type uses similar machinery to `std::fmt::Argument` for pairing an erased incoming type with a function for operating on it: ```rust -pub struct Value<'v>(_); +pub struct Value<'v>(Inner<'v>); impl<'v> Value<'v> { - pub fn from_debug(v: &'v impl Debug) -> Self { - .. + pub fn from_any(v: &'v T, from: FromAnyFn) -> Self { + Value(Inner::new(v, from)) } } -impl<'v> Debug for Value<'v> { - .. +struct Void { + _priv: (), + _oibit_remover: PhantomData<*mut dyn Fn()>, } -impl<'v> Display for Value<'v> { - .. +#[derive(Clone, Copy)] +struct Inner<'a> { + data: &'a Void, + from: FromAnyFn, } -``` -Structured serialization frameworks could then be introduced without breakage. This could either be done in terms of the `FromAny` machinery shown previously, by exposing a serialization contract directly, or both. +type FromAnyFn = fn(FromAny, &T) -> Result<(), Error>; -#### Adding another supported framework +impl<'a> Inner<'a> { + fn new(data: &'a T, from: FromAnyFn) -> Self { + unsafe { + Inner { + data: mem::transmute::<&'a T, &'a Void>(data), + from: mem::transmute::, FromAnyFn>(from), + } + } + } -Adding optional support for another serialization framework like `serde` or `sval` can be done by implementing the `serde::Serialize` or `sval::Value` traits, and adding constructor methods to `Value` that allow it to capture implementations of those traits. A side-effect of pushing all supported serialization frameworks through the one type is that all supported frameworks will have to provide bridging support for all other supported frameworks. This makes the barrier for new frameworks raise exponentially and discourages supporting too many, but also protects the end-user experience from degrading when multiple frameworks are used in a single logging pipeline. + fn visit(&self, backend: &mut dyn Backend) -> Result<(), Error> { + (self.from)(FromAny(backend), self.data) + } +} +``` -The `sval` library is designed as a compatible extension for structured logging, and might be the first serialization framework to consider supporting (it comes along with `serde` support). +The benefit of the `Value::from_any` approach over a dedicated trait is that `Value::from_any` doesn't make any constraints on the incoming `&'v T` besides needing to satisfy the `'v` lifetime. That makes it possible to materialize newtypes from the borrowed `&'v T` to satisfy serialization constraints for cases where the caller doesn't own `T` and can't implement traits on it. #### Ownership @@ -1119,96 +995,61 @@ impl<'a> ToValue for &'a str { A `Key` is a short-lived structure that can be represented as a UTF-8 string. This might be possible without allocating, or it might require a destination to write into: ```rust -pub struct Key<'k>(Inner<'k>); +pub struct Key<'k>(_); impl<'k> Key<'k> { - /// Create a key from a borrowed string and optional index. pub fn from_str(key: &'k (impl Borrow + ?Sized)) -> Self { - Key(Inner::Borrowed(key.borrow())) + .. } pub fn as_str(&self) -> &str { - match self.0 { - Inner::Borrowed(k) => k, - #[cfg(feature = "std")] - Inner::Owned(ref k) => &*k, - } + .. } } impl<'k> AsRef for Key<'k> { - fn as_ref(&self) -> &str { - self.as_str() - } + .. } impl<'k> Borrow for Key<'k> { - fn borrow(&self) -> &str { - self.as_str() - } + .. } impl<'k> From<&'k str> for Key<'k> { - fn from(k: &'k str) -> Self { - Key::from_str(k, None) - } + .. } impl<'k> PartialEq for Key<'k> { - fn eq(&self, other: &Self) -> bool { - self.as_str().eq(other.as_str()) - } + .. } impl<'k> Eq for Key<'k> {} impl<'k> PartialOrd for Key<'k> { - fn partial_cmp(&self, other: &Self) -> Option { - self.as_str().partial_cmp(other.as_str()) - } + .. } impl<'k> Ord for Key<'k> { - fn cmp(&self, other: &Self) -> Ordering { - self.as_str().cmp(other.as_str()) - } + .. } impl<'k> Hash for Key<'k> { - fn hash(&self, state: &mut H) - where - H: Hasher, - { - self.as_str().hash(state) - } -} - -impl<'k> fmt::Debug for Key<'k> { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - self.as_str().fmt(f) - } + .. } -impl<'k> fmt::Display for Key<'k> { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - self.as_str().fmt(f) - } +impl<'k> Debug for Key<'k> { + .. } -enum Inner<'k> { - Borrowed(&'k str), - #[cfg(feature = "std")] - Owned(String), +impl<'k> Display for Key<'k> { + .. } #[cfg(feature = "std")] mod std_support { - use super::*; - impl<'k> Key<'k> { - /// Create a key from an owned string and optional index. pub fn from_owned(key: impl Into) -> Self { - Key(Inner::Owned(key.into())) + .. } } @@ -1219,38 +1060,21 @@ mod std_support { } impl<'k> From for Key<'k> { - fn from(k: String) -> Self { - Key::from_owned(k, None) - } + .. } } #[cfg(feature = "kv_sval")] mod sval_support { - use super::*; - - use sval::value::{self, Value}; - - impl<'k> Value for Key<'k> { - fn stream(&self, stream: &mut value::Stream) -> Result<(), value::Error> { - self.as_str().stream(stream) - } + impl<'k> sval::Value for Key<'k> { + .. } } #[cfg(feature = "kv_serde")] mod serde_support { - use super::*; - - use serde::{Serialize, Serializer}; - impl<'k> Serialize for Key<'k> { - fn serialize(&self, serializer: S) -> Result - where - S: Serializer, - { - self.as_str().serialize(serializer) - } + .. } } ``` @@ -1263,13 +1087,9 @@ The `Key` type can either borrow or own its inner value. #### Thread-safety -The `Key` type is probably `Send` + `Sync`, but that's not guaranteed. - -#### Extensibility - -Future enhancements that could be made to `Key` that haven't been otherwise considered by this RFC. +The `Key` type is probably `Send` and `Sync`, but that's not guaranteed. -##### Adding an index to keys +#### Extensibility: Adding an index to keys The `Key` type could be extended to hold an optional index into a source. This could be used to retrieve a specific key-value pair more efficiently than scanning. @@ -1322,40 +1142,25 @@ pub trait Source { where Self: Sized, { - ErasedSource::erased(self) + .. } fn get<'kvs, Q>(&'kvs self, key: Q) -> Option> where Q: ToKey, { - struct Get<'k, 'v>(Key<'k>, Option>); - - impl<'k, 'kvs> Visitor<'kvs> for Get<'k, 'kvs> { - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { - if k == self.0 { - self.1 = Some(v); - } - - Ok(()) - } - } - - let mut visitor = Get(key.to_key(), None); - let _ = self.visit(&mut visitor); - - visitor.1 + .. } fn by_ref(&self) -> &Self { - self + .. } fn chain(self, other: KVS) -> Chained where Self: Sized, { - Chained(self, other) + .. } fn try_for_each(self, f: F) -> Result<(), Error> @@ -1364,20 +1169,7 @@ pub trait Source { F: FnMut(Key, Value) -> Result<(), E>, E: Into, { - struct ForEach(F, PhantomData); - - impl<'kvs, F, E> Visitor<'kvs> for ForEach - where - F: FnMut(Key, Value) -> Result<(), E>, - E: Into, - { - fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error> { - (self.0)(k, v).map_err(Into::into) - } - } - - let mut for_each = ForEach(f, Default::default()); - self.visit(&mut for_each) + .. } #[cfg(any(feature = "kv_serde", feature = "kv_sval"))] @@ -1385,7 +1177,7 @@ pub trait Source { where Self: Sized, { - AsMap(self) + .. } #[cfg(any(feature = "kv_serde", feature = "kv_sval"))] @@ -1393,7 +1185,7 @@ pub trait Source { where Self: Sized, { - AsSeq(self) + .. } } ``` @@ -1406,16 +1198,13 @@ An initial implementation of `Source` could be provided with just the `visit` an ```rust pub trait Source { - /// Serialize the key value pairs. fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>; - /// Erase this `Source` so it can be used without - /// requiring generic type parameters. fn erase(&self) -> ErasedSource where Self: Sized, { - ErasedSource::erased(self) + .. } } ``` @@ -1588,11 +1377,7 @@ where The `BTreeMap` and `HashMap` implementations provide more efficient implementations of `Source::get`. -#### Extensibility - -Future enhancements that could be made to `Source` that haven't been otherwise considered by this RFC. - -##### Sending `Source`s between threads +#### Extensibility: Sending `Source`s between threads Before a record could be processed on a background thread it would need to be converted into some owned variant. The `Source` trait is the point where having some way to convert from a borrowed to an owned value would make the most sense because that's where the knowledge of the underlying key-value storage is. @@ -1637,7 +1422,10 @@ pub trait Visitor<'kvs> { impl<'a, 'kvs, T: ?Sized> Visitor<'kvs> for &'a mut T where - T: Visitor<'kvs> { } + T: Visitor<'kvs> +{ + .. +} ``` A `Visitor` may serialize the keys and values as it sees them. It may also do other work, like sorting or de-duplicating them. Operations that involve ordering keys will probably require allocations. @@ -1821,367 +1609,3 @@ Supporting something like message templates in Rust using the `log!` macros woul # Unresolved questions [unresolved-questions]: #unresolved-questions - -# Appendix - -## Public API - -For context, ignoring the `log!` macros, this is roughly the additional public API this RFC proposes to support structured logging: - -```rust - -``` - -## Backends for `Value` - -Each supported serialization framework supported by the `Value` type implements an internal `Backend` trait. The exact machinery isn't really important because it's not public. This reference implementation is an internal serialization contract that includes primitive types, and methods for specific frameworks depending on crate features: - -```rust -trait Backend: fmt::Backend + sval::Backend + serde::Backend { - fn u64(&mut self, v: u64) -> Result<(), Error>; - fn i64(&mut self, v: i64) -> Result<(), Error>; - fn f64(&mut self, v: f64) -> Result<(), Error>; - fn bool(&mut self, v: bool) -> Result<(), Error>; - fn char(&mut self, v: char) -> Result<(), Error>; - fn str(&mut self, v: &str) -> Result<(), Error>; - fn none(&mut self) -> Result<(), Error>; -} -``` - -#### `std::fmt` backend - -```rust -mod fmt { - impl<'v> value::Value<'v> { - pub fn from_debug(v: &'v impl fmt::Debug) -> Self { - Self::from_any(v, |from, v| from.debug(v)) - } - } - - impl<'v> fmt::Debug for value::Value<'v> { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - self.0.visit(&mut FmtBackend(f)).map_err(|_| fmt::Error) - } - } - - impl<'v> fmt::Display for value::Value<'v> { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - write!(f, "{:?}", self) - } - } - - impl<'a> value::FromAny<'a> { - pub fn debug(self, v: impl fmt::Debug) -> Result<(), value::Error> { - self.0.debug(&v) - } - } - - pub(in crate::key_values::value) trait Backend { - fn debug(&mut self, v: &dyn Value) -> Result<(), value::Error>; - } - - pub(in crate::key_values::value) use fmt::Debug as Value; - - struct FmtBackend<'a, 'b>(&'a mut fmt::Formatter<'b>); - - impl<'a, 'b> value::Backend for FmtBackend<'a, 'b> { - fn u64(&mut self, v: u64) -> Result<(), value::Error> { - self.debug(&v) - } - - fn i64(&mut self, v: i64) -> Result<(), value::Error> { - self.debug(&v) - } - - fn f64(&mut self, v: f64) -> Result<(), value::Error> { - self.debug(&v) - } - - fn bool(&mut self, v: bool) -> Result<(), value::Error> { - self.debug(&v) - } - - fn char(&mut self, v: char) -> Result<(), value::Error> { - self.debug(&v) - } - - fn none(&mut self) -> Result<(), value::Error> { - self.debug(&Option::None::<()>) - } - - fn str(&mut self, v: &str) -> Result<(), value::Error> { - self.debug(&v) - } - } - - impl<'a, 'b> Backend for FmtBackend<'a, 'b> { - fn debug(&mut self, v: &dyn fmt::Debug) -> Result<(), value::Error> { - write!(self.0, "{:?}", v)?; - - Ok(()) - } - } - - #[cfg(feature = "kv_sval")] - impl<'a, 'b> value::sval::Backend for FmtBackend<'a, 'b> { - fn sval(&mut self, v: &dyn value::sval::Value) -> Result<(), value::Error> { - self.debug(&v) - } - } - - #[cfg(feature = "kv_serde")] - impl<'a, 'b> value::serde::Backend for FmtBackend<'a, 'b> { - fn serde(&mut self, v: &dyn value::serde::Value) -> Result<(), value::Error> { - self.debug(&v) - } - } -} -``` - -The `fmt::Backend` allows any `Value` to be formatted using `std::fmt`, which is exposed to consumers through the `Debug` and `Display` traits. - -#### `sval` backend - -```rust -mod sval { - #[cfg(feature = "kv_sval")] - mod imp { - impl<'v> value::Value<'v> { - pub fn from_sval(v: &'v (impl sval::Value + fmt::Debug)) -> Self { - Self::from_any(v, |from, v| from.sval(v)) - } - } - - impl<'v> sval::Value for value::Value<'v> { - fn stream(&self, stream: &mut sval::value::Stream) -> Result<(), sval::value::Error> { - self.0.visit(&mut SvalBackend(stream))?; - - Ok(()) - } - } - - impl<'a> value::FromAny<'a> { - pub fn sval(self, v: (impl sval::Value + fmt::Debug)) -> Result<(), value::Error> { - self.0.sval(&v) - } - } - - pub(in crate::key_values::value) trait Backend { - fn sval(&mut self, v: &dyn Value) -> Result<(), value::Error>; - } - - pub(in crate::key_values::value) trait Value: sval::Value + fmt::Debug {} - impl Value for T where T: sval::Value + fmt::Debug {} - - struct SvalBackend<'a, 'b>(&'a mut sval::value::Stream<'b>); - - impl<'a, 'b> SvalBackend<'a, 'b> { - fn any(&mut self, v: impl sval::Value) -> Result<(), value::Error> { - self.0.any(v)?; - - Ok(()) - } - } - - impl<'a, 'b> value::Backend for SvalBackend<'a, 'b> { - fn u64(&mut self, v: u64) -> Result<(), value::Error> { - self.sval(&v) - } - - fn i64(&mut self, v: i64) -> Result<(), value::Error> { - self.sval(&v) - } - - fn f64(&mut self, v: f64) -> Result<(), value::Error> { - self.sval(&v) - } - - fn bool(&mut self, v: bool) -> Result<(), value::Error> { - self.sval(&v) - } - - fn char(&mut self, v: char) -> Result<(), value::Error> { - self.sval(&v) - } - - fn none(&mut self) -> Result<(), value::Error> { - self.sval(&Option::None::<()>) - } - - fn str(&mut self, v: &str) -> Result<(), value::Error> { - self.sval(&v) - } - } - - impl<'a, 'b> Backend for SvalBackend<'a, 'b> { - fn sval(&mut self, v: &dyn Value) -> Result<(), value::Error> { - self.any(v) - } - } - - impl<'a, 'b> value::fmt::Backend for SvalBackend<'a, 'b> { - fn debug(&mut self, v: &dyn value::fmt::Value) -> Result<(), value::Error> { - self.any(format_args!("{:?}", v)) - } - } - - #[cfg(feature = "kv_serde")] - impl<'a, 'b> value::serde::Backend for SvalBackend<'a, 'b> { - fn serde(&mut self, v: &dyn value::serde::Value) -> Result<(), value::Error> { - self.any(sval::serde::to_value(v)) - } - } - } - - #[cfg(not(feature = "kv_sval"))] - mod imp { - pub(in crate::key_values::value) trait Backend {} - - impl Backend for V where V: value::Backend {} - } - - pub(super) use self::imp::*; -} -``` - -#### `serde` backend - -```rust -mod serde { - #[cfg(feature = "kv_serde")] - mod imp { - impl<'v> value::Value<'v> { - pub fn from_serde(v: &'v (impl serde::Serialize + fmt::Debug)) -> Self { - Self::from_any(v, |from, v| from.serde(v)) - } - } - - impl<'v> serde::Serialize for value::Value<'v> { - fn serialize(&self, serializer: S) -> Result - where - S: serde::Serializer, - { - let mut visitor = SerdeBackend { - serializer: Some(serializer), - ok: None, - }; - - self.0.visit(&mut visitor).map_err(value::Error::into_serde)?; - - Ok(visitor.ok.expect("missing return value")) - } - } - - impl<'a> value::FromAny<'a> { - pub fn serde(self, v: (impl serde::Serialize + fmt::Debug)) -> Result<(), value::Error> { - self.0.serde(&v) - } - } - - pub(in crate::key_values::value) trait Backend { - fn serde(&mut self, v: &dyn Value) -> Result<(), value::Error>; - } - - pub(in crate::key_values::value) trait Value: erased_serde::Serialize + fmt::Debug {} - impl Value for T where T: serde::Serialize + fmt::Debug {} - - impl<'a> serde::Serialize for &'a dyn Value { - fn serialize(&self, serializer: S) -> Result - where - S: serde::Serializer, - { - erased_serde::serialize(*self, serializer) - } - } - - struct SerdeBackend - where - S: serde::Serializer, - { - serializer: Option, - ok: Option, - } - - impl SerdeBackend - where - S: serde::Serializer, - { - fn serialize(&mut self, v: impl erased_serde::Serialize) -> Result<(), value::Error> { - self.ok = Some(erased_serde::serialize(&v, self.serializer.take().expect("missing serializer")).map_err(value::Error::from_serde)?); - - Ok(()) - } - } - - impl value::Backend for SerdeBackend - where - S: serde::Serializer, - { - fn u64(&mut self, v: u64) -> Result<(), value::Error> { - self.serde(&v) - } - - fn i64(&mut self, v: i64) -> Result<(), value::Error> { - self.serde(&v) - } - - fn f64(&mut self, v: f64) -> Result<(), value::Error> { - self.serde(&v) - } - - fn bool(&mut self, v: bool) -> Result<(), value::Error> { - self.serde(&v) - } - - fn char(&mut self, v: char) -> Result<(), value::Error> { - self.serde(&v) - } - - fn none(&mut self) -> Result<(), value::Error> { - self.serde(&Option::None::<()>) - } - - fn str(&mut self, v: &str) -> Result<(), value::Error> { - self.serde(&v) - } - } - - impl Backend for SerdeBackend - where - S: serde::Serializer, - { - fn serde(&mut self, v: &dyn Value) -> Result<(), value::Error> { - self.serialize(v) - } - } - - impl value::fmt::Backend for SerdeBackend - where - S: serde::Serializer, - { - fn debug(&mut self, v: &dyn value::fmt::Value) -> Result<(), value::Error> { - self.serialize(format_args!("{:?}", v)) - } - } - - #[cfg(feature = "kv_sval")] - impl value::sval::Backend for SerdeBackend - where - S: serde::Serializer, - { - fn sval(&mut self, v: &dyn value::sval::Value) -> Result<(), value::Error> { - self.serialize(sval::serde::to_serialize(v)) - } - } - } - - #[cfg(not(feature = "kv_serde"))] - mod imp { - pub(in crate::key_values::value) trait Backend {} - - impl Backend for V where V: value::Backend {} - } - - pub(super) use self::imp::*; -} -``` \ No newline at end of file From 033deb790f549e2023503857f4d2714016a84179 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Tue, 29 Jan 2019 16:50:13 +1000 Subject: [PATCH 10/20] update wording and structure --- rfcs/0000-structured-logging.md | 317 ++++++++++++++++++++++---------- 1 file changed, 215 insertions(+), 102 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index f8ac1c9d4..11453a07e 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -3,14 +3,13 @@ Add support for structured logging to the `log` crate in both `std` and `no_std` environments, allowing log records to carry typed data beyond a textual message. This document serves as an introduction to what structured logging is all about, and as an RFC for an implementation in the `log` crate. -`log` will provide an API for capturing structured data that's agnostic of the underlying serialization framework, whether that's `std::fmt`, `serde`, or `sval`. +`log` will provide an API for capturing structured data that offloads complex serialization to de-facto standards in the ecosystem, but avoids integrating them too tightly, or forcing any specific framework on consumers. The API is heavily inspired by `slog` and `tokio-trace`. > NOTE: Code in this RFC uses recent language features like `impl Trait`, but can be implemented without them. # Contents - - [Motivation](#motivation) - [What is structured logging?](#what-is-structured-logging) - [Why do we need structured logging in `log`?](#why-do-we-need-structured-logging-in-log) @@ -22,7 +21,7 @@ The API is heavily inspired by `slog` and `tokio-trace`. - [Reference-level explanation](#reference-level-explanation) - [Design considerations](#design-considerations) - [Cargo features](#cargo-features) - - [Key-values API](#key-values-api) + - [A complete key-values API](#a-complete-key-values-api) - [`Error`](#error) - [`Value`](#value) - [`ToValue`](#tovalue) @@ -31,6 +30,7 @@ The API is heavily inspired by `slog` and `tokio-trace`. - [`Source`](#source) - [`Visitor`](#visitor) - [`Record` and `RecordBuilder`](#record-and-recordbuilder) + - [A minimal key-values API](#a-minimal-key-values-api) - [The `log!` macros](#the-log-macros) - [Drawbacks, rationale, and alternatives](#drawbacks-rationale-and-alternatives) - [Prior art](#prior-art) @@ -41,17 +41,17 @@ The API is heavily inspired by `slog` and `tokio-trace`. ## What is structured logging? -Information in log records can be traditionally captured as a blob of text, including a level, a message, and maybe a few other pieces of metadata. There's a lot of potentially valuable information we throw away when we format data as text. Arbitrary textual representations often result in log records that are neither easy for humans to read, nor for machines to parse. +Information in log records can be traditionally captured as a blob of text, including a level, a message, and maybe a few other pieces of metadata. There's a lot of potentially valuable information we throw away when we format log records this way. Arbitrary textual representations often result in output that is neither easy for humans to read, nor for machines to parse. Structured logs can retain their original structure in a machine-readable format. They can be changed programmatically within a logging pipeline before reaching their destination. Once there, they can be analyzed using common database tools. -As an example of structured logging, a textual log like this: +As an example of structured logging, a textual record like this: ``` [INF 2018-09-27T09:32:03Z basic] [service: database, correlation: 123] Operation completed successfully in 18ms ``` -could be represented as a structured log like this: +could be represented as a structured record like this: ```json { @@ -65,7 +65,7 @@ could be represented as a structured log like this: } ``` -When log records are kept in a format like this, potentially interesting queries like _what are all records where the correlation is 123?_, or _how many errors were there in the last hour?_ can be computed efficiently. +When log records are kept in a structured format like this, potentially interesting queries like _what are all records where the correlation is 123?_, or _how many errors were there in the last hour?_ can be computed efficiently. Even when logging to a console for immediate consumption, the human-readable message can be presented better when it's not trying to include ambient metadata inline: @@ -94,11 +94,11 @@ A healthy logging ecosystem needs both `log` and frameworks like `slog`. As a st # Guide-level explanation [guide-level-explanation]: #guide-level-explanation -This section introduces the new structured logging API through a tour of how structured values can be captured and consumed. +This section introduces `log`'s structured logging API through a tour of how structured records can be captured and consumed. ## Logging structured key-value pairs -Structured logging is supported in `log` by allowing typed key-value pairs to be associated with a log record. A `;` separates structured key-value pairs from values that are replaced into the message: +Structured logging is supported in `log` by allowing typed key-value pairs to be associated with a log record. A `;` separates structured key-value pairs from other data that's interpolated into the message: ```rust info!( @@ -109,7 +109,9 @@ info!( ); ``` -Any `value` or `key = value` expressions before the `;` in the macro will be interpolated into the message as unstructured text using `std::fmt`. This is the `log!` macro we have today. Any `value` or `key = value` expressions after the `;` will be captured as structured key-value pairs. These structured key-value pairs can be inspected or serialized, retaining some notion of their original type. That means in the above example, the `message` key is unstructured, and the `correlation` and `user` keys are structured: +Any `value` or `key = value` expressions before the `;` in the macro will be interpolated into the message as unstructured text using `std::fmt`. This is the `log!` macro we have today. + +Any `value` or `key = value` expressions after the `;` will be captured as structured key-value pairs. These structured key-value pairs can be inspected or serialized, retaining some notion of their original type. That means in the above example, the `message` pair is unstructured, and the `correlation` and `user` pairs are structured: ``` info!( @@ -128,13 +130,13 @@ info!( ); ``` -### What can be logged? +### What can be captured as a structured value? -A type can be logged if it implements the `ToValue` trait: +A value can be captured as a structured value in a log record if it implements the `ToValue` trait: ```rust pub trait ToValue { - fn to_Value(&self) -> Value; + fn to_value(&self) -> Value; } ``` @@ -151,7 +153,7 @@ impl<'v> Debug for Value<'v> { We'll look at `Value` in more detail later. For now, we can think of it as a container that normalizes capturing and emitting the structure of values. -In the example from before: +So, in the example from before: ```rust info!( @@ -162,7 +164,7 @@ info!( ); ``` -the `correlation_id` and `user` fields can be used as structured values if they implement the `ToValue` trait: +the `correlation_id` and `user` pairs can be captured as structured values if they implement the `ToValue` trait: ``` info!( @@ -181,18 +183,18 @@ info!( ); ``` -Within `log` itself, a fixed set of primitive types from the standard library implement the `ToValue` trait: +Initially, that means a fixed set of primitive types from the standard library: - Standard formats: `Arguments` - Primitives: `bool`, `char` - Unsigned integers: `u8`, `u16`, `u32`, `u64`, `u128` - Signed integers: `i8`, `i16`, `i32`, `i64`, `i128` - Strings: `&str`, `String` -- Bytes: `&[u8]`, `Vec` +- Slices: `&[T]`, `Vec` - Paths: `&Path`, `PathBuf` - Special types: `Option`, `&T`, and `()`. -Each of these types implements `ToValue` in a way that retains their typing. Using `u8` as an example: +Each of these types implements `ToValue` in a way that opaquely retains some notion of their underlying structure. Using `u8` as an example: ```rust impl ToValue for u8 { @@ -226,11 +228,21 @@ impl FromAny { This machinery is very similar to the internals of `std::fmt`. -Only being able to log primitive types from the standard library is a bit limiting though. What if `correlation_id` is a `uuid::Uuid`, and `user` is a struct, `User`, with fields? +Only being able to log primitive types from the standard library is a bit limiting though. What if `correlation_id` is a `uuid::Uuid`, and `user` is a struct, `User`, with its own fields? #### Implementing `ToValue` for a simple value -`uuid::Uuid` could implement the `ToValue` trait directly by capturing its structure as a debuggable format: +A newtype structure like `uuid::Uuid` could implement the `ToValue` trait directly in terms of some underlying value that already implements `ToValue`: + +```rust +impl ToValue for Uuid { + fn to_value(&self) -> Value { + self.as_bytes().to_value() + } +} +``` + +Alternatively, `uuid::Uuid` could provide a nicer implementation using the `Debug` implementation of its hyphenated format: ```rust impl ToValue for Uuid { @@ -240,20 +252,20 @@ impl ToValue for Uuid { } ``` -There's some subtlety in this implementation. The actual value whose structure is captured is not the `&'v Uuid`, it's the owned `ToHyphenated<'v>` structure. This is why `Value::from_any` uses a separate function for capturing the structure of its values. It lets us capture a borrowed `Uuid` with the right lifetime `'v`, but materialize an owned `ToHyphenated` with the structure we want. +There's some subtlety in this second implementation. The actual value whose structure is captured is not the `&'v Uuid`, it's the owned `ToHyphenated<'v>`. This is why `Value::from_any` uses a separate function for capturing the structure of its values that doesn't depend on the lifetime of the given `&'v T`. It lets us capture a borrowed `Uuid` with the right lifetime `'v`, but materialize an owned `ToHyphenated` with the structure we want. #### Implementing `ToValue` for a complex value -A structure like `User` is a bit different. It could be represented using `Debug`, but then the contents of its fields would be lost in an opaque and unstructured string. It would be better represented as a map of key-value pairs. However, complex values like maps and sequences aren't directly supported in `log`. They're offloaded to serialization frameworks like `serde` and `sval` that are capable of handling them effectively. +A structure like `User` is a bit different from a newtype like `uuid::Uuid`. It could be represented using `Debug`, but then the contents of its fields would be lost in an opaque and unstructured string. It would be better represented as a map of key-value pairs. However, complex values like maps and sequences aren't directly supported in `log`. They're offloaded to serialization frameworks like `serde` and `sval` that are capable of handling them effectively. -Fundamental serialization frameworks do have direct integration with `log`'s `Value` type through Cargo features. Let's use `sval` as an example. It's a serialization framework that's built specifically for structured logging. Adding the `kv_sval` feature to `log` will enable its integration: +`serde` and `sval` have direct two-way integration with `log`'s `Value` type through optional Cargo features. Let's use `sval` as an example. It's a serialization framework that's built specifically for structured logging. Adding the `kv_sval` feature to `log` will enable it: ```toml [dependencies.log] features = ["kv_sval"] ``` -The `User` type can then derive `sval`'s `Value` trait and implement `log`'s `ToValue` trait in terms of `sval`: +Complex structures that derive `sval`'s `Value` trait can then implement `log`'s `ToValue` trait in terms of `sval`: ```rust #[derive(Debug, Value)] @@ -268,7 +280,7 @@ impl ToValue for User { } ``` -Using `serde` instead of `sval` is a similar story: +In this way, the underlying structure of a `User`, described as a map by `sval`, is retained when converting it into a `Value`. Using `serde` instead of `sval` is a similar story: ```toml [dependencies.log] @@ -290,7 +302,7 @@ impl ToValue for User { #### Capturing values without implementing `ToValue` -Instead of implementing `ToValue` on types throughout the ecosystem at all, callers of the `log!` macros could instead create ad-hoc `Value`s from their data: +Instead of implementing `ToValue` on types throughout the ecosystem, callers of the `log!` macros could instead create ad-hoc `Value`s from their data at the callsite: ```rust use log::key_values::Value; @@ -303,7 +315,7 @@ info!( ); ``` -In this example, neither `correlation_id` nor `user` need to implement any traits from `log`: +In this example, `correlation_id` and `user` don't need to implement any traits from `log`. Instead, they need to implement the corresponding trait from `sval` or `serde`: ``` info!( @@ -344,6 +356,15 @@ pub trait Source { .. } + // Run a function for each key-value pair + fn for_each(self, f: F) -> Result<(), Error> + where + Self: Sized, + F: FnMut(Key, Value), + { + .. + } + // Run a function for each key-value pair fn try_for_each(self, f: F) -> Result<(), Error> where @@ -376,7 +397,7 @@ pub trait Source { ### Writing key-value pairs as text -To demonstrate how to work with a `Source`, let's take the terminal log format from before: +To demonstrate how to work with a `Source`, let's take the textual log format from before: ``` [INF 2018-09-27T09:32:03Z] Operation completed successfully in 18ms @@ -386,7 +407,7 @@ correlation: 123 took: 18 ``` -Each key-value pair, shown as a `$key: $value` line, can be formatted from the `Source` using the `std::fmt` machinery: +Each key-value pair, shown as a `$key: $value` line, can be formatted from the `Source` on a record using the `std::fmt` machinery: ```rust use log::key_values::Source; @@ -398,13 +419,13 @@ fn log_record(w: impl Write, r: &Record) -> io::Result<()> { // Write each key-value pair on a new line record .key_values() - .try_for_each(|k, v| writeln!("{}: {}", k, v))?; + .for_each(|k, v| writeln!("{}: {}", k, v))?; Ok(()) } ``` -In the above example, the `Source::try_for_each` method iterates over each key-value pair in the `Source` and writes them to the terminal. +In the above example, the `Source::for_each` method iterates over each key-value pair in the `Source` and writes them to the output stream. ### Writing key-value pairs as JSON @@ -457,17 +478,17 @@ struct SerializeRecord { } ``` -This time, instead of using the `Source::try_for_each` method, we use the `Source::as_map` method to get an adapter that implements `serde::Serialize` by serializing each key-value pair as an entry in a `serde` map. +This time, instead of using the `Source::for_each` method, we use the `Source::as_map` method to get an adapter that implements `serde::Serialize` by serializing each key-value pair as an entry in a `serde` map. ## Integrating log frameworks with `log` -The `Source` trait we saw previously describes some container for structured key-value pairs that can be iterated through. Other log frameworks that want to integrate with the `log` crate should build `Record`s that contain some implementation of `Source` based on their own structured logging. +The `Source` trait we saw previously describes some container for structured key-value pairs that can be iterated or serialized. Other log frameworks that want to integrate with the `log` crate should build `Record`s that contain some implementation of `Source` based on their own structured logging. The previous section demonstrated some of the methods available on `Source` like `Source::try_for_each` and `Source::as_map`. Both of those methods are provided on top of a required lower-level `Source::visit` method, which looks something like this: ```rust trait Source { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>; // Provided methods } @@ -497,7 +518,7 @@ The `Source` trait could be implemented for `KeyValues` like this: use log::key_values::source::{self, Source}; impl Source for KeyValues { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut impl source::Visitor<'kvs>) -> Result<(), source::Error> { for (k, v) in self.data { visitor.visit_pair(source::Key::from_str(k), source::Value::from_serde(v)) } @@ -518,7 +539,7 @@ impl Source for SortRetainLast where KVS: Source, { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn source::Visitor<'kvs>) -> Result<(), source::Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut impl source::Visitor<'kvs>) -> Result<(), source::Error> { // `Seen` is a visitor that will capture key-value pairs // in a `BTreeMap`. We use it internally to sort and de-duplicate // the key-value pairs that `SortRetainLast` is wrapping. @@ -572,7 +593,7 @@ Don't create a new serialization API that requires `log` to become a public depe Provide an API that's suitable for two independent logging frameworks to integrate through if they want. Producers of structured data and consumers of structured data should be able to use different serialization frameworks opaquely and still get good results. As an example, a caller of `info!` should be able to log a map that implements `sval::Value`, and the implementor of the receiving `Log` trait should be able to format that map using `serde::Serialize`. -### Object safety +### Remain object safe `log` is already designed to be object-safe so this new structured logging API needs to be object-safe too. @@ -597,11 +618,11 @@ Using default features, implementors of the `Log` trait will be able to format s `sval` is a new serialization framework that's specifically designed with structured logging in mind. It's `no_std` and object-safe, but isn't stable and requires `rustc` `1.31.0`. Using the `kv_sval` feature, any `Value` will also implement `sval::Value` so its underlying structure will be visible to consumers of structured data using `sval::Stream`s. -Using the `kv_serde` feature, any `Value` will also implement `serde::Serialize` so its underlying structure will be visible to consumers of structured data using `serde::Serializer`s. +`serde` is the de-facto general-purpose serialization framework for Rust. It's widely supported and stable, but requires some significant runtime machinery in order to be object-safe. Using the `kv_serde` feature, any `Value` will also implement `serde::Serialize` so its underlying structure will be visible to consumers of structured data using `serde::Serializer`s. -## Key-values API +## A complete key-values API -The following section details the public API for structured values in `log`, along with possible future extensions and minimal initial implementations. Actual implementation details are excluded for brevity unless they're particularly noteworthy. See the original comment on the [RFC issue](https://github.com/rust-lang-nursery/log/pull/296#issue-222687727) for a reference implementation. +The following section details the public API for structured values in `log`, along with possible future extensions. Actual implementation details are excluded for brevity unless they're particularly noteworthy. See the original comment on the [RFC issue](https://github.com/rust-lang-nursery/log/pull/296#issue-222687727) for a reference implementation. ### `Error` @@ -785,30 +806,6 @@ impl<'a> FromAny<'a> { } ``` -#### A minimal initial API - -An initial implementation of `Value` could support just the `std::fmt` machinery: - -```rust -pub struct Value<'v>(_); - -impl<'v> Value<'v> { - pub fn from_debug(v: &'v impl Debug) -> Self { - .. - } -} - -impl<'v> Debug for Value<'v> { - .. -} - -impl<'v> Display for Value<'v> { - .. -} -``` - -Structured serialization frameworks could then be introduced without breakage. This could either be done in terms of the `FromAny` machinery shown previously, by exposing a serialization contract directly, or both. - #### Erasing values in `Value::from_any` Internally, the `Value` type uses similar machinery to `std::fmt::Argument` for pairing an erased incoming type with a function for operating on it: @@ -845,6 +842,7 @@ impl<'a> Inner<'a> { } } + // Backend is an internal trait that bridges supported serialization frameworks fn visit(&self, backend: &mut dyn Backend) -> Result<(), Error> { (self.from)(FromAny(backend), self.data) } @@ -1163,6 +1161,14 @@ pub trait Source { .. } + fn for_each(self, f: F) -> Result<(), Error> + where + Self: Sized, + F: FnMut(Key, Value), + { + .. + } + fn try_for_each(self, f: F) -> Result<(), Error> where Self: Sized, @@ -1192,23 +1198,6 @@ pub trait Source { `Source` doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. -#### A minimal initial API - -An initial implementation of `Source` could be provided with just the `visit` and `erase` methods: - -```rust -pub trait Source { - fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>; - - fn erase(&self) -> ErasedSource - where - Self: Sized, - { - .. - } -} -``` - #### Adapters Some useful adapters exist as provided methods on the `Source` trait. They're similar to adapters on the standard `Iterator` trait: @@ -1216,9 +1205,10 @@ Some useful adapters exist as provided methods on the `Source` trait. They're si - `by_ref` to get a reference to a `Source` within a method chain. - `chain` to concatenate one source with another. This is useful for composing implementations of `Log` together for contextual logging. - `get` to try find the value associated with a key. -- `try_for_each` to try execute some closure over all key-value pairs. This is a convenient way to do something with each key-value pair without having to create and implement a `Visitor`. +- `for_each` to execute some closure over all key-value pairs. This is a convenient way to do something with each key-value pair without having to create and implement a `Visitor`. +- `try_for_each` is like `for_each`, but takes a fallible closure. - `as_map` to get a serializable map. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. -- `as_seq` to get a serializable sequence of tuples. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. +- `as_seq` is like `as_map`, but for serializing as a sequence of tuples. None of these methods are required for the core API. They're helpful tools for working with key-value pairs with minimal machinery. Even if we don't necessarily include them right away it's worth having an API that can support them later without breakage. @@ -1254,7 +1244,7 @@ impl<'a> Source for ErasedSource<'a> { } trait ErasedSourceBridge { - fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error>; + fn erased_visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>; fn erased_get<'kvs>(&'kvs self, key: Key) -> Option>; } @@ -1262,7 +1252,7 @@ impl ErasedSourceBridge for KVS where KVS: Source + ?Sized, { - fn erased_visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + fn erased_visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> { self.visit(visitor) } @@ -1307,25 +1297,25 @@ When `std` is available, `Source` is implemented for some standard collections t ```rust impl Source for Box where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> { (**self).visit(visitor) } } impl Source for Arc where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> { (**self).visit(visitor) } } impl Source for Rc where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> { (**self).visit(visitor) } } impl Source for Vec where KVS: Source { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> { + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> { self.as_slice().visit(visitor) } } @@ -1335,7 +1325,7 @@ where K: Borrow + Ord, V: ToValue, { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> { for (k, v) in self { visitor.visit_pair(k.borrow().to_key(), v.to_value())?; @@ -1357,7 +1347,7 @@ where K: Borrow + Eq + Hash, V: ToValue, { - fn visit<'kvs>(&'kvs self, visitor: &mut dyn Visitor<'kvs>) -> Result<(), Error> + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> { for (k, v) in self { visitor.visit_pair(k.borrow().to_key(), v.to_value())?; @@ -1444,7 +1434,6 @@ Structured key-value pairs can be set on a `RecordBuilder`: ```rust impl<'a> RecordBuilder<'a> { - /// Set key values pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a> { self.record.kvs = kvs; self @@ -1463,15 +1452,137 @@ pub struct Record<'a> { } impl<'a> Record<'a> { - /// The key value pairs attached to this record. - /// - /// Pairs aren't guaranteed to be unique (the same key may be repeated with different values). pub fn key_values(&self) -> ErasedSource { self.kvs.clone() } } ``` +## A minimal key-values API + +The following API is just the fundamental pieces of what's proposed by this RFC. Everything else could be implemented on top of this subset without introducing breakage. It also offers the freedom to move in a different direction entirely: + +```rust +impl<'a> RecordBuilder<'a> { + pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a> { + self.record.kvs = kvs; + self + } +} + +#[derive(Clone, Debug)] +pub struct Record<'a> { + .. + + kvs: ErasedSource<'a>, +} + +impl<'a> Record<'a> { + pub fn key_values(&self) -> ErasedSource { + self.kvs.clone() + } +} + +pub struct Error(Inner); + +impl Error { + pub fn msg(msg: &'static str) -> Self { + .. + } +} + +impl Debug for Error { + .. +} + +impl Display for Error { + .. +} + +impl From for Error { + .. +} + +impl From for fmt::Error { + .. +} + +#[cfg(feature = "std")] +mod std_support { + impl Error { + pub fn custom(err: impl fmt::Display) -> Self { + .. + } + } + + impl From for Error { + .. + } + + impl From for io::Error { + .. + } + + impl error::Error for Error { + .. + } +} + +pub struct Value<'v>(_); + +impl<'v> Debug for Value<'v> { + .. +} + +impl<'v> Display for Value<'v> { + .. +} + +pub trait ToValue { + fn to_value(&self) -> Value; +} + +pub struct Key<'k>(_); + +impl<'k> Key<'k> { + pub fn from_str(key: &'k (impl Borrow + ?Sized)) -> Self { + .. + } + + pub fn as_str(&self) -> &str { + .. + } +} + +pub trait ToKey { + fn to_key(&self) -> Key; +} + +pub trait Source { + fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>; + + fn erase(&self) -> ErasedSource + where + Self: Sized, + { + .. + } +} + +pub struct ErasedSource<'a>(_); + +pub trait Visitor<'kvs> { + fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; +} + +impl<'a, 'kvs, T: ?Sized> Visitor<'kvs> for &'a mut T +where + T: Visitor<'kvs> +{ + .. +} +``` + ## The `log!` macros The `log!` macro will initially support a fairly spartan syntax for capturing structured data. The current `log!` macro looks like this: @@ -1547,19 +1658,21 @@ Will expand to something like: # Drawbacks, rationale, and alternatives [drawbacks]: #drawbacks -Structured logging is a non-trivial feature to support. It adds complexity and overhead to the `log` crate. +## Supporting structured logging at all + +Structured logging is a non-trivial feature to support. It adds complexity and overhead to the `log` crate. The alternative, not supporting structured logging, is not a suitable long-term solution for `log` unless we plan to eventually deprecate it. ## Internalizing `sval` and `serde` -Values captured from any one supported framework can be represented by any other. That means a value can be captured in terms of `sval` and consumed in terms of `serde`, with its underlying structure retained. This is done through a one-to-one integration from each framework to each other framework. +Values captured from any one supported framework can be represented by any other. That means a value can be captured by `sval` can be consumed by `serde`, with its underlying structure retained. This is done through an internal one-to-one integration from each framework to each other framework. ### Drawbacks The one-to-one bridge between serialization frameworks within `log` makes the effort needed to support them increase exponentially with each addition, and discourages it from supporting more than a few. -It also introduces direct coupling between `log` and these frameworks. For `sval` specifically, this is risky because it's not currently stable. Breaking changes are a possibility. +It also introduces direct coupling between `log` and these frameworks. For `sval` specifically, this is risky because it's not currently stable and breaking changes are a possibility. -The mechanism suggested in this RFC for erasing values in `Value::from_any` relies on unsafe code. It's the same as what's used in `std::fmt`, but that machinery isn't directly exposed to callers outside of unstable features. +The mechanism suggested in this RFC for erasing values in `Value::from_any` relies on unsafe code. It's the same as what's used in `std::fmt`, but in `std::fmt` the machinery isn't directly exposed to callers outside of unstable features. ### Alternatives @@ -1580,11 +1693,11 @@ pub trait Visitor { } ``` -This is fairly simple for primitive types like integers and strings, but becomes much more involved when dealing with complex values likes maps and sequences. A serialization framework needs to do more than just provide a contract, its API needs to work to support implementations on either side of that contract. Maintaining a useful serialization framework is a distraction for `log`. That's why the `sval` library was created; to manage the necessary complexity of building a serialization framework that's suitable for structured logging externally from the `log` crate. +This is fairly straightforward for primitive types like integers and strings, but becomes much more involved when dealing with complex values likes maps and sequences. A serialization framework needs to do more than just provide a contract, its API needs to work to support implementations on either side of that contract, otherwise it won't gain adoption. -So the public common serialization contract in `log` is effectively to integrate with one of a few fundamental frameworks. +Maintaining a useful serialization framework is a distraction for `log`. Serialization of structured values is a complex, necessary, but not primary function of `log`, so it should avoid owning that contract and the baggage that comes along with it if it can. That's why the `sval` library was created; to manage the necessary complexity of building a serialization framework that's suitable for structured logging externally from the `log` crate. -Within the `log` crate, internalizing fundamental serialization frameworks reduces the effort needed from building a complete framework down to shimming an existing framework. The effort of managing breaking changes in supported serialization frameworks isn't less than the effort of managing breaking changes in a common contract provided by `log`. The owner of that contract, whether it's `log` or `serde` or `sval`, has to consider the churn introduced by breakage. Serialization of structured values is a complex, necessary, but not primary feature of `log`, so if it should avoid owning that contract and the baggage that comes along with it if it can. +Within the `log` crate itself, internalizing fundamental serialization frameworks reduces the effort needed from building a complete framework down to shimming an existing framework. These shims would exist in the wider `log` ecosystem in either case. The effort of managing breaking changes in supported serialization frameworks isn't less than the effort of managing breaking changes in a common contract provided by `log`. The owner of that contract, whether it's `log` or `serde` or `sval`, has to consider the churn introduced by breakage. As a downstream consumer of that breakage, `log` is in the same boat as its consumers. # Prior art [prior-art]: #prior-art From 9b1c7040d8600594cdb3f29e21f95995d420c54c Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Wed, 30 Jan 2019 10:23:06 +1000 Subject: [PATCH 11/20] flesh out drawbacks and alternatives a bit more --- rfcs/0000-structured-logging.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 11453a07e..1a20b5358 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -1666,6 +1666,8 @@ Structured logging is a non-trivial feature to support. It adds complexity and o Values captured from any one supported framework can be represented by any other. That means a value can be captured by `sval` can be consumed by `serde`, with its underlying structure retained. This is done through an internal one-to-one integration from each framework to each other framework. +The choice of `sval` as a supported framework is because it's purpose-built for serializing values in structured logging. The choice of `serde` as a supported framework is because it's the de-facto standard in the Rust ecosystem and already used within `log`. + ### Drawbacks The one-to-one bridge between serialization frameworks within `log` makes the effort needed to support them increase exponentially with each addition, and discourages it from supporting more than a few. @@ -1676,6 +1678,8 @@ The mechanism suggested in this RFC for erasing values in `Value::from_any` reli ### Alternatives +#### Build a serialization contract in `log` + Instead of internalizing a few serialization frameworks, `log` could provide a public common contract for them to conform to: ```rust @@ -1693,12 +1697,18 @@ pub trait Visitor { } ``` -This is fairly straightforward for primitive types like integers and strings, but becomes much more involved when dealing with complex values likes maps and sequences. A serialization framework needs to do more than just provide a contract, its API needs to work to support implementations on either side of that contract, otherwise it won't gain adoption. +This is fairly straightforward for primitive types like integers and strings, but becomes much more involved when dealing with complex values likes maps and sequences. Not supporting these complex structures is limiting, and reduces `log`s interoperability with other frameworks that do. A serialization framework needs to do more than just provide a contract, its API needs to work to support implementations on either side of that contract, otherwise it won't gain adoption. Maintaining a useful serialization framework is a distraction for `log`. Serialization of structured values is a complex, necessary, but not primary function of `log`, so it should avoid owning that contract and the baggage that comes along with it if it can. That's why the `sval` library was created; to manage the necessary complexity of building a serialization framework that's suitable for structured logging externally from the `log` crate. Within the `log` crate itself, internalizing fundamental serialization frameworks reduces the effort needed from building a complete framework down to shimming an existing framework. These shims would exist in the wider `log` ecosystem in either case. The effort of managing breaking changes in supported serialization frameworks isn't less than the effort of managing breaking changes in a common contract provided by `log`. The owner of that contract, whether it's `log` or `serde` or `sval`, has to consider the churn introduced by breakage. As a downstream consumer of that breakage, `log` is in the same boat as its consumers. +#### Just pick an existing framework + +Instead of building a common shim around several serialization frameworks, `log` could just pick one and bake it in directly. This would have the benefit of offering the best end-user experience for existing users of that framework when interacting with the `log!` macros and `Source`s. It also means accepting the trade-offs that framework makes. For `serde`, that means requiring the standard library for boxing values. For `sval`, that means accepting churn as it matures. + +The API proposed using the `Value` type as an opaque container along with APIs for specific frameworks is a reasonable middle-ground between baking in a specific framework and only offering a contract without any direct support. It gives producers of structured data a way to plug their data using a framework of choice into a `Value`, either directly or through compatibility with a supported framework. It gives consumers of structured data a first-class experience for plugging `Value`s into their framework of choice by deriving the appropriate traits. It also gives `log` room to add and deprecate support for frameworks if mindshare shifts in the future. + # Prior art [prior-art]: #prior-art From 69b7d340d21380750cdeadf9a3743f4ce8e6ff9b Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Wed, 30 Jan 2019 10:40:27 +1000 Subject: [PATCH 12/20] fill in missing ToValue impls --- rfcs/0000-structured-logging.md | 48 ++++++++++++++++++++++++++++++++- 1 file changed, 47 insertions(+), 1 deletion(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 1a20b5358..124d01d81 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -873,7 +873,7 @@ It's the trait bound that values passed as structured data to the `log!` macros #### Implementors -`ToValue` is implemented for fundamental primitive types from the standard library: +`ToValue` is implemented for fundamental primitive types: ```rust impl<'v> ToValue for Value<'v> { @@ -986,6 +986,52 @@ impl<'a> ToValue for &'a str { Value::from_any(self, |from, v| from.str(*v)) } } + +impl<'a> ToValue for &'a Arguments { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.debug(*v)) + } +} +``` + +When `std` is available, `ToValue` is implemented for additional types: + +```rust +impl ToValue for Box where T: ToValue { + fn to_value(&self) -> Value { + (**self).to_value() + } +} + +impl ToValue for Arc where KVS: ToValue { + fn to_value(&self) -> Value { + (**self).to_value() + } +} + +impl ToValue for Rc where KVS: ToValue { + fn to_value(&self) -> Value { + (**self).to_value() + } +} + +impl ToValue for String { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.str(*v)) + } +} + +impl<'a> ToValue for &'a Path { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.debug(*v)) + } +} + +impl ToValue for PathBuf { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.debug(*v)) + } +} ``` ### `Key` From 6cb5a265fa07fc48c31b564f4ddf3442dffcfeef Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Wed, 30 Jan 2019 12:56:40 +1000 Subject: [PATCH 13/20] fix up some inconsistencies --- rfcs/0000-structured-logging.md | 103 ++++++++++++++++++++++---------- 1 file changed, 73 insertions(+), 30 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 124d01d81..090acfbdb 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -136,7 +136,7 @@ A value can be captured as a structured value in a log record if it implements t ```rust pub trait ToValue { - fn to_value(&self) -> Value; + fn to_value<'v>(&'v self) -> Value<'v>; } ``` @@ -190,7 +190,6 @@ Initially, that means a fixed set of primitive types from the standard library: - Unsigned integers: `u8`, `u16`, `u32`, `u64`, `u128` - Signed integers: `i8`, `i16`, `i32`, `i64`, `i128` - Strings: `&str`, `String` -- Slices: `&[T]`, `Vec` - Paths: `&Path`, `PathBuf` - Special types: `Option`, `&T`, and `()`. @@ -220,14 +219,13 @@ impl FromAny { .. } + // Not publicly exposed. Used by internal implementations fn u64(v: u64) -> Result<(), Error> { .. } } ``` -This machinery is very similar to the internals of `std::fmt`. - Only being able to log primitive types from the standard library is a bit limiting though. What if `correlation_id` is a `uuid::Uuid`, and `user` is a struct, `User`, with its own fields? #### Implementing `ToValue` for a simple value @@ -237,12 +235,22 @@ A newtype structure like `uuid::Uuid` could implement the `ToValue` trait direct ```rust impl ToValue for Uuid { fn to_value(&self) -> Value { - self.as_bytes().to_value() + self.as_u128().to_value() } } ``` -Alternatively, `uuid::Uuid` could provide a nicer implementation using the `Debug` implementation of its hyphenated format: +Alternatively, `uuid::Uuid` could provide a nicer implementation using its `Debug` implementation: + +```rust +impl ToValue for Uuid { + fn to_value(&self) -> Value { + Value::from_debug(self) + } +} +``` + +Finally, it could use the `Debug` implementation of its hyphenated format: ```rust impl ToValue for Uuid { @@ -252,7 +260,7 @@ impl ToValue for Uuid { } ``` -There's some subtlety in this second implementation. The actual value whose structure is captured is not the `&'v Uuid`, it's the owned `ToHyphenated<'v>`. This is why `Value::from_any` uses a separate function for capturing the structure of its values that doesn't depend on the lifetime of the given `&'v T`. It lets us capture a borrowed `Uuid` with the right lifetime `'v`, but materialize an owned `ToHyphenated` with the structure we want. +There's some subtlety in this last implementation. The actual value whose structure is captured is not the `&'v Uuid`, it's the owned `ToHyphenated<'v>`. This is why `Value::from_any` uses a separate function for capturing the structure of its values that doesn't depend on the lifetime of the given `&'v T`. It lets us capture a borrowed `Uuid` with the right lifetime `'v`, but materialize an owned `ToHyphenated` with the structure we want. #### Implementing `ToValue` for a complex value @@ -780,9 +788,17 @@ impl<'a> FromAny<'a> { .. } + fn u128(self, v: u128) -> Result<(), Error> { + .. + } + fn i64(self, v: i64) -> Result<(), Error> { .. } + + fn i128(self, v: i128) -> Result<(), Error> { + .. + } fn f64(self, v: f64) -> Result<(), Error> { .. @@ -871,6 +887,10 @@ pub trait ToValue { It's the trait bound that values passed as structured data to the `log!` macros need to satisfy. +#### Object safety + +The `ToValue` trait is object-safe. + #### Implementors `ToValue` is implemented for fundamental primitive types: @@ -921,6 +941,12 @@ impl ToValue for u64 { } } +impl ToValue for u128 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.u128(*v)) + } +} + impl ToValue for i8 { fn to_value(&self) -> Value { Value::from_any(self, |from, v| from.i64(*v as i64)) @@ -945,6 +971,12 @@ impl ToValue for i64 { } } +impl ToValue for i128 { + fn to_value(&self) -> Value { + Value::from_any(self, |from, v| from.i128(*v)) + } +} + impl ToValue for f32 { fn to_value(&self) -> Value { Value::from_any(self, |from, v| from.f64(*v as f64)) @@ -1034,9 +1066,11 @@ impl ToValue for PathBuf { } ``` +Other implementations for `std` types can be added in the same fashion. + ### `Key` -A `Key` is a short-lived structure that can be represented as a UTF-8 string. This might be possible without allocating, or it might require a destination to write into: +A `Key` is a short-lived structure that can be represented as a UTF-8 string: ```rust pub struct Key<'k>(_); @@ -1088,26 +1122,31 @@ impl<'k> Debug for Key<'k> { impl<'k> Display for Key<'k> { .. } +``` -#[cfg(feature = "std")] -mod std_support { - impl<'k> Key<'k> { - pub fn from_owned(key: impl Into) -> Self { - .. - } - } +When `std` is available, a `Key` can also contain an owned `String`: - impl ToKey for String { - fn to_key(&self) -> Key { - Key::from_str(self, None) - } +```rust +impl<'k> Key<'k> { + pub fn from_owned(key: impl Into) -> Self { + .. } +} - impl<'k> From for Key<'k> { - .. +impl ToKey for String { + fn to_key(&self) -> Key { + Key::from_str(self, None) } } +impl<'k> From for Key<'k> { + .. +} +``` + +When the `kv_sval` or `kv_serde` features are enabled, a `Key` can be serialized using `sval` or `serde`: + +```rust #[cfg(feature = "kv_sval")] mod sval_support { impl<'k> sval::Value for Key<'k> { @@ -1135,7 +1174,7 @@ The `Key` type is probably `Send` and `Sync`, but that's not guaranteed. #### Extensibility: Adding an index to keys -The `Key` type could be extended to hold an optional index into a source. This could be used to retrieve a specific key-value pair more efficiently than scanning. +The `Key` type could be extended to hold an optional index into a source. This could be used to retrieve a specific key-value pair more efficiently than scanning. There's some subtlety to consider though when the sources of keys are combined in various ways that might invalidate a previous index. ### `ToKey` @@ -1147,6 +1186,10 @@ pub trait ToKey { } ``` +#### Object safety + +The `ToKey` trait is object-safe. + #### Implementors The `ToKey` trait is implemented for common string containers in the standard library: @@ -1176,7 +1219,7 @@ impl<'k> ToKey for Key<'k> { ### `Source` -The `Source` trait is a bit like `std::iter::Iterator`. It gives us a way to inspect some arbitrary collection of key-value pairs using an object-safe visitor pattern: +The `Source` trait is a bit like `std::iter::Iterator` for key-value pairs. It gives us a way to inspect some arbitrary collection of key-value pairs using an object-safe visitor pattern: ```rust pub trait Source { @@ -1242,7 +1285,7 @@ pub trait Source { } ``` -`Source` doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. +The `Source` trait is the main extensibility point for structured logging in the `log` crate that's used by the `Record` type. It doesn't make any assumptions about how many key-value pairs it contains or how they're visited. That means the visitor may observe keys in any order, and observe the same key multiple times. #### Adapters @@ -1260,7 +1303,7 @@ None of these methods are required for the core API. They're helpful tools for w #### Object safety -`Source` is not object-safe because of the provided adapter methods not being object-safe. The only required method, `visit`, is safe though, so an object-safe version of `Source` that forwards this method can be reasonably written in a similar way to the object-safe `ErasedVisit`: +`Source` is not object-safe because of the provided adapter methods not being object-safe. The only required method, `visit`, is safe though, so an object-safe version of `Source` that forwards this method can be reasonably written: ```rust #[derive(Clone, Copy)] @@ -1415,9 +1458,9 @@ The `BTreeMap` and `HashMap` implementations provide more efficient implementati #### Extensibility: Sending `Source`s between threads -Before a record could be processed on a background thread it would need to be converted into some owned variant. The `Source` trait is the point where having some way to convert from a borrowed to an owned value would make the most sense because that's where the knowledge of the underlying key-value storage is. +Implementations of `Log` might want to offload the processing of records to some background thread. The record would need to be converted into some owned representation before being sent across threads. This is straightforward for the existing borrowed string metadata on a record, but less so for any structured data. The `Source` trait is the point where having some way to convert from a borrowed to an owned set of key-value pairs would make the most sense because that's where the knowledge of the underlying key-value storage is. -A new provided method could be added to the `Source` trait that allowed it to be converted into an owned variant that is `Send + Sync + 'static`: +A new provided method could be added to the `Source` trait that allows it to be converted into an owned variant that is `Send + Sync + 'static`: ```rust pub trait Source { @@ -1429,7 +1472,7 @@ pub trait Source { } #[derive(Clone)] -pub struct OwnedSource(Arc); +pub struct OwnedSource(Arc); impl OwnedSource { pub fn new(impl Into>) -> Self { @@ -1468,7 +1511,7 @@ A `Visitor` may serialize the keys and values as it sees them. It may also do ot #### Implementors -There aren't any public implementors of `Visitor` in the `log` crate. Other crates that use key-value pairs will implement `Visitor`. +There aren't any public implementors of `Visitor` in the `log` crate. Other crates that use key-value pairs will implement `Visitor`, or use the adapter methods on `Source` and never need to touch `Visitor`s directly. #### Object safety @@ -1476,7 +1519,7 @@ The `Visitor` trait is object-safe. ### `Record` and `RecordBuilder` -Structured key-value pairs can be set on a `RecordBuilder`: +Structured key-value pairs can be set on a `RecordBuilder` as an implementation of a `Source`: ```rust impl<'a> RecordBuilder<'a> { From 50a7408374e408cfce5f09f8853661369ec15c68 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Wed, 30 Jan 2019 15:06:05 +1000 Subject: [PATCH 14/20] clean up minimal initial impl --- rfcs/0000-structured-logging.md | 50 ++++++++++++--------------------- 1 file changed, 18 insertions(+), 32 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 090acfbdb..88c608bdc 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -1554,25 +1554,17 @@ The following API is just the fundamental pieces of what's proposed by this RFC. ```rust impl<'a> RecordBuilder<'a> { pub fn key_values(&mut self, kvs: ErasedSource<'a>) -> &mut RecordBuilder<'a> { - self.record.kvs = kvs; - self + .. } } -#[derive(Clone, Debug)] -pub struct Record<'a> { - .. - - kvs: ErasedSource<'a>, -} - impl<'a> Record<'a> { pub fn key_values(&self) -> ErasedSource { - self.kvs.clone() + .. } } -pub struct Error(Inner); +pub struct Error(_); impl Error { pub fn msg(msg: &'static str) -> Self { @@ -1597,24 +1589,25 @@ impl From for fmt::Error { } #[cfg(feature = "std")] -mod std_support { - impl Error { - pub fn custom(err: impl fmt::Display) -> Self { - .. - } - } - - impl From for Error { +impl Error { + pub fn custom(err: impl fmt::Display) -> Self { .. } +} - impl From for io::Error { - .. - } +#[cfg(feature = "std")] +impl From for Error { + .. +} - impl error::Error for Error { - .. - } +#[cfg(feature = "std")] +impl From for io::Error { + .. +} + +#[cfg(feature = "std")] +impl error::Error for Error { + .. } pub struct Value<'v>(_); @@ -1663,13 +1656,6 @@ pub struct ErasedSource<'a>(_); pub trait Visitor<'kvs> { fn visit_pair(&mut self, k: Key<'kvs>, v: Value<'kvs>) -> Result<(), Error>; } - -impl<'a, 'kvs, T: ?Sized> Visitor<'kvs> for &'a mut T -where - T: Visitor<'kvs> -{ - .. -} ``` ## The `log!` macros From 8e4758b9854e464ae48aeec58f5c597a0d97a3e3 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Mon, 18 Feb 2019 14:34:39 +1000 Subject: [PATCH 15/20] punt on macro syntax as something to explore externally --- rfcs/0000-structured-logging.md | 170 +------------------------------- 1 file changed, 5 insertions(+), 165 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 88c608bdc..47a97304b 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -98,37 +98,7 @@ This section introduces `log`'s structured logging API through a tour of how str ## Logging structured key-value pairs -Structured logging is supported in `log` by allowing typed key-value pairs to be associated with a log record. A `;` separates structured key-value pairs from other data that's interpolated into the message: - -```rust -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - correlation = correlation_id, - user -); -``` - -Any `value` or `key = value` expressions before the `;` in the macro will be interpolated into the message as unstructured text using `std::fmt`. This is the `log!` macro we have today. - -Any `value` or `key = value` expressions after the `;` will be captured as structured key-value pairs. These structured key-value pairs can be inspected or serialized, retaining some notion of their original type. That means in the above example, the `message` pair is unstructured, and the `correlation` and `user` pairs are structured: - -``` -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - ^^^^^^^^^^^^^^^^^^^ - unstructured - - correlation = correlation_id, - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - structured - - user - ^^^^ - structured -); -``` +Structured logging is supported in `log` by allowing typed key-value pairs to be associated with a log record. This support isn't surfaced in the `log!` macros initially, so you'll need to use a framework like `slog` or `tokio-trace` with `log` integration, or an alternative implementation of the `log!` macros to capture structured key-value pairs. ### What can be captured as a structured value? @@ -153,36 +123,6 @@ impl<'v> Debug for Value<'v> { We'll look at `Value` in more detail later. For now, we can think of it as a container that normalizes capturing and emitting the structure of values. -So, in the example from before: - -```rust -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - correlation = correlation_id, - user -); -``` - -the `correlation_id` and `user` pairs can be captured as structured values if they implement the `ToValue` trait: - -``` -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - ^^^^^^^^^^^^^^^^^^^ - impl Display - - correlation = correlation_id, - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - impl ToValue - - user - ^^^^ - impl ToValue -); -``` - Initially, that means a fixed set of primitive types from the standard library: - Standard formats: `Arguments` @@ -308,40 +248,6 @@ impl ToValue for User { } ``` -#### Capturing values without implementing `ToValue` - -Instead of implementing `ToValue` on types throughout the ecosystem, callers of the `log!` macros could instead create ad-hoc `Value`s from their data at the callsite: - -```rust -use log::key_values::Value; - -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - correlation = Value::from_serde(correlation_id), - user = Value::from_sval(user), -); -``` - -In this example, `correlation_id` and `user` don't need to implement any traits from `log`. Instead, they need to implement the corresponding trait from `sval` or `serde`: - -``` -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - - correlation = Value::from_serde(correlation_id), - ^^^^^^^^^^^^^^ - impl serde::Serialize + Debug - - user = Value::from_sval(user), - ^^^^ - impl sval::Value + Debug -); -``` - -Having to decorate every value in every `log!` macro is not ideal for users of the `log` crate, but it does open the door for alternative implementations of the `log!` macros to be more opinionated about what kinds of structured values they'll accept by default. - ## Supporting key-value pairs in `Log` implementations Capturing structured logs is only half the story. Implementors of the `Log` trait also need to be able to work with any key-value pairs associated with a log record. Key-value pairs are accessible on a log record through the `Record::key_values` method: @@ -599,7 +505,7 @@ Don't create a new serialization API that requires `log` to become a public depe ### Support arbitrary producers and arbitrary consumers -Provide an API that's suitable for two independent logging frameworks to integrate through if they want. Producers of structured data and consumers of structured data should be able to use different serialization frameworks opaquely and still get good results. As an example, a caller of `info!` should be able to log a map that implements `sval::Value`, and the implementor of the receiving `Log` trait should be able to format that map using `serde::Serialize`. +Provide an API that's suitable for two independent logging frameworks to integrate through if they want. Producers of structured data and consumers of structured data should be able to use different serialization frameworks opaquely and still get good results. As an example, a producer of a `log::Record` should be able to log a map that implements `sval::Value`, and the implementor of the receiving `Log` trait should be able to format that map using `serde::Serialize`. ### Remain object safe @@ -885,7 +791,7 @@ pub trait ToValue { } ``` -It's the trait bound that values passed as structured data to the `log!` macros need to satisfy. +It's the generic trait bound that macros capturing structured values can require. #### Object safety @@ -1660,75 +1566,9 @@ pub trait Visitor<'kvs> { ## The `log!` macros -The `log!` macro will initially support a fairly spartan syntax for capturing structured data. The current `log!` macro looks like this: - -```rust -log!(); -``` - -This RFC proposes an additional semi-colon-separated part of the macro for capturing key-value pairs: - -```rust -log!( ; ) -``` - -The `;` and structured values are optional. If they're not present then the behavior of the `log!` macro is the same as it is today. - -As an example, this is what a `log!` statement containing structured key-value pairs could look like: - -```rust -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - correlation = correlation_id, - user = user -); -``` - -There's a *big* design space around the syntax for capturing log records we could explore, especially when you consider procedural macros. The syntax proposed here for the `log!` macro is not designed to be really ergonomic. It's designed to be *ok*, and to encourage an exploration of the design space by offering a consistent base that other macros could build off. - -Having said that, there are a few nonintrusive quality-of-life features that make the `log!` macros nicer to use with structured data. - -### Expansion - -Structured key-value pairs in the `log!` macro expand to statements that borrow from their environment. - -```rust -info!( - "This is the rendered {message}. It is not structured", - message = "message"; - correlation = correlation_id, - user = user -); -``` - -Will expand to something like: +The existing `log!` macros will not be changed in the initial implementation of structured logging. Instead, `log` will rely on new `log!` macro implementations and existing structured frameworks like `slog` and `tokio-trace` to capture structured key-value pairs. -```rust -{ - let lvl = log::Level::Info; - - if lvl <= ::STATIC_MAX_LEVEL && lvl <= ::max_level() { - let correlation &correlation_id; - let user = &user; - - let kvs: &[(&str, &dyn::key_values::value::ToValue)] = - &[("correlation", &correlation), ("user", &user)]; - - ::__private_api_log( - ::std::fmt::Arguments::new_v1( - &["This is the rendered ", ". It is not structured"], - &match (&"message",) { - (arg0,) => [::std::fmt::ArgumentV1::new(arg0, ::std::fmt::Display::fmt)], - }, - ), - lvl, - &("bin", "mod", "mod.rs", 13u32), - &kvs, - ); - } -}; -``` +It's expected that an external library will be created to explore new implementations of the `log!` macros that are structured by design, rather than attempting to graft structured logging support onto the existing macros. The result of this work should eventually find its way back into the `log` crate. # Drawbacks, rationale, and alternatives [drawbacks]: #drawbacks From f157909f937b54f5910a29e1830df643f793c2b8 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Tue, 19 Feb 2019 11:19:57 +1000 Subject: [PATCH 16/20] we don't need ToKey or ToValue in a minimal API without macros --- rfcs/0000-structured-logging.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index 47a97304b..c1f45f861 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -1317,7 +1317,7 @@ impl Source for Vec where KVS: Source { impl Source for BTreeMap where - K: Borrow + Ord, + K: ToKey + Borrow + Ord, V: ToValue, { fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> @@ -1339,7 +1339,7 @@ where impl Source for HashMap where - K: Borrow + Eq + Hash, + K: ToKey + Borrow + Eq + Hash, V: ToValue, { fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error> @@ -1518,6 +1518,12 @@ impl error::Error for Error { pub struct Value<'v>(_); +impl<'v> Value<'v> { + pub fn from_debug(value: &'v impl Debug) -> Self { + .. + } +} + impl<'v> Debug for Value<'v> { .. } @@ -1526,10 +1532,6 @@ impl<'v> Display for Value<'v> { .. } -pub trait ToValue { - fn to_value(&self) -> Value; -} - pub struct Key<'k>(_); impl<'k> Key<'k> { @@ -1542,10 +1544,6 @@ impl<'k> Key<'k> { } } -pub trait ToKey { - fn to_key(&self) -> Key; -} - pub trait Source { fn visit<'kvs>(&'kvs self, visitor: &mut impl Visitor<'kvs>) -> Result<(), Error>; From d4c65bf773816405dd6dad9853c21e2236f6027a Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Tue, 26 Feb 2019 09:29:32 +1000 Subject: [PATCH 17/20] flesh out motivations for some of the Source adapters --- rfcs/0000-structured-logging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0000-structured-logging.md b/rfcs/0000-structured-logging.md index c1f45f861..467e6ee38 100644 --- a/rfcs/0000-structured-logging.md +++ b/rfcs/0000-structured-logging.md @@ -1199,8 +1199,8 @@ Some useful adapters exist as provided methods on the `Source` trait. They're si - `by_ref` to get a reference to a `Source` within a method chain. - `chain` to concatenate one source with another. This is useful for composing implementations of `Log` together for contextual logging. -- `get` to try find the value associated with a key. -- `for_each` to execute some closure over all key-value pairs. This is a convenient way to do something with each key-value pair without having to create and implement a `Visitor`. +- `get` to try find the value associated with a key. This is useful for well-defined key-value pairs that a framework built over `log` might want to provide, like timestamps or message templates. +- `for_each` to execute some closure over all key-value pairs. This is a convenient way to do something with each key-value pair without having to create and implement a `Visitor`. One potential downside of `for_each` is the `Result` return value, which seems surprising when the closure itself can't fail. The `Source::for_each` call might itself fail if the underlying `visit` call fails when iterating over its key-value pairs. This shouldn't be common though, so when paired with `try_for_each`, it might be reasonable to make `for_each` return a `()` and rely on `try_for_each` for surfacing any fallibility. - `try_for_each` is like `for_each`, but takes a fallible closure. - `as_map` to get a serializable map. This is a convenient way to serialize key-value pairs without having to create and implement a `Visitor`. - `as_seq` is like `as_map`, but for serializing as a sequence of tuples. From ac7efd7fbb35ded6d55aeb1e387739ae656f7496 Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Mon, 11 Mar 2019 09:09:51 +1000 Subject: [PATCH 18/20] rename rfc doc and exclude rfcs from pkg --- Cargo.toml | 2 ++ rfcs/{0000-structured-logging.md => 0296-structured-logging.md} | 0 2 files changed, 2 insertions(+) rename rfcs/{0000-structured-logging.md => 0296-structured-logging.md} (100%) diff --git a/Cargo.toml b/Cargo.toml index 4048ee686..665f795a1 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -14,6 +14,8 @@ A lightweight logging facade for Rust categories = ["development-tools::debugging"] keywords = ["logging"] +exclude = ["rfcs/**/*"] + [package.metadata.docs.rs] features = ["std", "serde"] diff --git a/rfcs/0000-structured-logging.md b/rfcs/0296-structured-logging.md similarity index 100% rename from rfcs/0000-structured-logging.md rename to rfcs/0296-structured-logging.md From a8a8f693d25b8aaea08dd5a2569cb8b2b73d064b Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Mon, 11 Mar 2019 09:13:24 +1000 Subject: [PATCH 19/20] Update 0296-structured-logging.md --- rfcs/0296-structured-logging.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rfcs/0296-structured-logging.md b/rfcs/0296-structured-logging.md index 467e6ee38..de33189fa 100644 --- a/rfcs/0296-structured-logging.md +++ b/rfcs/0296-structured-logging.md @@ -1,3 +1,7 @@ +- Feature Name: `structured_logging` +- Start Date: 2019-03-11 +- RFC PR: [log/rfcs#0296](https://github.com/rust-lang-nursery/log/pull/296) + # Summary [summary]: #summary From c50186cd1318513c5987b1583ab0cf04c881a48e Mon Sep 17 00:00:00 2001 From: Ashley Mannix Date: Mon, 11 Mar 2019 09:15:46 +1000 Subject: [PATCH 20/20] remove duplicate key --- Cargo.toml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/Cargo.toml b/Cargo.toml index b63bc7e4e..5b1d59cb3 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -13,9 +13,7 @@ A lightweight logging facade for Rust """ categories = ["development-tools::debugging"] keywords = ["logging"] -exclude = ["/.travis.yml", "/appveyor.yml"] - -exclude = ["rfcs/**/*"] +exclude = ["rfcs/**/*", "/.travis.yml", "/appveyor.yml"] [package.metadata.docs.rs] features = ["std", "serde"]