key_value capable log macros

[yoshuawuyts]

Continuation of #328 (comment).

Description

kv_unstable is available in log and provides a key_values field on the Record to capture key-value pairs. However currently in order to write data to loggers Record types need to be constructed manually, rather than being able to use the log macros. As per RFC#296 we should experiment with logging macros out of tree, and use it to inform how to add key-value logging to log's macros.

Constraints

[tbi]

@KodrAus could you perhaps clarify what the constraints are?

Prior Art

Slog

The slog crate uses ;"key" => "value" pairs in the macros:

#[macro_use]
extern crate slog;

fn main() {
    let drain = slog::Discard;
    let root = slog::Logger::root(
        drain,
        o!("key1" => "value1", "key2" => "value2")
    );
    info!(root, "test info log"; "log-key" => true);
}

Tracing

tracing supports two kinds of log styles: "a message + key-value pairs", and "only key-value pairs":

use tracing::field;

let addr = Ipv4Addr::new(127, 0, 0, 1);
let conn = Connection { port: 40, speed: 3.20 };

info!({ port = conn.port }, "connected to {}", addr);
info!(
    target: "connection_events",
    ip = %addr,
    conn.port,
    ?conn.speed,
);

Update this section with more examples

Creating a manual record

For reference, this is how to create a manual record with key-value pairs:

use log::kv::{Error, Visitor, Source};

struct Pair {
    val: String
}

impl Source for Pair {
    fn visit(&self, visitor: &mut dyn Visitor<'_>) -> Result<(), Error> {
        visitor.visit_pair("some key".into(), self.val.into())
    }
}

log::logger().log(&log::Record::builder()
    .args(format_args!("hello world!"))
    .key_values(&Pair { val: "some value".to_string() })
    .level(log::Level::Info)
    .target(module_path!())
    .module_path(Some(module_path!()))
    .file(Some(file!()))
    .line(Some(line!()))
    .build());

Conclusion

We should experiment with logging macros that can capture key-value pairs and report back with our findings in this issue. It'd be great if we could discuss their tradeoffs, and eventually come up with a design that's suitable for an RFC.

[Thomasdezeeuw]

Just an idea for alternative syntax: adding bracket around the key value pairs:

info(target: "some target", "some message: {}", msg_arg, {
    // And the one of the following.
    key1: value1,
    key1 = value1,
    key1 => value1,
})

Hopefully something like this will aid in formatting long lines, although I'm not sure how rustfmt would handle this.

[KodrAus]

I think our only real constraint is that existing log! statements should continue to work.

I was thinking about capturing key-value pairs implicitly from the format, so if they appear in the message then they appear as key-value pairs:

info!("connected to {service} at {uri}", service, uri);

however that could be a breaking change and would require procedural macro support.

There are other things the macros could do, like allow you to capture a std::error::Error explicitly in the warn! and error! macros.

[yoshuawuyts]

I've been thinking about this for a bit, and I think I like the direction @Thomasdezeeuw's proposed quite a lot. Specifically using : to create struct-like fields:

Example

info!("hello {}", "cats", {
    cat_1: "chashu",
    cat_2: "nori",
});

Something I particularly like about this is that it feels similar to the way we initialize structs right now. Except they're anonymous, and placed in-line.

Also I think the key-values should only be valid as the final argument to log macro invocations. I've been taking a closer look at existing alternatives lately, and it seems that at least some implementations allow key-values to be passed in multiple positions which has felt somewhat confusing to me.

Argument Detection

@KodrAus I've been thinking a about what you've proposed for a bit, and a problem I see is that while it makes it convenient to add kv arguments to the messages, it becomes problematic when certain arguments should not appear as key-value arguments.

I think it slightly boils down to a question of control. By manually passing arguments as key-value we optimize for control, at the expense of some convenience. But given how universal and flexible the log crate needs to be, I think that might be the right tradeoff here.

Though I'm keen to hear more about the automatic error capturing in macros? What would do you think that should look like? Are there perhaps experiments we could do with this?

Implementation

@Thomasdezeeuw would you be interested in writing a prototype of the key-value enabled log macros? If you don't have the time, perhaps @KodrAus or I could also take a shot at it, but it might perhaps be a bit longer (I'm for sure running at capacity right now).

---

I'm very excited about this overall; I've been using this feature in a few projects (including gotten a logger running in the browser w/ wasm) and it's been great! Thanks!

[Thomasdezeeuw]

@yoshuawuyts I'm not great at macros, but I'm willing to give a shot.

[Thomasdezeeuw]

I didn't get very far, see #346, I couldn't fix the ambiguity error. Maybe someone with a bit more Rust macro powers can?

[dtolnay]

Here is a proof of concept of @yoshuawuyts's preferred syntax as a tt muncher:

#[macro_export]
macro_rules! info {
    // info!("...")
    ($e:expr) => {
        $crate::info_impl!(($e));
    };

    // info!("...", args...)
    ($e:expr, $($rest:tt)*) => {
        $crate::info_impl!(($e) $($rest)*);
    };
}

#[macro_export]
#[doc(hidden)]
macro_rules! info_impl {
    // End of macro input
    (($($e:expr),*)) => {
        println!("msg={:?}", format!($($e),*));
    };

    // Trailing k-v pairs containing no trailing comma
    (($($e:expr),*) { $($key:ident : $value:expr),* }) => {
        println!("msg={:?}", format!($($e),*));
        $(
            println!("  {}={:?}", stringify!($key), $value);
        )*
    };

    // Trailing k-v pairs with trailing comma
    (($($e:expr),*) { $($key:ident : $value:expr,)* }) => {
        $crate::info_impl!(($($e),*) { $($key : $value),* });
    };

    // Last expression arg with no trailing comma
    (($($e:expr),*) $arg:expr) => {
        $crate::info_impl!(($($e,)* $arg));
    };

    // Expression arg
    (($($e:expr),*) $arg:expr, $($rest:tt)*) => {
        $crate::info_impl!(($($e,)* $arg) $($rest)*);
    };
}

fn main() {
    info!("hello");
    info!("hello",);
    info!("hello {}", "cats");
    info!("hello {}", "cats",);
    info!("hello {}", "cats", {
        cat_1: "chashu",
        cat_2: "nori",
    });
}

[Thomasdezeeuw]

@dtolnay that is awesome!

Is this the format we want to use?

[KodrAus]

Thanks @dtolnay!

@Thomasdezeeuw for the log crate itself I'd be on board with this syntax 👍 I don't think it breaks any existing consumers because the { $key = $value } matcher won't compile currently and something like info!("hello {}", { "cats" }); would continue to work as expected.

[dtolnay]

Future extensions I think we'll likely want:

• Field init shorthand (just like Field init shorthand rfcs#1682):

  let cat_1 = "chashu";
  let cat_2 = "nori";
  info!("hello {}", "cats", { cat_1, cat_2 });

• Conditional compilation support:

  info!("hello {}", "cats", {
      cat_1: "chashu",
      #[cfg(feature = "both_cats")]
      cat_2: "nori",
  });

[theduke]

A few additional things to consider:

• Literal strings as keys should definitely be supported, in case you need a non-identifier key (other languages, ...). This should not be a problem for the proposed syntax thanks to the :literal macro fragment.

• Maybe: dynamic keys?
  like: let key = format!("xyz{}", x); info!({ #key : 42 })

• Runtime-conditional keys in addition to compile-time conditionals (#[cfg(...)]).
  This could potentially be supported by "splatting" in a <K, V, T: IntoIter<Item = (K, V)>
  Something like: info!({ a: 33, ..map }) (this could potentially also replace the dynamic keys above)

[yoshuawuyts]

Yay, got a full impl working based on @dtolnay's parser, @Thomasdezeeuw's API proposal, and @hawkw's pointers on how the tracing's macros work!

• https://github.com/yoshuawuyts/kv-log-macro

The impl is modeled after log's internals, which means I can easily file a PR to add it log directly (similar to #346), which I'll do next. Thanks heaps everyone -- very excited this is working!!

Usage

use kv_log_macro::info;

fn main() {
    femme::start(log::LevelFilter::Info).unwrap();
    info!("hello");
    info!("hello",);
    info!("hello {}", "cats");
    info!("hello {}", "cats",);
    info!("hello {}", "cats", {
        cat_1: "chashu",
        cat_2: "nori",
    });
}

Screenshot