Add support for log4j2 property substitution and lookup variables

[fbaligand]

Hi,

EcsLayout component is great for log4j2.
But for now, we can't inject dynamic values in EcsLayout attributes.
For instance; serviceVersion and serviceNodeName

So it would be great to add support for log4j2 property substitution and lookup variables in EcsLayout attributes!

It would allow to write this EcsLayout configuration for instance:

<EcsLayout serviceName="myservice" serviceVersion="$${spring:project.version}" serviceNodeName="${env:HOSTNAME}"/>

[eyalkoren]

This sounds like an interesting feature, but not sure how soon we can prioritize it.
Would you like to give it a shot and try to implement yourself? If so, please create a PR. We still can't guarantee we would review that immediately, but for sure it would be prioritized higher as a PR.

Without looking into how it's implemented, I would first look for an access to the internal log4j2 code that parses, substitutes and does the lookup. I doubt if we would like to add the entire logic (and documentation) to do it within the ECS-logging library. Another hint if you try it out is that it will have to be compatible with all supported versions (2.6+).

[felixbarny]

As a workaround, you could set these properties as key/value pairs that already support lookups:

<EcsLayout serviceName="myservice">
  <KeyValuePair key="service.version" value="$${spring:project.version}"/>
  <KeyValuePair key="service.node.name" value="${env:HOSTNAME}"/>
</EcsLayout>

[fbaligand]

Great workaround @felixbarny!
Thanks for the share!

And thanks to have added this example into documentation!
https://www.elastic.co/guide/en/ecs-logging/java/master/setup.html

[kelunik]

Setting log4j2.formatMsgNoLookups is a (now discredited) mitigation for CVE-2021-45046. Lookups in the (user-provided) message have now been completely removed in log4j2 2.16, see also https://issues.apache.org/jira/browse/LOG4J2-3211.

However, setting log4j2.formatMsgNoLookups also disables lookups for <KeyValuePair /> defined in the configuration, because they're disabled here:

ecs-logging-java/log4j2-ecs-layout/src/main/java/co/elastic/logging/log4j2/EcsLayout.java

Line 177 in 260992b

} else if (valueNeedsLookup(additionalField.getValue()) && !FORMAT_MESSAGES_PATTERN_DISABLE_LOOKUPS) {

<KeyValuePair key="service.environment" value="${sys:environment}" />

This prevents lookups (i.e. ${...:...} – not %X etc.) from working even if defined in the configuration file.

What's the recommendation here? Simply not setting log4j2.formatMsgNoLookups anymore, because lookups have now been removed anyway? If so, why is this property still respected here?