Logging emitter to publish query and other metric events as valid json objects

[himanshug]

Description

This patch enables pushing druid metrics in log as valid json objects which are supported out of the box by various systems used for storing and searching logs. This aids us in investigating various issues in Druid cluster.

It has following changes.

• LoggingEmitter is modified to emit jsonMapper.writeValueAsString(event) instead of format("Event [%s]", jsonMapper.writeValueAsString(event)) .
• QueryMetrics puts the query context into metric event as is instead of converting it to String.
• EmittingRequestLoggerProvider uses only available DefaultRequestLogEventBuilderFactory by default when publishing request logs using the emitter.

Side note: I also drop following in my log4j.xml to send the json object published as is...

<Configuration status="WARN">
<Appenders>
 ...
  <Console name="msgonly" target="SYSTEM_OUT">
    <PatternLayout pattern="%m%n"/>
  </Console>
...
</Appenders>
<Loggers>
...
  <Logger name="org.apache.druid.java.util.emitter.core.LoggingEmitter" additivity="false" level="info">
    <AppenderRef ref="msgonly"/>
  </Logger>
...
</Loggers>
</Configuration>

---

This PR has:

• been self-reviewed.
• been tested in a Druid cluster.

---

Release Note Update:

LoggingEmitter is modified to emit jsonMapper.writeValueAsString(event) instead of format("Event [%s]", jsonMapper.writeValueAsString(event)) . Anyone depending on the older specifics of message printed by LoggingEmitter would need to update accordingly.

[himanshug]

all builds except coverage are passing which looks like spurious failure.

[leventov]

Please add a Javadoc comment explaining why this is needed.

[himanshug]

added

[leventov]

Why added this method in this PR? If this is the default factory, why would anybody want to specify druid.request.logging.requestLogEventBuilderFactory=xyz in the config?

Please clarify both Javadoc and the PR description about this. Also, please add instructions for Druid cluster operators regarding this PR (what configs should be changed, what existing things could break.) Adding something to Druid's documentation might also be needed.

[himanshug]

so without this patch , I added following in my properties ....

druid.emitter=logging
druid.request.logging.type=emitter
druid.request.logging.feed=requests

that failed with following error....

1) druid.request.logging.requestLogEventBuilderFactory - may not be null
  at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:164) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryableModule)
  at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:164) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryableModule)
  while locating com.google.common.base.Supplier<org.apache.druid.server.log.RequestLoggerProvider>
  at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:165) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryableModule)
  while locating org.apache.druid.server.log.RequestLoggerProvider
  at org.apache.druid.guice.QueryableModule.configure(QueryableModule.java:48) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryableModule)
  while locating org.apache.druid.server.log.RequestLogger
    for the 5th parameter of org.apache.druid.server.QueryLifecycleFactory.<init>(QueryLifecycleFactory.java:52)
  at org.apache.druid.server.QueryLifecycleFactory.class(QueryLifecycleFactory.java:52)
  while locating org.apache.druid.server.QueryLifecycleFactory
    for the 1st parameter of org.apache.druid.server.QueryResource.<init>(QueryResource.java:113)
  at org.apache.druid.server.QueryResource.class(QueryResource.java:78)
  while locating org.apache.druid.server.QueryResource

as per the documentation , above was not supposed to happen. so made the change in EmittingRequestLoggerProvider to hardcode usage of DefaultRequestLogEventBuilderFactory by default .
no change is needed in the docs as things would work as expected.

for this specific change: fixing the json creation is not strictly necessary, so removed that to keep things simple in this PR.

[himanshug]

also added a blurb to be added to release notes due to change in this PR . I don't think it impacts anyone but just in case.

[himanshug]

@leventov thanks for reviewing