Motivation
The LoggingEmitter currently emits all metrics it receives without filtering. On busy clusters with high query throughput, this can result in a very large volume of metric logs - on some of our clusters, we've seen ~3M metric logs in 15 minutes.
Proposed changes
- Add a file to capture the default metrics for the LoggingEmitter, similar to the Prometheus and Dropwizard emitters (e.g.,
defaultMetricDimensions.json).
- This file can contain the list of allowed metrics.
- In the future, we can extend this file to annotate additional user-defined dimensions, if needed.
- In the LoggingEmitter
emit(Event event) method, check the allowlist metrics map and decide whether to include the event or not.
Rationale
Typically, the LoggingEmitter is used in conjunction with a composite emitter that handles low- to medium-cardinality metrics, such as Prometheus, Dropwizard, or Kafka forwarding to a downstream system. Filtering out metrics that are already covered by these emitters would help reduce logging costs while continuing to rely on the appropriate emitter for low- to medium-cardinality metrics.
Operational impact
None. By default, all metrics will continue to be emitted for backwards compatibility with a runtime property. Operators can opt-in to enable filtering with the property.
Motivation
The LoggingEmitter currently emits all metrics it receives without filtering. On busy clusters with high query throughput, this can result in a very large volume of metric logs - on some of our clusters, we've seen ~3M metric logs in 15 minutes.
Proposed changes
defaultMetricDimensions.json).emit(Event event)method, check the allowlist metrics map and decide whether to include the event or not.Rationale
Typically, the LoggingEmitter is used in conjunction with a composite emitter that handles low- to medium-cardinality metrics, such as Prometheus, Dropwizard, or Kafka forwarding to a downstream system. Filtering out metrics that are already covered by these emitters would help reduce logging costs while continuing to rely on the appropriate emitter for low- to medium-cardinality metrics.
Operational impact
None. By default, all metrics will continue to be emitted for backwards compatibility with a runtime property. Operators can opt-in to enable filtering with the property.