Skip to content

Expend coverage for default mapping for prometheus emitter#16340

Merged
cryptoe merged 1 commit intoapache:masterfrom
ztzxt:configure-default-mapping-prometheus-emitter
Apr 27, 2024
Merged

Expend coverage for default mapping for prometheus emitter#16340
cryptoe merged 1 commit intoapache:masterfrom
ztzxt:configure-default-mapping-prometheus-emitter

Conversation

@ztzxt
Copy link
Copy Markdown
Contributor

@ztzxt ztzxt commented Apr 26, 2024

Description

Went over https://druid.apache.org/docs/latest/operations/metrics/ and added most of missing metrics to prometheus emitter default mapping.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@ztzxt
Copy link
Copy Markdown
Contributor Author

ztzxt commented Apr 26, 2024

We ended up expanding the current default mapping for prometheus-emitter. Therefore decided to share it with the upstream. When deciding between gauge/count types, I used similar (and already existing) metrics as an example. I didn't add the metric if it required some module unless the Prometheus metric was already generated with the default configuration. I re-ordered the existing metrics so they'd reflect the order in the document. I also removed the non-existent metrics such as segment/cost/*

diff for added/removed metric names:

cat extensions-contrib/prometheus-emitter/src/main/resources/defaultMetrics.json| jq -r 'keys[]' | sort > feature-metrics
git checkout master 
cat extensions-contrib/prometheus-emitter/src/main/resources/defaultMetrics.json| jq -r 'keys[]' | sort > master-metrics 
diff master-metrics feature-metrics 
0a1,6
> compact/segmentAnalyzer/fetchAndProcessMillis
> compact/task/count
> compactTask/availableSlot/count
> compactTask/maxSlot/count
> coordinator/global/time
> coordinator/time
2a9,10
> ingest/bytes/received
> ingest/count
9a18
> ingest/handoff/count
10a20,21
> ingest/handoff/time
> ingest/input/bytes
17a29,30
> ingest/notices/time
> ingest/pause/time
23a37,50
> ingest/segments/count
> ingest/sink/count
> ingest/tombstones/count
> interval/compacted/count
> interval/skipCompact/count
> interval/waitCompact/count
> jetty/numOpenConnections
> jetty/threadPool/busy
> jetty/threadPool/idle
> jetty/threadPool/isLowOnThreads
> jetty/threadPool/max
> jetty/threadPool/min
> jetty/threadPool/queueSize
> jetty/threadPool/total
36a64,67
> kill/pendingSegments/count
> kill/task/count
> killTask/availableSlot/count
> killTask/maxSlot/count
37a69,79
> metadata/kill/audit/count
> metadata/kill/compaction/count
> metadata/kill/datasource/count
> metadata/kill/rule/count
> metadata/kill/supervisor/count
> metadatacache/init/time
> metadatacache/refresh/count
> metadatacache/refresh/time
> metadatacache/schemaPoll/count
> metadatacache/schemaPoll/failed
> metadatacache/schemaPoll/time
45a88,90
> query/cache/delta/put/error
> query/cache/delta/put/ok
> query/cache/delta/put/oversized
54a100,102
> query/cache/total/put/error
> query/cache/total/put/ok
> query/cache/total/put/oversized
61d108
< query/intervalChunk/time
72a120
> segment/assignSkipped/count
74,76c122,123
< segment/cost/normalization
< segment/cost/normalized
< segment/cost/raw
---
> segment/compacted/bytes
> segment/compacted/count
79a127
> segment/dropSkipped/count
80a129,130
> segment/loadQueue/assigned
> segment/loadQueue/cancelled
83a134
> segment/loadQueue/success
84a136
> segment/moveSkipped/count
92a145,146
> segment/skipCompact/bytes
> segment/skipCompact/count
95a150
> segment/unneededEternityTombstone/count
97a153,157
> segment/waitCompact/bytes
> segment/waitCompact/count
> serverview/init/time
> serverview/sync/healthy
> serverview/sync/unstableTime
98a159,161
> sqlQuery/bytes
> sqlQuery/planningTimeMs
> sqlQuery/time
99a163
> sys/disk/queue
101a166
> sys/disk/transferTime
103a169,170
> sys/fs/files/count
> sys/fs/files/free
105a173,176
> sys/mem/free
107a179,181
> sys/net/read/dropped
> sys/net/read/errors
> sys/net/read/packets
108a183,185
> sys/net/write/collisions
> sys/net/write/errors
> sys/net/write/packets
114a192,209
> sys/tcpv4/activeOpens
> sys/tcpv4/attemptFails
> sys/tcpv4/estabResets
> sys/tcpv4/in/errs
> sys/tcpv4/in/segs
> sys/tcpv4/out/rsts
> sys/tcpv4/out/segs
> sys/tcpv4/passiveOpens
> sys/tcpv4/retrans/segs
> sys/uptime
> task/action/batch/attempts
> task/action/batch/queueTime
> task/action/batch/runTime
> task/action/batch/size
> task/action/failed/count
> task/action/log/time
> task/action/run/time
> task/action/success/count
116a212
> task/pending/time
118a215
> task/segmentAvailability/wait/time
120a218,221
> tier/historical/count
> tier/replication/factor
> tier/required/capacity
> tier/total/capacity
121a223
> zk/reconnect/time

@ztzxt ztzxt marked this pull request as draft April 26, 2024 14:02
@ztzxt ztzxt force-pushed the configure-default-mapping-prometheus-emitter branch from 3d91203 to 2d27f9c Compare April 26, 2024 14:03
@ztzxt ztzxt marked this pull request as ready for review April 26, 2024 14:18
Copy link
Copy Markdown
Contributor

@cryptoe cryptoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes lgtm.
Thanks @ztzxt .
There are a few more metrics which will be added as part of druid 30.
It would be lovely if you can patch that up as well :)

@cryptoe cryptoe merged commit 9aef8e0 into apache:master Apr 27, 2024
@cryptoe
Copy link
Copy Markdown
Contributor

cryptoe commented Apr 27, 2024

@ztzxt ztzxt deleted the configure-default-mapping-prometheus-emitter branch April 29, 2024 06:40
@kfaraz kfaraz added this to the 31.0.0 milestone Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants