[Offload] Add offload performance metrics for tiered storage.

[frankxieke]

Motivation

Currently, there is no offload metrics for tiered storage, so it is very hard for us to debug the performance issues. For example , we can not find why offload is slow or why read offload is slow. For above reasons. we need to add some offload metrics for monitoring.

Modifications

Add metrics during offload procedure and read offload data procedure. Including offloadTime, offloadError, offloadRate, readLedgerLatency, writeStoreLatency, writeStoreError, readOffloadIndexLatency, readOffloadDataLatency, readOffloadRate, readOffloadError.

Documentation

need documentation.

[Anonymitaet]

@frankxieke should we add metrics explanations here https://pulsar.apache.org/docs/en/next/reference-metrics/ ?

If so, would you like to add docs accordingly? Then you can ping me to review, thanks

[Anonymitaet]

Confirmed w/ @frankxieke, he will add docs and ping me review after this PR is finalized.

[michaeljmarshall]

@frankxieke - this will be a good addition to our metrics. Before we merge this PR, I would like to see these metrics behind a feature flag that defaults to disabled at the topic level. Perhaps we can use the config exposeManagedLedgerMetricsInPrometheus, which already exists, as the feature flag?

[michaeljmarshall]

@lhotari - I know you have contributed optimizations to the way Pulsar generates Prometheus metrics in the past. Would you mind taking a look at this PR?

[codelipenghui]

Pulsar can enable/disable the namespace/topic level metrics. so we should aggregate for different levels, if namespace/topic level disabled we only expose the broker level metrics, if namespace enabled but topic level disabled, we only expose the namespace level aggregate value.

[hangc0276]

I doubt why not choose provide OpStatsLogger (pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics/DataSketchesOpStatsLogger.java) to generate metrics instead of LedgerOffloaderMXBeanImpl.

An demo of using OpStatsLogger is streamnative/kop#389

[frankxieke]

rerun failure checks

[frankxieke]

rerun failure checks

[frankxieke]

rerun failure checks

rerun failure checks