Even more indexer/blockchain metrics #736
Conversation
WalkthroughThis set of changes introduces new Prometheus metrics and corresponding instrumentation across several components to improve observability of blockchain-related operations. New metrics are defined for tracking the time spent waiting for blockchain transaction receipts, the duration of publishing payloads to the blockchain, and the time taken to process blockchain logs. The codebase is updated to emit these metrics at relevant points, and the metrics catalog documentation is extended to describe the new metrics. Additionally, the metrics documentation generator is updated to recognize more metric constructor functions. Changes
Sequence Diagram(s)sequenceDiagram
participant PayerService
participant Metrics
participant Blockchain
PayerService->>Metrics: MeasurePublishToBlockchainMethod("group_message", fn)
activate Metrics
Metrics->>Blockchain: PublishGroupMessage()
Blockchain-->>Metrics: result, err
Metrics-->>PayerService: result, err (after emitting metric)
sequenceDiagram
participant BlockchainClient
participant Metrics
BlockchainClient->>BlockchainClient: WaitForTransaction()
Note right of BlockchainClient: Record start time
BlockchainClient->>BlockchainClient: ...wait for receipt...
BlockchainClient->>Metrics: EmitBlockchainWaitForTransaction(duration)
sequenceDiagram
participant Indexer
participant Metrics
loop For each event in eventChannel
Indexer->>Indexer: Record start time
Indexer->>Indexer: Store log, update block tracker
Indexer->>Metrics: EmitIndexerLogProcessingTime(duration)
end
Possibly related PRs
Suggested reviewers
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (1.64.8)Error: you are using a configuration file for golangci-lint v2 with golangci-lint v1: please use golangci-lint v2 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (3)
⏰ Context from checks skipped due to timeout of 90000ms (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
pkg/metrics/blockchain.go (1)
20-26: Consider defining custom histogram buckets.The
blockchainPublishPayloadhistogram doesn't specify custom buckets, so it will use Prometheus defaults. If you have expectations about the range of publish durations, consider defining custom buckets similar to how you did forblockchainWaitForTransaction.prometheus.HistogramOpts{ Name: "xmtpd_blockchain_publish_payload_seconds", Help: "Time to publish a payload to the blockchain", + Buckets: []float64{0.1, 0.5, 1, 2.5, 5, 10, 30}, },
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
doc/metrics_catalog.md(1 hunks)pkg/api/payer/service.go(4 hunks)pkg/blockchain/client.go(2 hunks)pkg/blockchain/rpcLogStreamer.go(0 hunks)pkg/indexer/indexer.go(2 hunks)pkg/metrics/blockchain.go(1 hunks)pkg/metrics/docs/generator.go(1 hunks)pkg/metrics/indexer.go(2 hunks)pkg/metrics/metrics.go(2 hunks)
💤 Files with no reviewable changes (1)
- pkg/blockchain/rpcLogStreamer.go
🧰 Additional context used
🧬 Code Graph Analysis (3)
pkg/blockchain/client.go (1)
pkg/metrics/blockchain.go (1)
EmitBlockchainWaitForTransaction(16-18)
pkg/indexer/indexer.go (1)
pkg/metrics/indexer.go (1)
EmitIndexerLogProcessingTime(115-117)
pkg/api/payer/service.go (3)
pkg/metrics/blockchain.go (1)
MeasurePublishToBlockchainMethod(33-39)pkg/abi/groupmessagebroadcaster/GroupMessageBroadcaster.go (1)
GroupMessageBroadcasterMessageSent(1093-1098)pkg/abi/identityupdatebroadcaster/IdentityUpdateBroadcaster.go (1)
IdentityUpdateBroadcasterIdentityUpdateCreated(824-829)
🪛 GitHub Check: Lint-Go
pkg/blockchain/client.go
[failure] 6-6:
could not import github.com/xmtp/xmtpd/pkg/metrics (-: # github.com/xmtp/xmtpd/pkg/metrics
pkg/metrics/metrics.go
[failure] 92-92:
undefined: blockchainPublish) (typecheck)
[failure] 92-92:
undefined: blockchainPublish) (typecheck)
[failure] 92-92:
undefined: blockchainPublish) (typecheck)
[failure] 92-92:
undefined: blockchainPublish (typecheck)
🪛 GitHub Check: Test (Node)
pkg/metrics/metrics.go
[failure] 92-92:
undefined: blockchainPublish
⏰ Context from checks skipped due to timeout of 90000ms (3)
- GitHub Check: Push Docker Images to GitHub Packages (xmtpd-cli)
- GitHub Check: Push Docker Images to GitHub Packages (xmtpd)
- GitHub Check: Upgrade Tests
🔇 Additional comments (14)
pkg/metrics/indexer.go (2)
67-72: Well-designed histogram for tracking log processing time.The new histogram metric is clearly defined with an appropriate name following Prometheus naming conventions, including the unit suffix (_seconds). This will provide valuable insights into blockchain log processing performance.
115-117: LGTM! Correctly converts duration to seconds.The function properly converts the duration to seconds before recording the observation, which is the standard practice for time-based metrics in Prometheus.
pkg/metrics/blockchain.go (1)
33-39: Good implementation of a generic measurement wrapper.The generic function elegantly handles timing measurements while preserving the return values and error handling of the wrapped function. The use of defer ensures the metric is always emitted regardless of how the function exits.
pkg/blockchain/client.go (1)
87-91:Details
✅ Verification successful
Well-placed instrumentation for transaction wait time.
The timing measurement is correctly implemented with defer to ensure it's always executed. This will provide valuable insights into blockchain transaction wait times, which is critical for performance monitoring.
🏁 Script executed:
#!/bin/bash # Check if the metric has been registered in pkg/metrics/metrics.go rg -A 1 -B 1 "blockchainWaitForTransaction" ./pkg/metrics/metrics.goLength of output: 153
Instrumentation Verified:
blockchainWaitForTransactionMetric RegisteredThe defer-based timing in
pkg/blockchain/client.go(lines 87–91) is correctly implemented, and theblockchainWaitForTransactionmetric is confirmed inpkg/metrics/metrics.go. No further changes needed.pkg/indexer/indexer.go (1)
254-254: Good placement of timing instrumentation for log processing.The timing code correctly captures the full duration of processing a blockchain log, including the storage operation and block tracker update. The measurement starts before any processing begins and ends after all operations are completed, providing an accurate view of the end-to-end processing time.
Also applies to: 356-356
pkg/metrics/docs/generator.go (1)
34-37: LGTM! Great extension of metric type recognitionThese additions enable the documentation generator to recognize non-Vec metric constructor functions, ensuring all metrics get properly documented regardless of how they're created.
pkg/api/payer/service.go (4)
284-288: Excellent instrumentation of group message publishingThe use of
MeasurePublishToBlockchainMethodprovides consistent timing metrics for blockchain operations, which aligns well with the PR objectives of improving observability.
310-314: Excellent instrumentation of identity update publishingThe instrumentation approach is consistent with the group message publishing, maintaining a uniform pattern for measuring blockchain operations.
346-347: Good debug logging for performance trackingThese debug logs complement the metrics by providing developer visibility into blockchain publishing performance during troubleshooting.
370-371: Useful debug log for tracking node processingThis log helps track when messages are being processed by nodes, improving the observability of the end-to-end message flow.
doc/metrics_catalog.md (4)
7-8: LGTM! Well-documented blockchain transaction wait time metricThe documentation clearly describes the purpose of this metric for tracking time spent waiting for transaction receipts.
9-9: LGTM! Clear documentation for payer nonce metricGood description that clarifies this tracks the least recently used nonce and includes the caveat that it's not guaranteed to be the highest nonce.
13-14: LGTM! Useful sync connection metricsThese metrics will help track failed outgoing sync connections, providing visibility into sync issues.
18-20: LGTM! Excellent documentation for new blockchain and indexer metricsThe newly added metrics for outgoing sync connections, blockchain publishing, and log processing are clearly documented with descriptive names and purposes.
Add instrumentation to measure blockchain and indexer performance metrics across XMTP servicesIntroduces new metrics instrumentation across multiple components:
📍Where to StartStart with the new metrics infrastructure in blockchain.go which defines the core measurement functions Changes since #736 opened
Macroscope summarized 29b87f5. |
Add blockchain performance monitoring metrics to measure transaction confirmation, payload publishing, and log processing times
Implements new Prometheus metrics across multiple components to monitor blockchain and indexer performance:
📍Where to Start
Start with the new blockchain metrics implementation in metrics/blockchain.go which defines the core metrics and helper functions used throughout the changes.
Macroscope summarized 40eef93.
Summary by CodeRabbit