Set backlog in gauge metric for Kafka Reader #31137
Conversation
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
Run Java_Kafka_IO_Direct PreCommit |
|
Run Java_Kafka_IO_Direct PreCommit |
|
Assigning reviewers. If you would like to opt out of this review, comment R: @Abacn for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
I vaguely remembered @johnjcasey mentioned before that there were some problem in gauge metrics, @johnjcasey do you have insight here? |
|
cc @dustin12 |
|
Yeah, unfortunately Gauges work in Beam, but are not implemented in Dataflow properly. @celeste-zeng might have more up to date information here |
|
Its still worth setting it in beam so if Dataflow implements Gauges the results will be piped through |
|
The change itself LGTM though |
…eadFromKafkaDoFn (#32921) * Revert "Set backlog in gauge metric (#31137)" * Revert "Add Backlog Metrics to Kafka Splittable DoFn Implementation (#31281)" This reverts commit fd4368f. * Call reportBacklog in nextBatch to report split metrics more often * Report SDF metrics for the active split/partition after processing a record batch * Use KafkaSourceDescriptor as cache key and log entry
3 participants
There are gauges created for per partition backlog in the kafka reader, but its never actually set.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.