KAFKA-5235: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimized number of requests to server#3051
KAFKA-5235: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimized number of requests to server#3051tashoyan wants to merge 44 commits intoapache:trunkfrom
Conversation
…is new implementation. Now it works for replicated topics.
…Add the ability to exclude internal topics.
…d empty partition set
… are available in the key. Remove unused constant.
…e tools. Deprecate old args and display warnings.
…g topic. Update tests to distinguish the two error handling paths.
|
Refer to this link for build results (access rights to CI server needed): |
|
@guozhangwang would you review this PR please? I hope it won't take much time. Why keep it awaiting in the queue for months? |
|
@tashoyan Thanks for the PR and sorry for the delay. Have you checked if the |
|
@ijuma |
|
@junrao @guozhangwang @gwenshap @ijuma may I kindly ask you to review this PR? |
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
…setsForTimes(): return null offset entries for non-existing timestamps - KAFKA-5534
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
|
Is there any other way of getting offset information without GetOffsetShell? If not, this is a quite important PR, isn't it? Right now, GetOffsetShell doesn't work with SASL_SSL, and I can't find any solution for this. Thanks for the amazing work! |
|
I also think that this PR is important. We need this functionality for our monitoring tools and for performance metrics. I hope, finally we will have it in the product. |
|
@tashoyan Sorry for the delay. Since you have made changes to the command line parameters, can you please submit a KIP (https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals)? |
|
@ijuma I don't have access to create KIP. Could you give it to me? I wrote to dev mailgroup, but no answer. |
This does the minimal amount of work so that the tool relies on public non-deprecated APIs (i.e. so that it doesn't rely on old clients code). Additional improvements have been proposed via KIP-308. There are a few other PRs that touch this class with overlapping goals: - apache#2891 - apache#3051 - apache#3320 One of them remains relevant in the context of KIP-308, but the others are not. I included the authors of the 3 PRs as co-authors. Co-authored-by: Arseniy Tashoyan <tashoyan@gmail.com> Co-authored-by: Vahid Hashemian <vahidhashemian@us.ibm.com> Co-authored-by: Mohammed Amine GARMES Co-authored-by: Ismael Juma <ismael@juma.me.uk>
This does the minimal amount of work so that the tool relies on public non-deprecated APIs (i.e. so that it doesn't rely on old clients code). Additional improvements have been proposed via KIP-308. There are a few other PRs that touch this class with overlapping goals: - apache#2891 - apache#3051 - apache#3320 One of them remains relevant in the context of KIP-308, but the others are not. I included the authors of the 3 PRs as co-authors. Co-authored-by: Arseniy Tashoyan <tashoyan@gmail.com> Co-authored-by: Vahid Hashemian <vahidhashemian@us.ibm.com> Co-authored-by: Mohammed Amine GARMES Co-authored-by: Ismael Juma <ismael@juma.me.uk>
This does the minimal amount of work so that the tool relies on public non-deprecated APIs (i.e. it no longer relies on Scala clients code). Additional improvements (not included here) have been proposed via KIP-308. There are a few other PRs that touch this class with overlapping goals: - #2891 - #3051 - #3320 One of them remains relevant in the context of KIP-308, but the others have been superseded. I included the authors of the 3 PRs as co-authors. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Vahid Hashemian <vahidhashemian@us.ibm.com>, Manikumar Reddy <manikumar.reddy@gmail.com> Co-authored-by: Arseniy Tashoyan <tashoyan@gmail.com> Co-authored-by: Vahid Hashemian <vahidhashemian@us.ibm.com> Co-authored-by: Mohammed Amine GARMES Co-authored-by: Ismael Juma <ismael@juma.me.uk>
This does the minimal amount of work so that the tool relies on public non-deprecated APIs (i.e. it no longer relies on Scala clients code). Additional improvements (not included here) have been proposed via KIP-308. There are a few other PRs that touch this class with overlapping goals: - apache#2891 - apache#3051 - apache#3320 One of them remains relevant in the context of KIP-308, but the others have been superseded. I included the authors of the 3 PRs as co-authors. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Vahid Hashemian <vahidhashemian@us.ibm.com>, Manikumar Reddy <manikumar.reddy@gmail.com> Co-authored-by: Arseniy Tashoyan <tashoyan@gmail.com> Co-authored-by: Vahid Hashemian <vahidhashemian@us.ibm.com> Co-authored-by: Mohammed Amine GARMES Co-authored-by: Ismael Juma <ismael@juma.me.uk>
|
Thanks @ijuma |
|
Maybe the easiest is to close and submit a new one once the KIP has been voted. |
This does the minimal amount of work so that the tool relies on public non-deprecated APIs (i.e. it no longer relies on Scala clients code). Additional improvements (not included here) have been proposed via KIP-308. There are a few other PRs that touch this class with overlapping goals: - apache#2891 - apache#3051 - apache#3320 One of them remains relevant in the context of KIP-308, but the others have been superseded. I included the authors of the 3 PRs as co-authors. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Vahid Hashemian <vahidhashemian@us.ibm.com>, Manikumar Reddy <manikumar.reddy@gmail.com> Co-authored-by: Arseniy Tashoyan <tashoyan@gmail.com> Co-authored-by: Vahid Hashemian <vahidhashemian@us.ibm.com> Co-authored-by: Mohammed Amine GARMES Co-authored-by: Ismael Juma <ismael@juma.me.uk>
|
Closing as this is addressed in #9430 (KIP-635). |
6 participants
This PR addresses two improvements:
KAFKA-5235 GetOffsetShell: retrieve offsets for all given topics and partitions with single request to the broker
KAFKA-5234 GetOffsetShell: retrieve offsets for multiple topics with single request
New implementation
kafka-get-offsets.shuses KafkaConsumer API. It makes at most two requests to the broker: 1) to query existing topics and partitions, 2) to grab all requested offsets. New implementation correctly handles non-existing topics and partitions asked by user:Previously, user could get offsets for one topic only. Now user can get offsets for many topics at once:
kafka-get-offsets.sh --bootstrap-servers vm:9092 --topics AAA,ZZZMoreover, now user is able to retrieve offsets for all topics - this is the default when no topics specified:
kafka-get-offsets.sh --bootstrap-servers vm:9092Thanks to this feature, there is no need anymore to retrieve all topics by means of
kafka-topics.sh.When no topics specified, the new
kafka-get-offsets.shtool takes into account only user-level topics and ignores Kafka-internal topics (i.e. consumer offsets). This behavior can be altered via a special command line argument:kafka-get-offsets.sh --bootstrap-servers vm:9092 --include-internal-topicsNew
kafka-get-offsets.shtool is consistent with other console tools with respect to command line argument names. In addition,kafka-get-offsets.shtool gives the possibility to pass an arbitrary setting to KafkaConsumer via--consumer-propertyargument. Old command line arguments are marked as deprecated with appropriate warning messages.I hope, now
kafka-get-offsets.shis easier in use and gives performance improvement.@junrao I suppose you may want to review.