Update Kafka Client to 2.1.1 and Disable Kafka 0.9 and 0.10 tests #1060
Conversation
|
|
|
I only disabled 0.9 and 0.10 tests, I will follow up with deleting the maven modules that create that shaded clients for tests |
Why cannot we support ListOffsets v0 request after the upgrade? It's better not to drop the support for older clients. |
|
I think one of the reasons is |
Yes, this is the problem, we are using the Kafka Client jars to encode/decode the requests and the responses, so we cannot parse any more the V0 ListOffSetRequest version. We would have to write our own version of the Request/Response parser. I think that those clients are very old and people could upgrade to the latest versions (at least 1.0+) |
AFAIK, they are still widely used and it's not easy to upgrade the client. Especially some applications are not written by Java but they use the old protocol. For example, #539 was opened not long ago. Another example is that TEG of Tencent has adopted KoP for some time, they still have some users with old Kafka protocol. The most important reason is that we've already supported 0.9 and 1.0. If we dropped the support, it would be a regression.
It's okay. My suggestion is that we can merge this PR to master after tests passed. But it can only be merged to existing branches like In addition, please fix the Codacy Static Code Analysis. And I found |
|
I will fix the PR and have CI passing. thanks |
|
I am rebasing this patch on top of current master. |
(cherry picked from commit b5531c9)
eolivelli
force-pushed
the
impl/kafka-21
branch
from
94864a5 to
73ac6f5
Compare
|
@BewareMyPower @Demogorgon314 I have rebased this patch. I have been running this code in production for a while without any particular issues |
|
We can ignore the Codacy check at this moment, it's an existing error.
But we need to fix the incompatibility brought by the constructor of
|
|
@BewareMyPower thanks for your suggestion, the patch was based on an old version of KOP without the changes around OAuth. |
|
it looks like our implementation is not compatible with Kafka requirements: we are passing this stuff as content of the response: logs:
|
|
no sorry, I used the wrong constructor. |
|
test is passing now locally |
|
@BewareMyPower PTAL when you can. |
Fixes: #1293 ### Motivation #1060 updates the `kafka-clients` dependency to 2.1.1 but it drops the support for Kafka clients 0.9 and 0.10 because `ListOffsetRequest.offsetData()` method has removed. ### Modifications Add the old version of `ListOffsetRequest` to be compatible with Kafka clients 0.9 and 0.10.
) This patch upgrade the Kafka Client dependency to 2.1.1. The reason is that Kafka 2.0.0 dependency has some CVEs reported and we cannot ship this project to users if it contains CVEs. The downside is that we are dropping compatibility with 0.9 and 0.10 clients because version 0 of ListOffSets is no more supported. (cherry picked from commit ae29ee3)
Fixes: #1293 ### Motivation #1060 updates the `kafka-clients` dependency to 2.1.1 but it drops the support for Kafka clients 0.9 and 0.10 because `ListOffsetRequest.offsetData()` method has removed. ### Modifications Add the old version of `ListOffsetRequest` to be compatible with Kafka clients 0.9 and 0.10. (cherry picked from commit 723d3be)
Fixes: streamnative#1293 ### Motivation streamnative#1060 updates the `kafka-clients` dependency to 2.1.1 but it drops the support for Kafka clients 0.9 and 0.10 because `ListOffsetRequest.offsetData()` method has removed. ### Modifications Add the old version of `ListOffsetRequest` to be compatible with Kafka clients 0.9 and 0.10. (cherry picked from commit 723d3be)
5 participants
This patch upgrade the Kafka Client dependency to 2.1.1.
The reason is that Kafka 2.0.0 dependency has some CVEs reported and we cannot ship this project to users if it contains CVEs.
The downside is that we are dropping compatibility with 0.9 and 0.10 clients because version 0 of ListOffSets is no more supported.