KAFKA-13859: Disable idempotence on SCRAM authentication

[dengziming]

More detailed description of your change
After upgrading from 2.X to 3.0.1 we set "enable.idempotence"=true by default, but the InitProducerIdRequest involves the TRANSACTIONAL_ID are authorized to write if we are using SCRAM authentication, or it will fail.

Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[dengziming]

Hello @showuon @ijuma @hachikuji , This problem is similar to what we fixed in #11788, however, it's more subtle. And I'm not sure whether we should try a different approach, what's your thought?

[showuon]

@dengziming , thanks for the quick investigation! Nice finding! Yes, once idempotent producer is enabled, the transactionManager will be created. And in Sender run loop here, we'll try to create the producer ID with null transaction ID. In the server side, we always check the transaction-ID write permission when handling the InitProducerIdRequest, which causes the authorization error.

But my question will be, could we, on the other hand, fix the issue in the server side, to skip the transaction ID permission check if transaction ID is null?

Also, we might test cases for it. You could refer to the test: AuthorizerIntegrationTest#testAuthorizeByResourceTypeMultipleAddAndRemove.

Great find again! Thank you!

[showuon]

@dengziming , have a 2nd look, I found it's not right. Here, we won't check the transactional_ID write permission if transactionalId==null.

Does that mean the user doesn't have IDEMPOTENT_WRITE permission? I think we might need the tests to confirm our finding. Thanks.

[dengziming]

@showuon Yeah, you are right, Here is the log of clients when I reproduce this error with the 3.0.1 client, it shows we are failing the IDEMPOTENT_WRITE permission.

[2022-04-28 15:11:11,425] TRACE [Producer clientId=producer-1] Received transactional response InitProducerIdResponseData(throttleTimeMs=0, errorCode=31, producerId=-1, producerEpoch=0) for request InitProducerIdRequestData(transactionalId=null, transactionTimeoutMs=2147483647, producerId=-1, producerEpoch=-1) (org.apache.kafka.clients.producer.internals.TransactionManager:1285)
[2022-04-28 15:11:11,425] INFO [Producer clientId=producer-1] Transiting to fatal error state due to org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed. (org.apache.kafka.clients.producer.internals.TransactionManager:500)
[2022-04-28 15:11:11,425] DEBUG [Producer clientId=producer-1] Transition from state INITIALIZING to error state FATAL_ERROR (org.apache.kafka.clients.producer.internals.TransactionManager:1089)
org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

[showuon]

Thanks @dengziming . I checked the KIP-679 and found this:

We are relaxing the ACL restriction from IDEMPOTENT_WRITE to WRITE earlier (release version 2.8) and changing the producer defaults later (release version 3.0) in order to give the community users enough time to upgrade their broker first. So their later client-side upgrading, which enables idempotence by default, won't get blocked by the IDEMPOTENT_WRITE ACL required by the old version brokers.

Could you help check if we only needWRITE permission, not IDEMPOTENT_WRITE?

[dengziming]

I checked the KIP-679 and found this:

It seems this is what I met here, a 3.0.1 client send to a 2.X broker will fail with ClusterAuthorizationException, this is not a bug according to KIP-679 but an intentional change.

[dengziming]

Close this since this is designed intentionally, we should help the users to make this change.

[showuon]

Thanks @dengziming !

[ijuma]

To be clear, the broker would have to be older than 2.8 for the issue to occur. The server change for KIP-679 happened in Apache Kafka 2.8.

[showuon]

To be clear, the broker would have to be older than 2.8 for the issue to occur. The server change for KIP-679 happened in Apache Kafka 2.8.

Yes, the user confirmed that their broker version is in v2.6.2 and client version is in v3.0.1. So, this is the expected behavior. Thanks @dengziming and @ijuma !