KAFKA-17921: Support SASL_PLAINTEXT protocol with java.security.auth.login.config by FrankYang0529 · Pull Request #17671 · apache/kafka

FrankYang0529 · 2024-11-03T04:30:30Z

Support SASL_PLAINTEXT protocol in ClusterTest framework.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

chia7712

@FrankYang0529 thanks for this patch

chia7712 · 2024-12-29T06:01:40Z

+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class JaasModule {


Is this duplicate to core JaasModule?

Yes, this is duplicated. Original JaasModule is in core module. I think we can remove original one after we migrate all sasl test case to new framework. WDYT?

chia7712 · 2024-12-29T06:05:56Z

            waitForAllFutures(futureEntries);
            futureEntries.clear();
            Utils.delete(baseDirectory);
+            if (jaasFile.isPresent()) {


Could you please use ifPresent instead?

chia7712 · 2024-12-29T06:09:28Z

+
+import javax.security.auth.login.Configuration;
+
+public class JaasTestUtils {


It is already in xxx.test package, maybe we can call it JaasUtils?

chia7712 · 2024-12-29T06:11:20Z

+    public static final String KAFKA_PLAIN_ADMIN = "plain-admin";
+    public static final String KAFKA_PLAIN_ADMIN_PASSWORD = "plain-admin-secret";
+
+    public static File writeJaasContextsToFile(List<JaasSection> jaasSections) throws IOException {


Do we really need this object? How about using map? It is more simple

chia7712 · 2024-12-29T06:12:19Z

+
+    private final Map<String, String> entries;
+
+    private JaasModule(String name, boolean debug, Map<String, String> entries) {


Could you please try to use record class?

chia7712 · 2024-12-29T06:13:25Z

+        configs.put(SaslConfigs.SASL_JAAS_CONFIG,
+            String.format("org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";",
+                JaasTestUtils.KAFKA_PLAIN_USER1, JaasTestUtils.KAFKA_PLAIN_USER1_PASSWORD));
+        try (Admin admin = clusterInstance.admin(configs)) {


Those helper should return authorized client object, right? Otherwise, developers have to configure them manually

chia7712

@FrankYang0529 thanks for this patch. two minor comments are left PTAL

chia7712 · 2024-12-30T12:57:35Z

+            assertDoesNotThrow(() -> admin.describeAcls(AclBindingFilter.ANY).values().get());
+        }
+        String topic = "sasl-plaintext-topic";
+        Assertions.assertDoesNotThrow(() -> clusterInstance.createTopic(topic, 1, (short) 1));


Since the APIs in question only throw runtime exceptions, there's no need to use assertDoesNotThrow, which is typically employed to convert checked exceptions into runtime exceptions.

chia7712 · 2024-12-30T12:58:49Z

+        }
+
+        // client with non-admin credentials
+        Map<String, Object> nonAdminConfig = Map.of(


the helper methods of clusterInstance should be able configure the sasl.jaas.config automatically - that is why we add those helper to clusterInstance

Hi @chia7712, thanks for the review. Yes, we already set default user to admin, consumer, and producer helper function automatically. Here, we would like to test developer overrides sasl.jaas.config with non-admin credentials. We also test overriding with unknown credential, so we can make sure the cluster instance with sasl plaintext works.

FrankYang0529 · 2025-01-06T03:00:21Z

Hi @chia7712, thanks for the review. I addressed all comments. Could you review it when you have time? Thank you.

chia7712

@FrankYang0529 thanks for this patch! it is great to see our new test infra has such support

chia7712 · 2025-01-07T14:02:15Z

+    public static final String KAFKA_PLAIN_ADMIN = "plain-admin";
+    public static final String KAFKA_PLAIN_ADMIN_PASSWORD = "plain-admin-secret";
+
+    public static File writeJaasContextsToFile(Map<String, JaasSection> jaasSections) throws IOException {


It seems we don't use the key, so maybe we can change the collection from Map to Set?

chia7712 · 2025-01-07T14:05:56Z

            futureEntries.clear();
            Utils.delete(baseDirectory);
+            jaasFile.ifPresent(f -> {
+                try {


To enhance readability, we should explore alternatives to nested try-catch blocks, even if it involves using a more traditional coding style. for example: if (jaasFile.isPresent()) Utils.delete(jaasFile.get());

chia7712 · 2025-01-07T14:07:42Z

+    @ClusterTest(
+        types = {Type.KRAFT, Type.CO_KRAFT},
+        brokerSecurityProtocol = SecurityProtocol.SASL_PLAINTEXT,
+        controllerSecurityProtocol = SecurityProtocol.SASL_PLAINTEXT,


Do we support the SASL_PLAINTEXT on controller? If not, could you please remove this config and open a jira for it?

Yes, we already support it in this PR. I update KafkaClusterTestKit.Builder#createNodeConfig to respect controllerSecurityProtocol setting.

…login.config Signed-off-by: PoAn Yang <payang@apache.org>

chia7712 · 2025-01-07T15:02:45Z

            // reduce log cleaner offset map memory usage
            props.putIfAbsent(CleanerConfig.LOG_CLEANER_DEDUPE_BUFFER_SIZE_PROP, "2097152");

+            if (brokerSecurityProtocol.equals(SecurityProtocol.SASL_PLAINTEXT.name)) {


it seems we set security for both controller and broker based on the brokerSecurityProtocol, right? if so, that is weird to me as we totally ignore controllerSecurityProtocol ...

Signed-off-by: PoAn Yang <payang@apache.org>

chia7712 · 2025-01-07T19:24:56Z

+                ExecutionException.class,
+                () -> admin.describeAcls(AclBindingFilter.ANY, new DescribeAclsOptions().timeoutMs(5000)).values().get()
+            );
+            assertInstanceOf(TimeoutException.class, exception.getCause());


I open https://issues.apache.org/jira/browse/KAFKA-18440 to fix it. please feel free to take over the jira if you have free cycle

Thanks for review. I will handle it.

chia7712 · 2025-01-07T19:32:42Z

The failed tests are unrelated and they pass on my local

chia7712 · 2025-01-07T19:34:02Z

https://issues.apache.org/jira/browse/KAFKA-8459
https://issues.apache.org/jira/browse/KAFKA-14249
https://issues.apache.org/jira/browse/KAFKA-18441

…ogin.config (#17671) Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

…og-compaction-write-record-v2 * apache-github/trunk: (34 commits) MINOR: Bump year to 2025 in NOTICE file (apache#18427) KAFKA-18411 Remove ZkProducerIdManager (apache#18413) KAFKA-18408 tweak the 'tag' field for BrokerHeartbeatRequest.json, BrokerRegistrationChangeRecord.json and RegisterBrokerRecord.json (apache#18421) KAFKA-18414 Remove KRaftRegistrationResult (apache#18401) KAFKA-17921 Support SASL_PLAINTEXT protocol with java.security.auth.login.config (apache#17671) KAFKA-18384 Remove ZkAlterPartitionManager (apache#18364) KAFKA-10790: Add deadlock detection to producer#flush (apache#17946) KAFKA-18412: Remove EmbeddedZookeeper (apache#18399) MINOR : Improve Exception log in NotEnoughReplicasException(apache#12394) MINOR: Improve PlaintextAdminIntegrationTest#testConsumerGroups (apache#18409) MINOR: Remove unused local variable (apache#18410) MINOR: Remove RaftManager.maybeDeleteMetadataLogDir and AutoTopicCreationManagerTest.scala (apache#17365) KAFKA-18368 Remove TestUtils#MockZkConnect and remove zkConnect from TestUtils#createBrokerConfig (apache#18352) MINOR: Update Consumer group timeout default to 30 sec (apache#16406) MINOR: Fix typo in CommitRequestManager (apache#18407) MINOR: cleanup JavaDocs for deprecation warnings (apache#18402) KAFKA-18303; Update ShareCoordinator to use new record format (apache#18396) MINOR: Update Consumer and Producer JavaDocs for committing offsets (apache#18336) KAFKA-16446: Improve controller event duration logging (apache#15622) KAFKA-18388 test-kraft-server-start.sh should use log4j2.yaml (apache#18370) ...

…ogin.config (apache#17671) Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

FrankYang0529 force-pushed the KAFKA-17921 branch 2 times, most recently from 7490314 to 3d304c4 Compare November 10, 2024 06:03

FrankYang0529 requested a review from chia7712 November 12, 2024 04:12

FrankYang0529 force-pushed the KAFKA-17921 branch from 3d304c4 to fd8f47b Compare November 13, 2024 11:21

FrankYang0529 force-pushed the KAFKA-17921 branch 2 times, most recently from 535eedb to ee1d321 Compare November 27, 2024 07:31

FrankYang0529 force-pushed the KAFKA-17921 branch 2 times, most recently from c8ddc37 to e205bb8 Compare December 18, 2024 10:18

FrankYang0529 mentioned this pull request Dec 18, 2024

KAFKA-17833: Convert DescribeAuthorizedOperationsTest to use kraft #18252

Merged

3 tasks

FrankYang0529 force-pushed the KAFKA-17921 branch from e205bb8 to cefdcde Compare December 27, 2024 08:11

FrankYang0529 mentioned this pull request Dec 27, 2024

KAFKA-17631: Convert SaslApiVersionsRequestTest to kraft #18330

Merged

3 tasks

chia7712 reviewed Dec 29, 2024

View reviewed changes

FrankYang0529 force-pushed the KAFKA-17921 branch 2 times, most recently from 7e4b4a6 to c9ea0ad Compare December 29, 2024 14:08

FrankYang0529 requested a review from chia7712 December 30, 2024 02:04

chia7712 reviewed Dec 30, 2024

View reviewed changes

FrankYang0529 force-pushed the KAFKA-17921 branch from c9ea0ad to dacb066 Compare December 30, 2024 14:51

FrankYang0529 requested a review from chia7712 December 31, 2024 00:12

chia7712 reviewed Jan 7, 2025

View reviewed changes

KAFKA-17921: Support SASL_PLAINTEXT protocol with java.security.auth.…

12d2393

…login.config Signed-off-by: PoAn Yang <payang@apache.org>

chia7712 reviewed Jan 7, 2025

View reviewed changes

address comment

9020816

Signed-off-by: PoAn Yang <payang@apache.org>

FrankYang0529 force-pushed the KAFKA-17921 branch from dacb066 to 9020816 Compare January 7, 2025 15:21

address comment

9631359

Signed-off-by: PoAn Yang <payang@apache.org>

chia7712 approved these changes Jan 7, 2025

View reviewed changes

chia7712 merged commit 228b325 into apache:trunk Jan 7, 2025

chia7712 pushed a commit that referenced this pull request Jan 7, 2025

KAFKA-17921 Support SASL_PLAINTEXT protocol with java.security.auth.l…

e6c35b6

…ogin.config (#17671) Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

FrankYang0529 deleted the KAFKA-17921 branch January 7, 2025 23:58

pranavt84 pushed a commit to pranavt84/kafka that referenced this pull request Jan 27, 2025

KAFKA-17921 Support SASL_PLAINTEXT protocol with java.security.auth.l…

8e49994

…ogin.config (apache#17671) Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

manoj-mathivanan pushed a commit to manoj-mathivanan/kafka that referenced this pull request Feb 19, 2025

KAFKA-17921 Support SASL_PLAINTEXT protocol with java.security.auth.l…

7d1007a

…ogin.config (apache#17671) Reviewers: Chia-Ping Tsai <chia7712@gmail.com>


		import javax.security.auth.login.Configuration;

		public class JaasTestUtils {


		private final Map<String, String> entries;

		private JaasModule(String name, boolean debug, Map<String, String> entries) {

Conversation

FrankYang0529 commented Nov 3, 2024

Committer Checklist (excluded from commit message)

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankYang0529 commented Jan 6, 2025

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 commented Jan 7, 2025

Uh oh!

chia7712 commented Jan 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants