KAFKA-10338; Support PEM format for SSL key and trust stores (KIP-651) by rajinisivaram · Pull Request #9345 · apache/kafka

rajinisivaram · 2020-09-28T19:01:02Z

Adds support for SSL key and trust stores to be specified in PEM format either as files or directly as configuration values.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

rajinisivaram · 2020-09-29T12:18:45Z

retest this please

omkreddy

@rajinisivaram Thanks for the PR. LGTM.

omkreddy · 2020-10-06T10:21:35Z

unused usePemCerts?

omkreddy · 2020-10-06T11:01:05Z

Can we remove empty else block?

omkreddy · 2020-10-06T11:22:55Z

SslConfigs.SSL_TRUSTSTORE_CERTIFICATES_CONFIG => SslConfigs.SSL_TRUSTSTORE_CERTIFICATES_DOC

omkreddy · 2020-10-06T11:23:37Z

SslConfigs.SSL_TRUSTSTORE_CERTIFICATES_CONFIG => SslConfigs.SSL_TRUSTSTORE_CERTIFICATES_DOC

omkreddy · 2020-10-06T12:21:09Z

unused node variable here and other methods.

fixed these and few other warnings

rajinisivaram · 2020-10-06T14:56:53Z

@omkreddy Thanks for the review, have addressed the comments.

rajinisivaram · 2020-10-06T18:12:48Z

@omkreddy Thanks for the review, merging to trunk.

apache#9345) Adds support for SSL key and trust stores to be specified in PEM format either as files or directly as configuration values. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>

* commit '2804257fe221f37e5098bd': (67 commits) KAFKA-10562: Properly invoke new StateStoreContext init (apache#9388) MINOR: trivial cleanups, javadoc errors, omitted StateStore tests, etc. (apache#8130) KAFKA-10564: only process non-empty task directories when internally cleaning obsolete state stores (apache#9373) KAFKA-9274: fix incorrect default value for `task.timeout.ms` config (apache#9385) KAFKA-10362: When resuming Streams active task with EOS, the checkpoint file is deleted (apache#9247) KAFKA-10028: Implement write path for feature versioning system (KIP-584) (apache#9001) KAFKA-10402: Upgrade system tests to python3 (apache#9196) KAFKA-10186; Abort transaction with pending data with TransactionAbortedException (apache#9280) MINOR: Remove `TargetVoters` from `DescribeQuorum` (apache#9376) Revert "KAFKA-10469: Resolve logger levels hierarchically (apache#9266)" MINOR: Don't publish javadocs for raft module (apache#9336) KAFKA-9929: fix: add missing default implementations (apache#9321) KAFKA-10188: Prevent SinkTask::preCommit from being called after SinkTask::stop (apache#8910) KAFKA-10338; Support PEM format for SSL key and trust stores (KIP-651) (apache#9345) KAFKA-10527; Voters should not reinitialize as leader in same epoch (apache#9348) MINOR: Refactor unit tests around RocksDBConfigSetter (apache#9358) KAFKA-6733: Printing additional ConsumerRecord fields in DefaultMessageFormatter (apache#9099) MINOR: Annotate test BlockingConnectorTest as integration test (apache#9379) MINOR: Fix failing test due to KAFKA-10556 PR (apache#9372) KAFKA-10439: Connect's Values to parse BigInteger as Decimal with zero scale. (apache#9320) ...

apache#9345) Adds support for SSL key and trust stores to be specified in PEM format either as files or directly as configuration values. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>

teabot · 2021-09-10T15:32:29Z

How is PEM certificate renewal possible on the producer/consumer client? Is this documented anywhere?

amccague · 2021-11-29T12:16:29Z

+            else if (keyPassword == null)
+                throw new InvalidConfigurationException("SSL PEM key store is specified, but key password is not specified.");


Is there an intention here to not allow a FileBasedPemStore containing an unencrypted private key when a key could be provided to PemStore with a null keyPassword?

vendamere · 2022-03-14T14:44:14Z

Was this change meant to work for the schema registry client as well? I've tried connecting to the schema registry using these settings, but it fails with a message indicating it's looking for a file still.

teabot · 2022-03-14T17:40:51Z

See: confluentinc/schema-registry#2062

…

On Mon, 14 Mar 2022 at 14:44, Peter Vendamere ***@***.***> wrote: Was this change meant to work for the schema registry client as well? I've tried connecting to the schema registry using these settings, but it fails with a message indicating it's looking for a file still. — Reply to this email directly, view it on GitHub <#9345 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABX4VWRLMTKQFQABRUB553U75GETANCNFSM4R43VXXA> . You are receiving this because you commented.Message ID: ***@***.***>

bachmanity1 · 2024-01-27T13:33:03Z

How is PEM certificate renewal possible on the producer/consumer client? Is this documented anywhere?

Hi @teabot , could find the answer to your question? I have also faced a similar. More precisely, is it possible to update the certificate without requiring a restart of the broker?

teabot · 2024-01-27T21:33:50Z

Hello, It's been quite a while since I looked at this. We ended up taking the PEM approach into the Confluent Schema registry also ( https://medium.com/expedia-group-tech/kafka-schema-registry-pem-authentication-bb434f32f99f). IIRC, one simply disconnects, and reconnects with the new certificate, but it's been a while so I might be mistaken. Elliot.

…

On Sat, 27 Jan 2024 at 13:33, bachmanity1 ***@***.***> wrote: How is PEM certificate renewal possible on the producer/consumer client? Is this documented anywhere? Hi @teabot <https://github.com/teabot> , could find the answer to your question? I have also faced a similar. More precisely, is it possible to update the certificate without requiring a restart of the broker? — Reply to this email directly, view it on GitHub <#9345 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABX4VTDVUMUPNELFSBVMGLYQT6ZVAVCNFSM4R43VXXKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJRGMYTKNZVG44A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

rajinisivaram requested a review from omkreddy September 28, 2020 19:01

rajinisivaram force-pushed the KAFKA-10338-ssl-pem branch from 0594fc4 to 671a2a5 Compare September 29, 2020 09:30

omkreddy approved these changes Oct 6, 2020

View reviewed changes

rajinisivaram added 3 commits October 6, 2020 15:35

KAFKA-10338; Support PEM format for SSL key and trust stores (KIP-651)

ba3773c

Add new SSL configs to KafkaConfig

b203a83

Address review comments

19b83fa

rajinisivaram force-pushed the KAFKA-10338-ssl-pem branch from 2f229f1 to 19b83fa Compare October 6, 2020 14:55

rajinisivaram merged commit 7be8bd8 into apache:trunk Oct 6, 2020

bplommer mentioned this pull request Mar 1, 2021

Create KafkaCredentialStore typelevel/fs2-kafka#530

Merged

amccague reviewed Nov 29, 2021

View reviewed changes

igpetrov mentioned this pull request Oct 14, 2022

[Kafka] [Discover]: Kafka MVP for producer and consumer camunda/connector-kafka#3

Closed

3 tasks

		else if (keyPassword == null)
		throw new InvalidConfigurationException("SSL PEM key store is specified, but key password is not specified.");

Conversation

rajinisivaram commented Sep 28, 2020

Committer Checklist (excluded from commit message)

Uh oh!

rajinisivaram commented Sep 29, 2020

Uh oh!

omkreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rajinisivaram commented Oct 6, 2020

Uh oh!

rajinisivaram commented Oct 6, 2020

Uh oh!

teabot commented Sep 10, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vendamere commented Mar 14, 2022

Uh oh!

teabot commented Mar 14, 2022 via email

Uh oh!

bachmanity1 commented Jan 27, 2024

Uh oh!

teabot commented Jan 27, 2024 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants