KAFKA-16505: Adding dead letter queue in Kafka Streams by Dabz · Pull Request #17942 · apache/kafka

Dabz · 2024-11-25T16:04:50Z

First Pull Request to implement KIP-1034 to add support of Dead Letter
Queue in Kafka Streams. A second PR will store the source record key and
value byte[] in the context to add it to the DLQ records.

Reviewers: Lucas Brutschy lbrutschy@confluent.io, Bruno Cadonna
cadonna@apache.org

cadonna

Thanks for the PR, @Dabz !

I started to review it, but did not finish since I need to think about how this KIP uses mutable enums which I guess is an anti-pattern.
Nevertheless, I wanted to give my first comments.

cadonna · 2024-11-27T12:57:48Z

    public static final String ENABLE_METRICS_PUSH_DOC = "Whether to enable pushing of internal client metrics for (main, restore, and global) consumers, producers, and admin clients." +
        " The cluster must have a client metrics subscription which corresponds to a client.";

+    public static final String ERRORS_DEAD_LETTER_QUEUE_TOPIC_NAME_CONFIG = "errors.dead.letter.queue.topic.name";


Could you please add unit tests?

cadonna · 2024-11-27T13:26:07Z

You cannot add a class to the public API that was not mentioned in the KIP.

I renamed this class to ExceptionHandlerUtils, make it packaged protected, and made all methods statics to avoid inheritance.

cadonna · 2024-11-27T13:26:49Z

+    public static final String HEADER_ERRORS_OFFSET_NAME = "__streams.errors.offset";
+
+
+    public boolean shouldBuildDeadLetterQueueRecord() {


Why is this public? It is only used within this class.

Changed it to package protected

cadonna · 2024-11-27T14:30:42Z

+            return this;
+        }
+
+        public List<ProducerRecord<byte[], byte[]>> drainDeadLetterQueueRecords() {


According to the KIP this should be named deadLetterQueueReords().

Good point, I renamed it

cadonna · 2024-11-27T14:35:12Z

+/**
+ * {@code CommonExceptionHandler} Contains utilities method that could be used by all exception handlers
+ */
+public class CommonExceptionHandler implements Configurable {


It would be better to define this class as a immutable utility class in the internal package that provides some utility methods to the default exception handlers. That means, no inheritance.

lucasbru · 2024-12-03T08:57:01Z

Indeed - using a single collection of dead letter records here that will shared across all stream threads doesn't seem like the right approach to me. Was this intended?

I'm not sure if there is a way to replace the enum by a class of some kind where we can properly define a non-static collection of records and remain binary compatible.

It seems to me that this needs a new design proposal and an updated KIP?

cadonna · 2024-12-03T12:45:04Z

An option to fix this might be to add a new method deadLetterQueueRecords() to the handler interface that returns a list of records to add to the dead letter queue. By default the list is empty.

cadonna

Thanks for the updates @sebastienviale !

I did a first pass. Here my comments.

cadonna · 2025-01-16T13:39:26Z

-        return handle(record, exception);
+        throw new UnsupportedOperationException();


Shouldn't this still call the other deprecated method?
Imagine a user implemented

handle(final ProducerRecord<byte[], byte[]> record, final Exception exception)

but not

ProductionExceptionHandlerResponse handle(final ErrorHandlerContext context, final ProducerRecord<byte[], byte[]> record, final Exception exception)

Streams would throw an UnsupportedOperationException although it did not before upgrading to this version.

Maybe it would be beneficial to add some unit tests that verify this redirection. With such unit tests, this removal would had happened without some thoughts about why the test failed.

indeed, I added a Unit Test that failed if an UnsupportedOperationException is thrown

cadonna · 2025-01-16T13:54:18Z

+    /**
+     * Represents the result of handling a production exception.
+     * <p>
+     * The {@code Response} class encapsulates a {@link ProductionExceptionHandlerResponse},


Suggested change

* The {@code Response} class encapsulates a {@link ProductionExceptionHandlerResponse},

* The {@code Response} class encapsulates a {@link Result},

cadonna · 2025-01-16T13:58:56Z

+     * {@link ProducerRecord} instances to be sent to a dead letter queue.
+     * </p>
+     */
+    class Response {


Could you please add some unit tests for this class?

cadonna · 2025-01-16T14:14:27Z

+                                                                            final ErrorHandlerContext context,
+                                                                            final Exception e) {
+        if (deadLetterQueueTopicName == null) {
+            throw new InvalidConfigurationException(String.format("%s can not be null while building DeadLetterQueue record", StreamsConfig.ERRORS_DEAD_LETTER_QUEUE_TOPIC_NAME_CONFIG));


Suggested change

throw new InvalidConfigurationException(String.format("%s can not be null while building DeadLetterQueue record", StreamsConfig.ERRORS_DEAD_LETTER_QUEUE_TOPIC_NAME_CONFIG));

throw new InvalidConfigurationException(String.format("%s cannot be null while building dead letter queue record", StreamsConfig.ERRORS_DEAD_LETTER_QUEUE_TOPIC_NAME_CONFIG));

cadonna · 2025-01-16T14:14:52Z

+    static ProducerRecord<byte[], byte[]> buildDeadLetterQueueRecord(final String deadLetterQueueTopicName,
+                                                                            final byte[] key,
+                                                                            final byte[] value,
+                                                                            final ErrorHandlerContext context,
+                                                                            final Exception e) {


Suggested change

static ProducerRecord<byte[], byte[]> buildDeadLetterQueueRecord(final String deadLetterQueueTopicName,

final byte[] key,

final byte[] value,

final ErrorHandlerContext context,

final Exception e) {

static ProducerRecord<byte[], byte[]> buildDeadLetterQueueRecord(final String deadLetterQueueTopicName,

final byte[] key,

final byte[] value,

final ErrorHandlerContext context,

final Exception e) {

cadonna · 2025-01-16T14:48:13Z

+    @SuppressWarnings("deprecation")
+    @Deprecated
    @Override
    public DeserializationHandlerResponse handle(final ErrorHandlerContext context,


You can remove those deprecated handler methods. They are not called anywhere in the Streams code.

cadonna · 2025-01-16T14:48:57Z

+    @Deprecated
    @Override
-    public ProcessingHandlerResponse handle(final ErrorHandlerContext context, final Record<?, ?> record, final Exception exception) {
+    public ProcessingHandlerResponse handle(final ErrorHandlerContext context,


You can remove this deprecated handler method. It is not called anywhere in the Streams code.

cadonna · 2025-01-16T14:49:32Z

+    public Response handleError(final ErrorHandlerContext context,
+                                final ConsumerRecord<byte[], byte[]> record,
+                                final Exception exception) {
+        log.warn(


This should be log.error(). It is failing, not resuming.

cadonna · 2025-01-16T14:49:53Z

+    @SuppressWarnings("deprecation")
+    @Deprecated
    @Override
    public DeserializationHandlerResponse handle(final ErrorHandlerContext context,


You can remove those deprecated handler methods. They are not called anywhere in the Streams code.

cadonna · 2025-01-16T14:50:32Z

+    public ProcessingHandlerResponse handle(final ErrorHandlerContext context,
+                                            final Record<?, ?> record, final Exception exception) {


You can remove this deprecated handler method. It is not called anywhere in the Streams code.

sebastienviale · 2025-01-24T15:22:46Z

Thanks for the updates @sebastienviale !

I did a first pass. Here my comments.

@cadonna thanks for your review, I pushed changes

This PR is part of the KIP-1034. It brings the support for the source raw key and the source raw value in the `ErrorHandlerContext`. Required by the routing to DLQ implemented by #17942. Reviewers: Bruno Cadonna <cadonna@apache.org> Co-authored-by: Damien Gasparina <d.gasparina@gmail.com>

This PR is part of the KIP-1034. It brings the support for the source raw key and the source raw value in the `ErrorHandlerContext`. Required by the routing to DLQ implemented by apache#17942. Reviewers: Bruno Cadonna <cadonna@apache.org> Co-authored-by: Damien Gasparina <d.gasparina@gmail.com>

Copilot

Pull Request Overview

Adds support for Dead Letter Queue (DLQ) in Kafka Streams by extending exception handler APIs and wiring DLQ record production throughout the core processing, deserialization, and production paths.

Introduced a new Response API (with deadLetterQueueRecords) for deserialization, processing, and production exception handlers.
Updated core classes (StreamTask, RecordDeserializer, RecordCollector, ProcessorNode) to emit DLQ records when configured.
Added ExceptionHandlerUtils for building DLQ records with appropriate headers and comprehensive tests validating DLQ behavior.

Reviewed Changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
streams/src/test/java/org/apache/kafka/test/MockRecordCollector.java	Added overload of `send` to collect serialized records for DLQ.
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java	Renamed handler methods, switched to `Response` type.
streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordDeserializerTest.java	Switched to new `Response` API and added DLQ tests.
streams/src/test/java/org/apache/kafka/streams/processor/internals/RecordCollectorTest.java	Switched to new `Response` API and added DLQ tests.
streams/src/test/java/org/apache/kafka/streams/processor/internals/ProcessorNodeTest.java	Switched to new `Response` API and added DLQ tests.
streams/src/test/java/org/apache/kafka/streams/errors/ExceptionHandlerUtilsTest.java	New tests for DLQ record headers.
streams/src/test/java/org/apache/kafka/streams/StreamsConfigTest.java	Added test for default DLQ config.
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java	Integrated DLQ sends into processing error flow.
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordDeserializer.java	Integrated DLQ sends into deserialization error flow.
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java	Integrated DLQ sends into production and serialization flows.
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollector.java	Added new `send` signature for DLQ.
streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorNode.java	Integrated DLQ sends into processing node error flow.
streams/src/main/java/org/apache/kafka/streams/errors/internals/ExceptionHandlerUtils.java	New utility for building DLQ records with headers.
streams/src/main/java/org/apache/kafka/streams/errors/ProductionExceptionHandler.java	Introduced `Response`, deprecated old API, added DLQ support.
streams/src/main/java/org/apache/kafka/streams/errors/ProcessingExceptionHandler.java	Introduced `Response`, deprecated old API, added DLQ support.
streams/src/main/java/org/apache/kafka/streams/errors/LogAndFailProcessingExceptionHandler.java	Added DLQ topic configuration and records.
streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueProcessingExceptionHandler.java	Added DLQ topic configuration and records.
streams/src/main/java/org/apache/kafka/streams/errors/LogAndFailExceptionHandler.java	Added DLQ topic configuration and records.
streams/src/main/java/org/apache/kafka/streams/errors/LogAndContinueExceptionHandler.java	Added DLQ topic configuration and records.
streams/src/main/java/org/apache/kafka/streams/errors/DefaultProductionExceptionHandler.java	Added DLQ topic configuration and records.
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java	Defined new `errors.dead.letter.queue.topic.name` config.
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/SwallowUnknownTopicErrorIntegrationTest.java	Switched to new `Response` API.
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/ProcessingExceptionHandlerIntegrationTest.java	Switched to new `Response` API.

Comments suppressed due to low confidence (1)

streams/src/test/java/org/apache/kafka/streams/StreamsConfigTest.java:1619

This test method is missing the @test annotation, so it will not be executed by the test runner.

    public void shouldSetDefaultDeadLetterQueue() {

lucasbru

Thanks for the PR! I left a few questions for my own understanding, and a few nits. Mostly looking good to me, though!

WIP Implementing KIP-1034 - adding new interfaces

# Conflicts: # streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java

lucasbru

LGTM!

@cadonna would you like to take another look?

@Dabz @sebastienviale I wonder if we could have a little integration tests for this feature, to test that it actually works end-to-end?

cadonna · 2025-07-21T07:30:49Z

@cadonna would you like to take another look?

@lucasbru I trust you completely. 🙂

lucasbru · 2025-07-21T11:47:31Z

I restarted the CI since it was triggering a flaky test.

I created https://issues.apache.org/jira/browse/KAFKA-16505 for tracking the integration test.

Implements KIP-1034 to add support of Dead Letter Queue in Kafka Streams. Reviewers: Lucas Brutschy <lbrutschy@confluent.io>, Bruno Cadonna <cadonna@apache.org> Co-authored-by: Sebastien Viale <sebastien.viale@michelin.com>

chia7712 · 2026-01-28T18:05:16Z

+        e.printStackTrace(stackTracePrintWriter);
+
+        try (final StringSerializer stringSerializer = new StringSerializer()) {
+            producerRecord.headers().add(HEADER_ERRORS_EXCEPTION_NAME, stringSerializer.serialize(null, e.toString()));


KIP-1034 states that this filed is for the "Name of the thrown exception", so we should use e.getClass().getName() instead, right?

Also, shouldn't we include the headers from the origin record in the DLQ record? I noticed KIP-1034 states "Existing context headers are automatically forwarded into the new DLQ record"

open #21370

Yes, you are right there is a mismatch between the KIP and the implementation here.

…21370) This patch fixed two mismatch between [KIP-1034](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1034%3A+Dead+letter+queue+in+Kafka+Streams#KIP1034:DeadletterqueueinKafkaStreams-ProposedChanges) and the actual implementation: - Incorrect value of `HEADER_ERRORS_EXCEPTION_NAME`, it should be the name of the thrown exception, not exception's String representation. - Original headers from record that causes exception should be added to dlq record headers. References: - #17942 (comment) - #17942 (comment) - [KIP-1034](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1034%3A+Dead+letter+queue+in+Kafka+Streams#KIP1034:DeadletterqueueinKafkaStreams-ProposedChanges) Reviewers: Lucas Brutschy <lbrutschy@confluent.io>

…pache#21370) This patch fixed two mismatch between [KIP-1034](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1034%3A+Dead+letter+queue+in+Kafka+Streams#KIP1034:DeadletterqueueinKafkaStreams-ProposedChanges) and the actual implementation: - Incorrect value of `HEADER_ERRORS_EXCEPTION_NAME`, it should be the name of the thrown exception, not exception's String representation. - Original headers from record that causes exception should be added to dlq record headers. References: - apache#17942 (comment) - apache#17942 (comment) - [KIP-1034](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1034%3A+Dead+letter+queue+in+Kafka+Streams#KIP1034:DeadletterqueueinKafkaStreams-ProposedChanges) Reviewers: Lucas Brutschy <lbrutschy@confluent.io>

…21378) This is a backport of PR #21370, with an additional checkstyle fix (adding a `final` keyword to make checkstyle works in 4.2 branch). --- This patch fixed two mismatch between [KIP-1034](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1034%3A+Dead+letter+queue+in+Kafka+Streams#KIP1034:DeadletterqueueinKafkaStreams-ProposedChanges) and the actual implementation: - Incorrect value of `HEADER_ERRORS_EXCEPTION_NAME`, it should be the name of the thrown exception, not exception's String representation. - Original headers from record that causes exception should be added to dlq record headers. References: - #17942 (comment) - #17942 (comment) - [KIP-1034](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1034%3A+Dead+letter+queue+in+Kafka+Streams#KIP1034:DeadletterqueueinKafkaStreams-ProposedChanges) Reviewers: Lucas Brutschy <lbrutschy@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>

github-actions bot added the streams label Nov 25, 2024

Dabz force-pushed the KAFKA-16505-Dead-letter-queue-in-Kafka-Streams branch from 4fc7717 to 5698e60 Compare November 25, 2024 16:15

cadonna reviewed Nov 27, 2024

View reviewed changes

cadonna reviewed Jan 16, 2025

View reviewed changes

sebastienviale force-pushed the KAFKA-16505-Dead-letter-queue-in-Kafka-Streams branch from 93cd94d to 93c831f Compare January 17, 2025 09:27

sebastienviale force-pushed the KAFKA-16505-Dead-letter-queue-in-Kafka-Streams branch from fc4ac3c to efa1cae Compare February 25, 2025 12:36

sebastienviale force-pushed the KAFKA-16505-Dead-letter-queue-in-Kafka-Streams branch from efa1cae to 739f443 Compare March 24, 2025 13:04

loicgreffier mentioned this pull request Jun 5, 2025

KAFKA-16505: Add source raw key and value #18739

Merged

sebastienviale force-pushed the KAFKA-16505-Dead-letter-queue-in-Kafka-Streams branch from 22a8d80 to 2c32c96 Compare June 5, 2025 11:59

lucasbru requested review from Copilot and lucasbru July 7, 2025 16:28

Copilot AI reviewed Jul 7, 2025

View reviewed changes

lucasbru reviewed Jul 11, 2025

View reviewed changes

Dabz and others added 12 commits July 17, 2025 09:46

KFK-16505

0c23328

WIP Implementing KIP-1034 - adding new interfaces

KAFKA-16505 Adding test for DLQ support

dc4069e

# Conflicts: # streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java

KAFKA-16505 Adding test for DLQ support

6a14db0

KAFKA-16505: add source raw bytes in processorContex

3cee49c

Implementing KIP-1034 - adding new interfaces

6c75500

KIP-1034 - adding new interfaces

d9c13fa

KAFKA-16505 fixing unit test

012c3a4

KAFKA-16505 simplifying default implementations

ace6816

KAFKA-16505 deprecated old handler response enums

4546d91

KAFKA-16505

d083b6a

KAFKA-16505 remove unused methods and add tests

ebb4440

KAFKA-16505

d1d41a1

sebastienviale added 4 commits July 17, 2025 09:47

KAFKA-16505: fix unit tests

899b878

KFK-16505 fix after rebase

288f286

KFK-16505 complete rawKey and rawValue

b55a6a5

KFK-16505

f1b8286

sebastienviale force-pushed the KAFKA-16505-Dead-letter-queue-in-Kafka-Streams branch from c1c6eea to f1b8286 Compare July 17, 2025 07:49

lucasbru changed the title ~~Kafka-16505: Adding dead letter queue in Kafka Streams~~ KAFKA-16505: Adding dead letter queue in Kafka Streams Jul 18, 2025

lucasbru approved these changes Jul 18, 2025

View reviewed changes

lucasbru merged commit cdc2d95 into apache:trunk Jul 21, 2025
36 of 38 checks passed

chia7712 reviewed Jan 28, 2026

View reviewed changes

unknowntpo mentioned this pull request Jan 29, 2026

MINOR: Correct exception naming and preserve original headers in DLQ #21370

Merged

unknowntpo mentioned this pull request Jan 31, 2026

MINOR: Correct exception naming and preserve original headers in DLQ #21378

Merged

		public static final String HEADER_ERRORS_OFFSET_NAME = "__streams.errors.offset";


		public boolean shouldBuildDeadLetterQueueRecord() {

		return handle(record, exception);
		throw new UnsupportedOperationException();

	* The {@code Response} class encapsulates a {@link ProductionExceptionHandlerResponse},
	* The {@code Response} class encapsulates a {@link Result},

	throw new InvalidConfigurationException(String.format("%s can not be null while building DeadLetterQueue record", StreamsConfig.ERRORS_DEAD_LETTER_QUEUE_TOPIC_NAME_CONFIG));
	throw new InvalidConfigurationException(String.format("%s cannot be null while building dead letter queue record", StreamsConfig.ERRORS_DEAD_LETTER_QUEUE_TOPIC_NAME_CONFIG));

		public ProcessingHandlerResponse handle(final ErrorHandlerContext context,
		final Record<?, ?> record, final Exception exception) {

Conversation

Dabz commented Nov 25, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cadonna left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucasbru commented Dec 3, 2024

Uh oh!

cadonna commented Dec 3, 2024

Uh oh!

cadonna left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sebastienviale commented Jan 24, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Dabz commented Nov 25, 2024 •

edited by github-actions bot

Loading