KAFKA-12829: Remove deprecated Topology#addProcessor of old Processor API by mjsax · Pull Request #18154 · apache/kafka

mjsax · 2024-12-12T03:09:37Z

Must be cherry-picked to 4.0 branch.

mjsax · 2024-12-12T03:12:41Z

 import org.apache.kafka.test.MockApiProcessor;
 import org.apache.kafka.test.MockApiProcessorSupplier;
-import org.apache.kafka.test.MockInternalNewProcessorContext;
+import org.apache.kafka.test.MockInternalProcessorContext;


Follow up clean from #18103

Rename this class, what result in lot of updates in many files...

mjsax · 2024-12-12T03:13:13Z



-public class KStreamNewProcessorApiTest {
+public class KStreamProcessorApiTest {


This is a similar update -- we don't have the old Processor API any longer, so dropping the New from this name, too.

mjsax · 2024-12-12T03:14:13Z

-        props.setProperty(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
-        props.setProperty(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
+        props.setProperty(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.StringSerde.class.getName());
+        props.setProperty(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class.getName());


Side cleanup to get rid of some IntelliJ warning. (also cleaner code.

Similar stuff below.

mjsax · 2024-12-12T03:14:33Z

-        assertEquals("value2", store.get("key2"));
-        assertEquals("value3", store.get("key3"));
-        assertNull(store.get("key4"));
-    }


Diff is just weird.. this was not removed

mjsax · 2024-12-12T03:15:15Z

-
-    @Deprecated // testing old PAPI
-    @Test
-    public void testDrivingConnectedStateStoreInDifferentProcessorsTopologyWithOldAPI() {


removed this (and some other tests) -- verified that we have equivalent tests for new Processor API for all of them

mjsax · 2024-12-12T03:15:33Z

-        assertEquals("key3", results.get(2).key);
-        assertEquals("value3", results.get(2).value);
-
-    }


Weird diff..

mjsax · 2024-12-12T03:16:29Z

                .addSink("sink", OUTPUT_TOPIC_1, new DroppingPartitioner(), "processor");
    }

-    @Deprecated // testing old PAPI


This annotation was wrong -- should have been @SuppressWarning("deprecation")

Updating this code to use new Processor API now.

mjsax · 2024-12-12T03:16:36Z

            .addSink("counts", OUTPUT_TOPIC_1, "processor");
    }

-    @Deprecated // testing old PAPI


mjsax · 2024-12-12T03:17:14Z

-     * A processor that stores each key-value pair in an in-memory key-value store registered with the context.
-     */
-    @SuppressWarnings("deprecation") // Old PAPI. Needs to be migrated.
-    protected static class OldAPIStatefulProcessor extends org.apache.kafka.streams.processor.AbstractProcessor<String, String> {


There is already a StatefulProcessor for the new PAPI -- nothing to be migrated -- deleting.

mjsax · 2024-12-12T03:17:40Z

    }

-    @SuppressWarnings("deprecation") // Old PAPI. Needs to be migrated.
-    private <K, V> org.apache.kafka.streams.processor.ProcessorSupplier<K, V> define(final org.apache.kafka.streams.processor.Processor<K, V> processor) {


just dropped this one -- not really useful to have..

mjsax · 2024-12-12T03:18:07Z

-    }
-
-    @SuppressWarnings("deprecation") // Old PAPI. Needs to be migrated.
-    private <K, V> org.apache.kafka.streams.processor.ProcessorSupplier<K, V> defineWithStoresOldAPI(final Supplier<org.apache.kafka.streams.processor.Processor<K, V>> supplier,


There is already defineWithStores() for new PAPI -- nothing to be migrated -- deleting

mjsax · 2024-12-12T03:18:17Z

    private <KIn, VIn, KOut, VOut> ProcessorSupplier<KIn, VIn, KOut, VOut> defineWithStores(final Supplier<Processor<KIn, VIn, KOut, VOut>> supplier,
                                                                                            final Set<StoreBuilder<?>> stores) {
-        return new ProcessorSupplier<KIn, VIn, KOut, VOut>() {
+        return new ProcessorSupplier<>() {


side cleanup

pegasas · 2024-12-13T08:09:10Z

        return topology
            .addSource("source", STRING_DESERIALIZER, STRING_DESERIALIZER, INPUT_TOPIC_1)
-            .addProcessor("processor", define(new OldAPIStatefulProcessor(storeName)), "source")
+            .addProcessor("processor", () -> new StatefulProcessor(storeName), "source")


Indeed, it looks like OldAPIStatefulProcessor has been corrected to StatefulProcessor for migrating to current API. Got it! Thanks for teaching!

chia7712 · 2024-12-15T07:43:41Z

@mjsax could you please fix the conflicts ?

… API

chia7712

@mjsax thanks for this patch. LGTM and one small question is left.

chia7712 · 2024-12-18T07:10:44Z

-import static org.mockito.Mockito.when;
-
-@SuppressWarnings("deprecation") // this is a test of a deprecated API
-public class MockProcessorContextTest {


The deprecated org.apache.kafka.streams.processor.MockProcessorContext still exists in the codebase. Should we retain the unit tests?

Good question. I suppose the trouble is that the unit test uses the removed classes, but the unit under test will not be removed yet. Can the tests be rewritten with transformers?

Good catch.

bbejeck

Thanks for the PR @mjsax - LGTM

lucasbru

Yes, LGTM modulo the question raised by Chia

mjsax · 2024-12-19T19:34:55Z

Java 23:

Found 1 test failures:
FAILED ❌ PlaintextAdminIntegrationTest > testConsumerGroupsDeprecatedConsumerGroupState(String, String).quorum=kraft.groupProtocol=consumer
Found 1 flaky test failures:
FLAKY ⚠️  SslTransportLayerTest > "testSelectorPollReadSize(Args).args=tlsProtocol=TLSv1.2, useInlinePem=false"

Java 17:

Found 5 test failures:
FAILED ❌ AuthorizerIntegrationTest > testConsumeUsingAssignWithNoAccess(String, String).quorum=kraft.groupProtocol=consumer
FAILED ❌ PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread(String, String).quorum=kraft.groupProtocol=classic
FAILED ❌ RemoteLogManagerTest > testRLMOpsWhenMetadataIsNotReady()
FAILED ❌ PlaintextAdminIntegrationTest > testConsumerGroupsDeprecatedConsumerGroupState(String, String).quorum=kraft.groupProtocol=consumer
FAILED ❌ PlaintextAdminIntegrationTest > testConsumerGroups(String, String).quorum=kraft.groupProtocol=consumer
Found 2 flaky test failures:
FLAKY ⚠️  TransactionsWithTieredStoreTest > "testBumpTransactionalEpochWithTV2Enabled(String, String, boolean).quorum=kraft, groupProtocol=consumer, isTV2Enabled=true"
FLAKY ⚠️  SslTransportLayerTest > "testSelectorPollReadSize(Args).args=tlsProtocol=TLSv1.2, useInlinePem=false"

… API (#18154) Reviewers: John Huang <pegasashjy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>, Bill Bejeck <bill@confluent.io>, Lucas Brutschy <lbrutschy@confluent.io>

mjsax · 2024-12-20T00:21:30Z

Merged to trunk and cherry-picked to 4.0 branch.

…e-old-protocol-versions * apache-github/trunk: KAFKA-18312: Added entityType: topicName to SubscribedTopicNames in ShareGroupHeartbeatRequest.json (apache#18285) HOTFIX: fix incompatible types: Optional<TimestampAndOffset> cannot be converted to Option<TimestampAndOffset> (apache#18284) MINOR Fix some test-catalog issues (apache#18272) KAFKA-18180: Move OffsetResultHolder to storage module (apache#18100) KAFKA-18301; Make coordinator records first class citizen (apache#18261) KAFKA-18262 Remove DefaultPartitioner and UniformStickyPartitioner (apache#18204) KAFKA-18296 Remove deprecated KafkaBasedLog constructor (apache#18257) KAFKA-12829: Remove old Processor and ProcessorSupplier interfaces (apache#18238) KAFKA-18292 Remove deprecated methods of UpdateFeaturesOptions (apache#18245) KAFKA-12829: Remove deprecated Topology#addProcessor of old Processor API (apache#18154) KAFKA-18035, KAFKA-18306, KAFKA-18092: Address TransactionsTest flaky tests (apache#18264) MINOR: change the default linger time in the new coordinator (apache#18274) KAFKA-18305: validate controller.listener.names is not in inter.broker.listener.name for kcontrollers (apache#18222) KAFKA-18207: Serde for handling transaction records (apache#18136) KAFKA-13722: Refactor Kafka Streams store interfaces (apache#18243) KAFKA-17131: Refactor TimeDefinitions (apache#18241) MINOR: Fix MessageFormatters (apache#18266) Mark flaky tests for Dec 18, 2024 (apache#18263)

… API (apache#18154) Reviewers: John Huang <pegasashjy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>, Bill Bejeck <bill@confluent.io>, Lucas Brutschy <lbrutschy@confluent.io>

mjsax added the streams label Dec 12, 2024

mjsax commented Dec 12, 2024

View reviewed changes

mjsax mentioned this pull request Dec 12, 2024

KAFKA-12829: Remove deprecated Topology#addProcessor of old Processor API #17190

Closed

3 tasks

pegasas reviewed Dec 13, 2024

View reviewed changes

pegasas approved these changes Dec 13, 2024

View reviewed changes

KAFKA-12829: Remove deprecated Topology#addProcessor of old Processor…

e63d9a3

… API

mjsax force-pushed the kafka-12829-remove-topology-process branch from 0b3c223 to e63d9a3 Compare December 16, 2024 19:12

mjsax mentioned this pull request Dec 17, 2024

KAFKA-12829: Remove old Processor and ProcessorSupplier interfaces #18238

Merged

chia7712 approved these changes Dec 18, 2024

View reviewed changes

bbejeck approved these changes Dec 18, 2024

View reviewed changes

lucasbru approved these changes Dec 18, 2024

View reviewed changes

review comments

2423c65

mjsax merged commit 9aa900f into apache:trunk Dec 20, 2024

mjsax deleted the kafka-12829-remove-topology-process branch December 20, 2024 00:11



		public class KStreamNewProcessorApiTest {
		public class KStreamProcessorApiTest {

Conversation

mjsax commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 commented Dec 15, 2024

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bbejeck left a comment

Choose a reason for hiding this comment

Uh oh!

lucasbru left a comment

Choose a reason for hiding this comment

Uh oh!

mjsax commented Dec 19, 2024

Uh oh!

mjsax commented Dec 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mjsax commented Dec 12, 2024 •

edited

Loading