-
Notifications
You must be signed in to change notification settings - Fork 15.1k
KAFKA-4794: Add access to OffsetStorageReader from SourceConnector #2604
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
601e6f6
KAFKA-4794: Add access to OffsetStorageReader from SourceConnector
fhussonnois 868d6fe
KAFKA-4794: Avoid unnecessary lines
rhauch d1d6dee
KAFKA-4794: Simplify SourceConnectorContext and SinkConnectorContext …
rhauch d8b4c26
KAFKA-4794: Add unit tests for Connector, SinkConnector and SourceCon…
rhauch 7784286
KAFKA-4794: Removed ambiguous case of connector not implementing sour…
rhauch File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
25 changes: 25 additions & 0 deletions
25
connect/api/src/main/java/org/apache/kafka/connect/sink/SinkConnectorContext.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package org.apache.kafka.connect.sink; | ||
|
|
||
| import org.apache.kafka.connect.connector.ConnectorContext; | ||
|
|
||
| /** | ||
| * A context to allow a {@link SinkConnector} to interact with the Kafka Connect runtime. | ||
| */ | ||
| public interface SinkConnectorContext extends ConnectorContext { | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
32 changes: 32 additions & 0 deletions
32
connect/api/src/main/java/org/apache/kafka/connect/source/SourceConnectorContext.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package org.apache.kafka.connect.source; | ||
|
|
||
| import org.apache.kafka.connect.connector.ConnectorContext; | ||
| import org.apache.kafka.connect.storage.OffsetStorageReader; | ||
|
|
||
| /** | ||
| * A context to allow a {@link SourceConnector} to interact with the Kafka Connect runtime. | ||
| */ | ||
| public interface SourceConnectorContext extends ConnectorContext { | ||
|
|
||
| /** | ||
| * Returns the {@link OffsetStorageReader} for this SourceConnectorContext. | ||
| * @return the OffsetStorageReader for this connector. | ||
| */ | ||
|
rhauch marked this conversation as resolved.
Outdated
|
||
| OffsetStorageReader offsetStorageReader(); | ||
| } | ||
88 changes: 88 additions & 0 deletions
88
connect/api/src/test/java/org/apache/kafka/connect/connector/ConnectorTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package org.apache.kafka.connect.connector; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.HashMap; | ||
| import java.util.List; | ||
| import java.util.Map; | ||
|
|
||
| import org.junit.Before; | ||
| import org.junit.Test; | ||
|
|
||
| public abstract class ConnectorTest { | ||
|
|
||
| protected ConnectorContext context; | ||
| protected Connector connector; | ||
| protected AssertableConnector assertableConnector; | ||
|
|
||
| @Before | ||
| public void beforeEach() { | ||
| connector = createConnector(); | ||
| context = createContext(); | ||
| assertableConnector = (AssertableConnector) connector; | ||
| } | ||
|
|
||
| @Test | ||
| public void shouldInitializeContext() { | ||
| connector.initialize(context); | ||
| assertableConnector.assertInitialized(); | ||
| assertableConnector.assertContext(context); | ||
| assertableConnector.assertTaskConfigs(null); | ||
| } | ||
|
|
||
| @Test | ||
| public void shouldInitializeContextWithTaskConfigs() { | ||
| List<Map<String, String>> taskConfigs = new ArrayList<>(); | ||
| connector.initialize(context, taskConfigs); | ||
| assertableConnector.assertInitialized(); | ||
| assertableConnector.assertContext(context); | ||
| assertableConnector.assertTaskConfigs(taskConfigs); | ||
| } | ||
|
|
||
| @Test | ||
| public void shouldStopAndStartWhenReconfigure() { | ||
| Map<String, String> props = new HashMap<>(); | ||
| connector.initialize(context); | ||
| assertableConnector.assertContext(context); | ||
| assertableConnector.assertStarted(false); | ||
| assertableConnector.assertStopped(false); | ||
| connector.reconfigure(props); | ||
| assertableConnector.assertStarted(true); | ||
| assertableConnector.assertStopped(true); | ||
| assertableConnector.assertProperties(props); | ||
| } | ||
|
|
||
| protected abstract ConnectorContext createContext(); | ||
|
|
||
| protected abstract Connector createConnector(); | ||
|
|
||
| public interface AssertableConnector { | ||
|
|
||
| void assertContext(ConnectorContext expected); | ||
|
|
||
| void assertInitialized(); | ||
|
|
||
| void assertTaskConfigs(List<Map<String, String>> expectedTaskConfigs); | ||
|
|
||
| void assertStarted(boolean expected); | ||
|
|
||
| void assertStopped(boolean expected); | ||
|
|
||
| void assertProperties(Map<String, String> expected); | ||
| } | ||
| } |
146 changes: 146 additions & 0 deletions
146
connect/api/src/test/java/org/apache/kafka/connect/sink/SinkConnectorTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package org.apache.kafka.connect.sink; | ||
|
|
||
| import java.util.List; | ||
| import java.util.Map; | ||
|
|
||
| import org.apache.kafka.common.config.ConfigDef; | ||
| import org.apache.kafka.connect.connector.ConnectorContext; | ||
| import org.apache.kafka.connect.connector.ConnectorTest; | ||
| import org.apache.kafka.connect.connector.Task; | ||
|
|
||
| import static org.junit.Assert.assertEquals; | ||
| import static org.junit.Assert.assertSame; | ||
| import static org.junit.Assert.assertTrue; | ||
|
|
||
| public class SinkConnectorTest extends ConnectorTest { | ||
|
|
||
| @Override | ||
| protected TestSinkConnectorContext createContext() { | ||
| return new TestSinkConnectorContext(); | ||
| } | ||
|
|
||
| @Override | ||
| protected TestSinkConnector createConnector() { | ||
| return new TestSinkConnector(); | ||
| } | ||
|
|
||
| private static class TestSinkConnectorContext implements SinkConnectorContext { | ||
|
|
||
| @Override | ||
| public void requestTaskReconfiguration() { | ||
| // Unexpected in these tests | ||
| throw new UnsupportedOperationException(); | ||
| } | ||
|
|
||
| @Override | ||
| public void raiseError(Exception e) { | ||
| // Unexpected in these tests | ||
| throw new UnsupportedOperationException(); | ||
| } | ||
| } | ||
|
|
||
| protected static class TestSinkConnector extends SinkConnector implements ConnectorTest.AssertableConnector { | ||
|
|
||
| public static final String VERSION = "an entirely different version"; | ||
|
|
||
| private boolean initialized; | ||
| private List<Map<String, String>> taskConfigs; | ||
| private Map<String, String> props; | ||
| private boolean started; | ||
| private boolean stopped; | ||
|
|
||
| @Override | ||
| public String version() { | ||
| return VERSION; | ||
| } | ||
|
|
||
| @Override | ||
| public void initialize(ConnectorContext ctx) { | ||
| super.initialize(ctx); | ||
| initialized = true; | ||
| this.taskConfigs = null; | ||
| } | ||
|
|
||
| @Override | ||
| public void initialize(ConnectorContext ctx, List<Map<String, String>> taskConfigs) { | ||
| super.initialize(ctx, taskConfigs); | ||
| initialized = true; | ||
| this.taskConfigs = taskConfigs; | ||
| } | ||
|
|
||
| @Override | ||
| public void start(Map<String, String> props) { | ||
| this.props = props; | ||
| started = true; | ||
| } | ||
|
|
||
| @Override | ||
| public Class<? extends Task> taskClass() { | ||
| return null; | ||
| } | ||
|
|
||
| @Override | ||
| public List<Map<String, String>> taskConfigs(int maxTasks) { | ||
| return null; | ||
| } | ||
|
|
||
| @Override | ||
| public void stop() { | ||
| stopped = true; | ||
| } | ||
|
|
||
| @Override | ||
| public ConfigDef config() { | ||
| return new ConfigDef() | ||
| .define("required", ConfigDef.Type.STRING, ConfigDef.Importance.HIGH, "required docs") | ||
| .define("optional", ConfigDef.Type.STRING, "defaultVal", ConfigDef.Importance.HIGH, "optional docs"); | ||
| } | ||
|
|
||
| @Override | ||
| public void assertContext(ConnectorContext expected) { | ||
| assertSame(expected, context); | ||
| assertSame(expected, context()); | ||
| } | ||
|
|
||
| @Override | ||
| public void assertInitialized() { | ||
| assertTrue(initialized); | ||
| } | ||
|
|
||
| @Override | ||
| public void assertTaskConfigs(List<Map<String, String>> expectedTaskConfigs) { | ||
| assertSame(expectedTaskConfigs, taskConfigs); | ||
| } | ||
|
|
||
| @Override | ||
| public void assertStarted(boolean expected) { | ||
| assertEquals(expected, started); | ||
| } | ||
|
|
||
| @Override | ||
| public void assertStopped(boolean expected) { | ||
| assertEquals(expected, stopped); | ||
| } | ||
|
|
||
| @Override | ||
| public void assertProperties(Map<String, String> expected) { | ||
| assertSame(expected, props); | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.