Skip to content

KAFKA-14784: Connect offset reset REST API#13818

Merged
C0urante merged 7 commits intoapache:trunkfrom
yashmayya:KAFKA-14784-offset-reset-api
Jun 23, 2023
Merged

KAFKA-14784: Connect offset reset REST API#13818
C0urante merged 7 commits intoapache:trunkfrom
yashmayya:KAFKA-14784-offset-reset-api

Conversation

@yashmayya
Copy link
Copy Markdown
Contributor

@yashmayya yashmayya commented Jun 6, 2023

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@yashmayya yashmayya added connect kip Requires or implements a KIP labels Jun 6, 2023
@yashmayya
Copy link
Copy Markdown
Contributor Author

Sink connector offsets alter requests:

ConnectorsResource::alterConnectorOffsets -> AbstractHerder::alterConnectorOffsets -> (Distributed|Standalone)Herder::modifyConnectorOffsets -> Worker::alterConnectorOffsets -> Worker::modifySinkConnectorOffsets -> Worker::alterSinkConnectorOffset

Source connector offsets alter requests:

ConnectorsResource::alterConnectorOffsets -> AbstractHerder::alterConnectorOffsets -> (Distributed|Standalone)Herder::modifyConnectorOffsets -> Worker::alterConnectorOffsets -> Worker::modifySourceConnectorOffsets

Sink connector offsets reset requests:

ConnectorsResource::resetConnectorOffsets -> AbstractHerder::resetConnectorOffsets -> (Distributed|Standalone)Herder::modifyConnectorOffsets -> Worker::resetConnectorOffsets -> Worker::modifySinkConnectorOffsets -> Worker::resetSinkConnectorOffsets

Source connector offsets reset requests:

ConnectorsResource::resetConnectorOffsets -> AbstractHerder::resetConnectorOffsets -> (Distributed|Standalone)Herder::modifyConnectorOffsets -> Worker::resetConnectorOffsets -> Worker::modifySourceConnectorOffsets

The current flows for altering and resetting offsets along with the use of null offsets in multiple places to distinguish between alter and reset offsets requests might seem a little clunky (especially for sink connectors), but I've tried to optimize for code re-use in both the herder implementations as well as the worker.

Comment thread connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java Outdated
@yashmayya yashmayya requested a review from C0urante June 6, 2023 12:11
Copy link
Copy Markdown
Contributor

@C0urante C0urante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thanks Yash! I've made it through all of the functional changes and the integration tests; will review the unit tests in the next pass.

Comment thread connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java Outdated
Comment thread connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java Outdated
Comment thread connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java Outdated
Comment thread connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java Outdated
Comment thread connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java Outdated
@yashmayya yashmayya force-pushed the KAFKA-14784-offset-reset-api branch from 2170332 to 397b622 Compare June 7, 2023 18:08
…ffsets; use a timer to bound total runtime for modifying connector offsets in the Worker; add integration test to verify various invalid offset scenarios while attempting to alter connector offsets; other minor logging and Javadoc improvements
@yashmayya yashmayya force-pushed the KAFKA-14784-offset-reset-api branch from e8fda70 to 355c687 Compare June 9, 2023 09:56
Copy link
Copy Markdown
Contributor

@C0urante C0urante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great! Some minor comments and then this should be ready to merge (pending CI build). Thanks Yash!

Comment thread connect/runtime/src/test/java/org/apache/kafka/connect/runtime/WorkerTest.java Outdated
@yashmayya yashmayya force-pushed the KAFKA-14784-offset-reset-api branch from 7907f2c to cd4c1f2 Compare June 23, 2023 13:00
Copy link
Copy Markdown
Contributor

@C0urante C0urante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯 LGTM, thanks Yash!

@C0urante
Copy link
Copy Markdown
Contributor

Test failures appear unrelated; merging...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connect kip Requires or implements a KIP

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants