KAFKA-8523 Enabling InsertField transform to be used with tombstone events by gunnarmorling · Pull Request #6914 · apache/kafka

gunnarmorling · 2019-06-11T10:29:59Z

Fixes https://issues.apache.org/jira/browse/KAFKA-8523

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

gunnarmorling · 2019-06-11T10:58:31Z

Hey @rhauch, a minor fix to the InsertFieldSMT -- Feedback welcome!

ftardif · 2019-09-17T12:13:07Z

So, does an insert config on a tombstone <k,null> will still produce a <k, null> (hence ignored), or will it produce a <k, v> where v contains only the inserted fields. If it does not propagate the null value, I am concerned it will skew the elasticsearch connector config behavior.on.null.values
that can be used to explicitly delete index entries on null.

rhauch

Thanks, @gunnarmorling! This looks great, but I have two minor suggestions.

ftardif · 2019-09-18T16:55:22Z

before merging this PR, can we discuss the impact of applying transformation on null values? I strongly believe we should rather propagate the null values untouched even when transformations are configured.

see: https://issues.apache.org/jira/browse/KAFKA-8523?focusedCommentId=16931777&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16931777

rhauch

@ftardif, please see the discussion on https://issues.apache.org/jira/browse/KAFKA-8523. I think @gunnarmorling, you, and I all agree that this PR should be changed to pass through and not modify tombstone records.

This PR has not yet been approved, but just to be safe I've marked this PR as Request changes.

ftardif · 2019-09-25T13:57:52Z

@gunnarmorling are you planning to provide an adjusted PR soon in order to start testing this functionality?

gunnarmorling · 2019-09-25T14:01:36Z

Yes, I'll try and send something in the course of the week. Thanks for the reminder!

…

gunnarmorling · 2019-09-27T20:04:34Z

@rhauch, @ftardif, I've updated the PR and force-pushed as per the discussion on the JIRA issue; tombstones are returned unmodified now by the InsertField transformation. I have also addressed the other review remarks regarding imports and method naming.

ftardif · 2019-10-01T12:49:08Z

thanks @gunnarmorling . What are the next steps to get that PR merged?

gunnarmorling · 2019-10-01T12:52:58Z

Good question; I suppose approval by @rhauch or any other committer should do the trick.

rhauch

LGTM. Thanks, @gunnarmorling!

I plan to merge shortly.

gunnarmorling · 2019-10-02T16:35:28Z

Excellent! To be sure, would that release be 2.4? Is there a way to learn about the timelines for code freezes for future reference? Thanks again!

…

rhauch · 2019-10-02T17:40:15Z

@gunnarmorling yes, I hope to get this into the AK 2.4.0 release. For information about release plans, checkout the dev list email thread with subject [DISCUSS] Apache Kafka 2.4.0 release (or the specific release you're interested in), which points to the AK 2.4.0 release plan.

rhauch

LGTM.

…vents (#6914) * KAFKA-8523 Avoiding raw type usage * KAFKA-8523 Gracefully handling tombstone events in InsertField SMT

rhauch · 2019-10-03T19:25:50Z

Merged back as far back as the 1.0 branch. See https://issues.apache.org/jira/browse/KAFKA-8523 for details.

gunnarmorling · 2019-10-03T19:36:30Z

Excellent; thanks for merging and backporting, Randall!

…

gunnarmorling force-pushed the KAFKA-8523 branch from 0250049 to cd2626b Compare June 11, 2019 10:30

rhauch reviewed Sep 17, 2019

View reviewed changes

Comment thread connect/transforms/src/main/java/org/apache/kafka/connect/transforms/InsertField.java Outdated

Comment thread connect/transforms/src/test/java/org/apache/kafka/connect/transforms/InsertFieldTest.java Outdated

rhauch requested changes Sep 18, 2019

View reviewed changes

KAFKA-8523 Avoiding raw type usage

63762e8

gunnarmorling force-pushed the KAFKA-8523 branch from cd2626b to f430dc1 Compare September 27, 2019 20:02

KAFKA-8523 Gracefully handling tombstone events in InsertField SMT

c327e31

gunnarmorling force-pushed the KAFKA-8523 branch from f430dc1 to c327e31 Compare September 27, 2019 20:03

rhauch reviewed Oct 2, 2019

View reviewed changes

rhauch approved these changes Oct 2, 2019

View reviewed changes

rhauch merged commit e767703 into apache:trunk Oct 3, 2019

rhauch pushed a commit that referenced this pull request Oct 3, 2019

KAFKA-8523 Enabling InsertField transform to be used with tombstone e…

6e153c4

…vents (#6914) * KAFKA-8523 Avoiding raw type usage * KAFKA-8523 Gracefully handling tombstone events in InsertField SMT

rhauch pushed a commit that referenced this pull request Oct 3, 2019

KAFKA-8523 Enabling InsertField transform to be used with tombstone e…

4377367

…vents (#6914) * KAFKA-8523 Avoiding raw type usage * KAFKA-8523 Gracefully handling tombstone events in InsertField SMT

rhauch pushed a commit that referenced this pull request Oct 3, 2019

KAFKA-8523 Enabling InsertField transform to be used with tombstone e…

95455bd

…vents (#6914) * KAFKA-8523 Avoiding raw type usage * KAFKA-8523 Gracefully handling tombstone events in InsertField SMT

rhauch pushed a commit that referenced this pull request Oct 3, 2019

KAFKA-8523 Enabling InsertField transform to be used with tombstone e…

a4de733

…vents (#6914) * KAFKA-8523 Avoiding raw type usage * KAFKA-8523 Gracefully handling tombstone events in InsertField SMT

rhauch pushed a commit that referenced this pull request Oct 3, 2019

KAFKA-8523 Enabling InsertField transform to be used with tombstone e…

c2af356

…vents (#6914) * KAFKA-8523 Avoiding raw type usage * KAFKA-8523 Gracefully handling tombstone events in InsertField SMT

rhauch pushed a commit that referenced this pull request Oct 3, 2019

KAFKA-8523 Enabling InsertField transform to be used with tombstone e…

87736fe

…vents (#6914) * KAFKA-8523 Avoiding raw type usage * KAFKA-8523 Gracefully handling tombstone events in InsertField SMT

This was referenced Mar 11, 2020

KAFKA-9706: Handle null keys/values in Flatten transformation #8279

Merged

KAFKA-9707: Fix InsertField.Key should apply to keys of tombstone records #8280

Merged

kkonstantine added the connect label Oct 16, 2020

Conversation

gunnarmorling commented Jun 11, 2019

Committer Checklist (excluded from commit message)

Uh oh!

gunnarmorling commented Jun 11, 2019

Uh oh!

ftardif commented Sep 17, 2019

Uh oh!

rhauch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ftardif commented Sep 18, 2019

Uh oh!

rhauch left a comment

Choose a reason for hiding this comment

Uh oh!

ftardif commented Sep 25, 2019

Uh oh!

gunnarmorling commented Sep 25, 2019 via email

Uh oh!

gunnarmorling commented Sep 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ftardif commented Oct 1, 2019

Uh oh!

gunnarmorling commented Oct 1, 2019

Uh oh!

rhauch left a comment

Choose a reason for hiding this comment

Uh oh!

gunnarmorling commented Oct 2, 2019 via email

Uh oh!

rhauch commented Oct 2, 2019

Uh oh!

rhauch left a comment

Choose a reason for hiding this comment

Uh oh!

rhauch commented Oct 3, 2019

Uh oh!

gunnarmorling commented Oct 3, 2019 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gunnarmorling commented Sep 27, 2019 •

edited

Loading