Skip to content

add sequenceName and currentCheckPoint for backwards compatibility#8864

Merged
fjy merged 10 commits intoapache:masterfrom
surekhasaharan:kafka-rolling-update
Nov 19, 2019
Merged

add sequenceName and currentCheckPoint for backwards compatibility#8864
fjy merged 10 commits intoapache:masterfrom
surekhasaharan:kafka-rolling-update

Conversation

@surekhasaharan
Copy link
Copy Markdown

Fixes #8837

Description

Added sequenceName and currentCheckPoint to CheckPointDataSourceMetadataAction for older version objects (<0.15) to work with newer version (<0.16).

This PR has:

  • been self-reviewed.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • been tested in a test Druid cluster.

return checkpointMetadata;
}

// For backwards compatibility
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can add a unit test for the serialization/deserialization between old and new DTOs

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, was trying to add a serde test for CheckPointDataSourceMetadataAction, but wasn't easy to construct an object because of abstract SeekableStreamDataSourceMetadata in this package.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One option is to manually create a JSON string and then deserialize it to a different version. If you don't want to manually create the JSON, then one option is to add a class in the test file that is the old DTO.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a serde test for kafka

@lgtm-com
Copy link
Copy Markdown

lgtm-com Bot commented Nov 15, 2019

This pull request introduces 1 alert when merging eb9aa74 into ce4ee42 - view on LGTM.com

new alerts:

  • 1 for Inconsistent equals and hashCode

@lgtm-com
Copy link
Copy Markdown

lgtm-com Bot commented Nov 15, 2019

This pull request introduces 1 alert when merging 7388323 into ce4ee42 - view on LGTM.com

new alerts:

  • 1 for Inconsistent equals and hashCode

}

@Override
public boolean equals(Object o)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you have an override for equals, you'll need to also override hashCode

FYI, the SonarLint intellij plugin will give you a warning if you don't override both.

While typing this, I just noticed LGTM flagged the same issue.

// For backwards compatibility
@Deprecated
@JsonProperty
public SeekableStreamDataSourceMetadata getCurrentCheckPoint()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is just for Jackson, you can make it private and then get rid of the @Deprecated if you want

*/
@Deprecated
@JsonProperty("sequenceName")
public String getBaseSequenceName()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comment as above

Copy link
Copy Markdown
Contributor

@ccaominh ccaominh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Copy link
Copy Markdown
Contributor

@gianm gianm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good, just some comments about the comments.

}

/**
* Returning a dummy value so the objects from older versions still work
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would returning a dummy value make objects from older versions work with the current version? Modifying the serialization code now won't change what older versions do.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is to add the missing property (sequenceName) in newer serialized JSON. So the task fails on older middlemanager since it requires this property, I think it doesn't matter what value is returned here, as long as the property is present since the value is not really used in any production code. But i guess the javadoc can be updated to say it's for rolling updates.

/**
* Returning a dummy value so the objects from older versions still work
* with current version
* TODO : this should be removed in the next release
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think "next release" is right, maybe a few releases down the road. How about: it should be removed when we don't need rolling-update compatibility with version X anymore (please fill in X with the right value).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good, i guess X should be 0.15 or earlier

@gianm
Copy link
Copy Markdown
Contributor

gianm commented Nov 19, 2019

This needs a merge from master to address the CI issue. Could you please do that and adjust the comments?

Copy link
Copy Markdown
Contributor

@gianm gianm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 after CI

@fjy fjy merged commit cf6643e into apache:master Nov 19, 2019
@surekhasaharan surekhasaharan deleted the kafka-rolling-update branch November 19, 2019 21:23
jon-wei pushed a commit to jon-wei/druid that referenced this pull request Nov 26, 2019
…pache#8864)

* add sequenceName and currentCheckPoint for backwards compatibility

* Add serde unit test in kafka

* fix checkstyle

* add hashcode

* update javadoc
jon-wei pushed a commit to jon-wei/druid that referenced this pull request Nov 27, 2019
…pache#8864)

* add sequenceName and currentCheckPoint for backwards compatibility

* Add serde unit test in kafka

* fix checkstyle

* add hashcode

* update javadoc
@jon-wei jon-wei added this to the 0.17.0 milestone Dec 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kafka indexing task can fail during rolling update

5 participants