Skip to content

MINOR: Prohibit setting StreamsConfig commit.interval.ms to a negative value#5809

Merged
mjsax merged 4 commits intoapache:trunkfrom
occho:streams-prohibit-negative-commit-interval
Oct 23, 2018
Merged

MINOR: Prohibit setting StreamsConfig commit.interval.ms to a negative value#5809
mjsax merged 4 commits intoapache:trunkfrom
occho:streams-prohibit-negative-commit-interval

Conversation

@occho
Copy link
Copy Markdown
Contributor

@occho occho commented Oct 17, 2018

Prohibit setting StreamsConfig commit.interval.ms to a negative value to avoid possible ambiguity of what it indicates.

So far, setting the property to a negative value can be used to turn off periodic offset-commit by StreamThread. With this change, that will no longer work.
Instead of using a negative value, LONG.MAX_VALUE can be used to achieve the almost same thing.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@occho
Copy link
Copy Markdown
Contributor Author

occho commented Oct 17, 2018

We might be able to remove the check interval >= 0, but I'm not sure whether we can assume their value always come from StreamsConfig. So, I did not update the code.

if (commitTimeMs >= 0 && now - lastCommitMs > commitTimeMs) {

if (flushInterval >= 0 && now >= lastFlush + flushInterval) {

@occho
Copy link
Copy Markdown
Contributor Author

occho commented Oct 17, 2018

@guozhangwang Could you review this?

Copy link
Copy Markdown
Contributor

@guozhangwang guozhangwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. This is from a discussion on the accepted range of this config, cc @bbejeck @vvcephei @mjsax for another review.

Although it changes the public config's range, I think this is really a "bug" that we should fix rather than changing the semantics, so we can skip the KIP process.

@mjsax
Copy link
Copy Markdown
Member

mjsax commented Oct 18, 2018

It's a judgement call. I am fine with the change. We recently updated KafkaStreams#close() similarly.

@occho I think it's save the remove the check >= 0 in both classes. Can you update the code accordingly?

@occho
Copy link
Copy Markdown
Contributor Author

occho commented Oct 18, 2018

@occho I think it's save the remove the check >= 0 in both classes. Can you update the code accordingly?

Sure. Thank you for your advice.
Updated.

Copy link
Copy Markdown
Member

@bbejeck bbejeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me. I think we should have some unit tests demonstrating the fix i.e an Exception results when trying to set commit.interval.ms to a negative value

@occho
Copy link
Copy Markdown
Contributor Author

occho commented Oct 19, 2018

Added a test.

@occho
Copy link
Copy Markdown
Contributor Author

occho commented Oct 19, 2018

Ah, wait, I need more fix.

@occho
Copy link
Copy Markdown
Contributor Author

occho commented Oct 19, 2018

Updated and rebased onto the latest trunk.

Copy link
Copy Markdown
Member

@bbejeck bbejeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @occho just one very minor nit, otherwise LGTM

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe keep this test but pass in 0 instead?

Copy link
Copy Markdown
Contributor Author

@occho occho Oct 21, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding, the test name is wrong and should be like shouldNotFlushWhenFlushIntervalIsNegative.
The code that shouldNotFlushWhenFlushIntervalIsZero tests is about this line:

if (flushInterval >= 0 && now >= lastFlush + flushInterval) {

With the code on trunk, when flushInterval is set to negative, stateMaintainer.flushState() is not called.
But, since this spec is removed, I think it is okay to remove the test.

WDYT?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, thanks for the explanation. Sounds good to me.

@bbejeck
Copy link
Copy Markdown
Member

bbejeck commented Oct 21, 2018

retest this please

Copy link
Copy Markdown
Member

@mjsax mjsax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Thanks for the PR @occho!

@mjsax mjsax merged commit e9bbfde into apache:trunk Oct 23, 2018
@occho
Copy link
Copy Markdown
Contributor Author

occho commented Oct 23, 2018

Thank you for reviewing. :)

@occho occho deleted the streams-prohibit-negative-commit-interval branch November 12, 2018 07:19
pengxiaolong pushed a commit to pengxiaolong/kafka that referenced this pull request Jun 14, 2019
…e value (apache#5809)

Reviewers: Guozhang Wang <guozhang@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants