Skip to content

KAFKA-4923: Add Exaclty-Once Semantics to Streams (testing)#2974

Closed
mjsax wants to merge 1 commit intoapache:trunkfrom
mjsax:kafka-4923-add-eos-to-streams-add-broker-check-and-system-test
Closed

KAFKA-4923: Add Exaclty-Once Semantics to Streams (testing)#2974
mjsax wants to merge 1 commit intoapache:trunkfrom
mjsax:kafka-4923-add-eos-to-streams-add-broker-check-and-system-test

Conversation

@mjsax
Copy link
Copy Markdown
Member

@mjsax mjsax commented May 4, 2017

  • add broker compatibility system tests

 - add broker compatibility system tests
@mjsax
Copy link
Copy Markdown
Member Author

mjsax commented May 4, 2017

Call for review @enothereska @dguy

@mjsax
Copy link
Copy Markdown
Member Author

mjsax commented May 4, 2017

@asfbot
Copy link
Copy Markdown

asfbot commented May 4, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3460/
Test FAILed (JDK 8 and Scala 2.12).

@asfbot
Copy link
Copy Markdown

asfbot commented May 4, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3463/
Test PASSed (JDK 7 and Scala 2.10).

@asfbot
Copy link
Copy Markdown

asfbot commented May 4, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/3469/
Test PASSed (JDK 8 and Scala 2.11).

Comment thread checkstyle/checkstyle.xml
<module name="BooleanExpressionComplexity">
<!-- default is 3 -->
<property name="max" value="4"/>
<property name="max" value="5"/>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you just add a suppression for the file that needs to have 5 booleans? There is little point in having rules if people are just going to keep on changing the values

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also little point in adding an exception for each single file that break the rule IMHO. I just think that 4 (and even 5) is a rather small number anyway. (But this goes back to you other comment -- if we only check a single API key, we could keep 4 here.)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. well the point is to not break the rule! If you really feel you need to break the rule then add a suppression

}

private boolean brokerSupportsTransactions(final ApiVersionsResponse apiVersionsResponse) {
return apiVersionsResponse.apiVersion(ApiKeys.INIT_PRODUCER_ID.id) != null
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually need to check for all of these ApiKeys? i.e., we know that they are all going in as part of the same release, can we not just check for at least that version instead?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think there is a strict need -- I thought it might be "cleaner" to check all. But maybe that overkill. Curious what others think about this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it doesn't harm.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree that it is safer to check all: though it may unlike to happen but we may remove some of the APIs moving forward.

@mjsax
Copy link
Copy Markdown
Member Author

mjsax commented May 4, 2017

@enothereska
Copy link
Copy Markdown
Contributor

What kind of testing is this PR supposed to cover (so I know if it's complete or not)?

@mjsax
Copy link
Copy Markdown
Member Author

mjsax commented May 4, 2017

It adds:

  • 0.11 Streams fails if EOS enabled and brokers are pre-0.11
  • 0.11 Streams work for pre-0.11 brokers if EOS is disabled

The whole system test should cover: https://github.com/apache/kafka/pull/2974/files#diff-7924cd3d80f77980ae421699e631e2ccR29

Copy link
Copy Markdown
Contributor

@dguy dguy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments were only minor. LGTM

@mjsax
Copy link
Copy Markdown
Member Author

mjsax commented May 19, 2017

@guozhangwang Call for review and merging.

@asfbot
Copy link
Copy Markdown

asfbot commented May 19, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4216/
Test PASSed (JDK 7 and Scala 2.11).

}

private boolean brokerSupportsTransactions(final ApiVersionsResponse apiVersionsResponse) {
return apiVersionsResponse.apiVersion(ApiKeys.INIT_PRODUCER_ID.id) != null
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree that it is safer to check all: though it may unlike to happen but we may remove some of the APIs moving forward.

asfgit pushed a commit that referenced this pull request May 22, 2017
…treams

 - add broker compatibility system tests

Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Damian Guy, Eno Thereska, Guozhang Wang

Closes #2974 from mjsax/kafka-4923-add-eos-to-streams-add-broker-check-and-system-test

(cherry picked from commit 495836a)
Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
@guozhangwang
Copy link
Copy Markdown
Contributor

Merged to trunk and 0.11.0.

@asfgit asfgit closed this in 495836a May 22, 2017
@mjsax mjsax deleted the kafka-4923-add-eos-to-streams-add-broker-check-and-system-test branch May 22, 2017 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants