MINOR: Add verification step for Streams archetype to Jenkins build by mjsax · Pull Request #6431 · apache/kafka

mjsax · 2019-03-11T23:44:32Z

Updates ./jenkins.sh to build stream archetype and install it in local maven cache. Afterward, archetype is used to create a new maven project and maven project is compiled for verification.

mjsax · 2019-03-12T05:02:58Z

Retest this please.

mjsax · 2019-03-14T01:34:39Z

Call for review @guozhangwang @bbejeck @vvcephei @ableegoldman (\cc @ijuma)

mjsax · 2019-03-14T01:35:20Z

This fix should actually be provided by #6194 -- I included it here to let the build pass.

mjsax · 2019-03-14T01:36:49Z

The created archetype, only works for Java8 -- we would need to create a different archetype for Java11.

What we could do, is to include the Java11 setting as comments in the archetype (thoughts)?

We could do that, or we can just publish another archetype for use with Java 11+. Then, when we drop support for Java 8, it's as simple as just deleting the Java 8 archetype.

Not sure how we could have two archetypes at the same time? I guess we would need to encode the Java version in the archetype name?

-DarchetypeGroupId=org.apache.kafka -DarchetypeArtifactId=streams-quickstart-java <--- change this to `streams-quickstart-java-8` and `streams-quickstart-java-11` ? -DarchetypeVersion=$version

Yep, that would be the way to do it.

@guozhangwang Can/should we do this in this PR? Or would this need a KIP? Or just create a JIRA for a follow up PR?

We can create a JIRA ticket for a follow-up PR (this would need a KIP, yes).

Created https://issues.apache.org/jira/browse/KAFKA-8116

mjsax · 2019-03-14T06:45:44Z

Java8 failed as expected:

21:35:30 [ERROR] COMPILATION ERROR : 
21:35:30 [INFO] -------------------------------------------------------------
21:35:30 [ERROR] /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11@2/streams/quickstart/test-streams-archetype/streams.examples/src/main/java/myapps/LineSplit.java:[48] 
21:35:30 	.flatMapValues(value -> Arrays.asList(value.split("\\W+")))
21:35:30 	                                            ^^^^^
21:35:30 The method split(String) is undefined for the type Object
21:35:30 1 problem (1 error)

vvcephei

Hey @mjsax , thanks for taking care of this!

I've responded to your question with my $0.02.

vvcephei · 2019-03-14T16:54:18Z

We could do that, or we can just publish another archetype for use with Java 11+. Then, when we drop support for Java 8, it's as simple as just deleting the Java 8 archetype.

guozhangwang

LGTM. Please feel free to merge after jenkins.

guozhangwang · 2019-03-15T01:36:45Z

nit: one more indent.

guozhangwang · 2019-03-15T01:37:49Z

We can create a JIRA ticket for a follow-up PR (this would need a KIP, yes).

Update docs

mjsax · 2019-03-16T02:59:03Z

Rebased to get the fix from #6194.

mjsax · 2019-03-16T22:40:59Z

Java11 failed with known flaky tests.

Java8 failed with (https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/20398/console):

22:29:21 org.apache.kafka.streams.KafkaStreamsTest.statefulTopologyShouldCreateStateDirectory failed, log available in /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/streams/build/reports/testOutput/org.apache.kafka.streams.KafkaStreamsTest.statefulTopologyShouldCreateStateDirectory.test.stdout
22:29:21 
22:29:21 org.apache.kafka.streams.KafkaStreamsTest > statefulTopologyShouldCreateStateDirectory FAILED
22:29:21     java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /tmp/kafka-9Bd3Q/appId/1_0/rocksdb/statefulTopologyShouldCreateStateDirectory-counts/000007.dbtmp
22:29:21         at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:88)
22:29:21         at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
22:29:21         at java.util.Iterator.forEachRemaining(Iterator.java:115)
22:29:21         at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
22:29:21         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
22:29:21         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
22:29:21         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
22:29:21         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
22:29:21         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
22:29:21         at org.apache.kafka.streams.KafkaStreamsTest.startStreamsAndCheckDirExists(KafkaStreamsTest.java:744)
22:29:21         at org.apache.kafka.streams.KafkaStreamsTest.statefulTopologyShouldCreateStateDirectory(KafkaStreamsTest.java:661)
22:29:21 
22:29:21         Caused by:
22:29:21         java.nio.file.NoSuchFileException: /tmp/kafka-9Bd3Q/appId/1_0/rocksdb/statefulTopologyShouldCreateStateDirectory-counts/000007.dbtmp
22:29:21             at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
22:29:21             at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
22:29:21             at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
22:29:21             at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
22:29:21             at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
22:29:21             at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
22:29:21             at java.nio.file.Files.readAttributes(Files.java:1737)
22:29:21             at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
22:29:21             at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
22:29:21             at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
22:29:21             at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:84)
22:29:21             ... 10 more

Is this an environmental issue or indicates a flaky test?

Retest this please.

mjsax · 2019-03-18T18:00:04Z

Java 11 failed. Created ticket.
Java 8 passed.

Retest this please.

bbejeck

LGTM

bbejeck · 2019-03-21T19:51:36Z

Merged #6431 into trunk.

* apache/trunk: (23 commits) KAFKA-7986: Distinguish logging from different ZooKeeperClient instances (apache#6493) KAFKA-8102: Add an interval-based Trogdor transaction generator (apache#6444) MINOR: Fix misspelling in protocol documentation KAFKA-8150: Fix bugs in handling null arrays in generated RPC code (apache#6489) KAFKA-8014: Extend Connect integration tests to add and remove workers dynamically (apache#6342) MINOR: Remove line for testing repartition topic name (apache#6488) MINOR: add MacOS requirement to Streams docs MINOR: fix message protocol help text for ElectPreferredLeadersResult (apache#6479) MINOR: list-topics should not require topic param MINOR: Clean up ThreadCacheTest (apache#6485) MINOR: Avoid unnecessary collection copy in MetadataCache (apache#6397) KAFKA-8142: Fix NPE for nulls in Headers (apache#6484) KAFKA-7243: Add unit integration tests to validate metrics in Kafka Streams (apache#6080) MINOR: Add verification step for Streams archetype to Jenkins build (apache#6431) KAFKA-7819: Improve RoundTripWorker (apache#6187) KAFKA-7989: RequestQuotaTest should wait for quota config change before running tests (apache#6482) KAFKA-8098: Fix Flaky Test testConsumerGroups KAFKA-6958: Add new NamedOperation interface to enforce consistency in naming operations (apache#6409) MINOR: capture result timestamps in Kafka Streams DSL tests (apache#6447) MINOR: updated names for deprecated streams constants (apache#6466) ...

…pache#6431) Updates ./jenkins.sh to build stream archetype and install it in local maven cache. Afterward, archetype is used to create a new maven project and maven project is compiled for verification. Reviewers: Guozhang Wang <wangguoz@gmail.com>, John Roesler <john@confluent.io>, Bill Bejeck <bbejeck@gmail.com>

mjsax added the streams label Mar 11, 2019

mjsax force-pushed the add-quickstart-test branch 3 times, most recently from 0d080ae to 82cd5ab Compare March 13, 2019 22:02

mjsax commented Mar 14, 2019

View reviewed changes

vvcephei reviewed Mar 14, 2019

View reviewed changes

guozhangwang approved these changes Mar 15, 2019

View reviewed changes

mjsax added 4 commits March 15, 2019 19:58

MINOR: Add verification step for Streams archetype to Jenkins build

fe76355

Remove archetype fix

e3fa62e

Update docs

Fix typos

7f21b28

Github comments

7fc213e

mjsax force-pushed the add-quickstart-test branch from 3fd34ea to 7fc213e Compare March 16, 2019 02:58

bbejeck approved these changes Mar 21, 2019

View reviewed changes

bbejeck merged commit 4cae452 into apache:trunk Mar 21, 2019

mjsax deleted the add-quickstart-test branch March 22, 2019 00:59

Conversation

mjsax commented Mar 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjsax commented Mar 12, 2019

Uh oh!

mjsax commented Mar 14, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjsax commented Mar 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vvcephei left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guozhangwang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjsax commented Mar 16, 2019

Uh oh!

mjsax commented Mar 16, 2019

Uh oh!

mjsax commented Mar 18, 2019

Uh oh!

bbejeck left a comment

Choose a reason for hiding this comment

Uh oh!

bbejeck commented Mar 21, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mjsax commented Mar 11, 2019 •

edited

Loading

mjsax commented Mar 14, 2019 •

edited

Loading