[SPARK-14073][Streaming][test-maven]Move flume back to Spark #11895

zsxwing · 2016-03-22T18:14:26Z

What changes were proposed in this pull request?

This PR moves flume back to Spark as per the discussion in the dev mail-list.

How was this patch tested?

Existing Jenkins tests.

zsxwing · 2016-03-22T21:14:51Z

retest this please

SparkQA · 2016-03-22T22:27:30Z

Test build #53796 has finished for PR 11895 at commit 19ed34d.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

SparkQA · 2016-03-22T23:14:54Z

Test build #53817 has finished for PR 11895 at commit 19ed34d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

zsxwing · 2016-03-23T01:07:18Z

retest this please

SparkQA · 2016-03-23T03:29:36Z

Test build #53863 has finished for PR 11895 at commit 4352499.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

SparkQA · 2016-03-23T03:42:32Z

Test build #53842 has finished for PR 11895 at commit 4352499.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

zsxwing · 2016-03-23T03:47:11Z

retest this please

SparkQA · 2016-03-23T04:00:50Z

Test build #53880 has finished for PR 11895 at commit 4352499.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

zsxwing · 2016-03-23T04:05:03Z

retest this please

SparkQA · 2016-03-23T04:48:11Z

Test build #53882 has finished for PR 11895 at commit 4352499.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

SparkQA · 2016-03-23T05:03:00Z

Test build #53887 has finished for PR 11895 at commit 4352499.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

zsxwing · 2016-03-23T05:09:26Z

retest this please

SparkQA · 2016-03-23T05:32:40Z

Test build #53896 has finished for PR 11895 at commit 4352499.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

zsxwing · 2016-03-23T05:41:59Z

retest this please

SparkQA · 2016-03-23T06:52:16Z

Test build #53901 has finished for PR 11895 at commit 4352499.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

zsxwing · 2016-03-23T15:51:17Z

retest this please

SparkQA · 2016-03-23T16:53:48Z

Test build #53941 has finished for PR 11895 at commit 4352499.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

zsxwing · 2016-03-23T17:00:55Z

retest this please

SparkQA · 2016-03-23T17:41:08Z

Test build #53947 has finished for PR 11895 at commit 4352499.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

zsxwing · 2016-03-23T17:46:33Z

retest this please

SparkQA · 2016-03-23T18:52:22Z

Test build #53953 has finished for PR 11895 at commit 4352499.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

zsxwing · 2016-03-23T18:56:04Z

retest this please

SparkQA · 2016-03-23T19:48:33Z

Test build #53961 has finished for PR 11895 at commit 4352499.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaFlumeEventCount
- class SparkSink extends AbstractSink with Logging with Configurable
- class FlumeInputDStream[T: ClassTag](
- class SparkFlumeEvent() extends Externalizable
- class FlumeEventServer(receiver: FlumeReceiver) extends AvroSourceProtocol
- class FlumeReceiver(
- class CompressionChannelPipelineFactory extends ChannelPipelineFactory
- class FlumeUtils(object):

SparkQA · 2016-03-23T22:35:09Z

Test build #53970 has finished for PR 11895 at commit 7e18879.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-24T00:52:48Z

Test build #53985 has finished for PR 11895 at commit d1e7435.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-03-24T05:52:01Z

sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java

I can reproduce this behavior by putting Spark to some folder containing "@" locally.

Here are the debug output I tried in Jenkins:

Running test.org.apache.spark.sql.JavaDataFrameSuite raw:file:/home/jenkins/workspace/SparkPullRequestBuilder%402/sql/core/target/scala-2.11/test-classes/text-suite.txt Find: /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/target/scala-2.11/test-classes/text-suite.txt raw:file:/home/jenkins/workspace/SparkPullRequestBuilder%402/sql/core/target/scala-2.11/test-classes/text-suite.txt Find: /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/target/scala-2.11/test-classes/text-suite.txt raw:file:/home/jenkins/workspace/SparkPullRequestBuilder%402/sql/core/target/scala-2.11/test-classes/text-suite.txt Find: /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/target/scala-2.11/test-classes/text-suite.txt raw:file:/home/jenkins/workspace/SparkPullRequestBuilder%402/sql/core/target/scala-2.11/test-classes/text-suite.txt Find: /home/jenkins/workspace/SparkPullRequestBuilder@2/sql/core/target/scala-2.11/test-classes/text-suite.txt Tests run: 17, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 7.628 sec <<< FAILURE! - in test.org.apache.spark.sql.JavaDataFrameSuite testTextLoad(test.org.apache.spark.sql.JavaDataFrameSuite) Time elapsed: 0.356 sec <<< ERROR! org.apache.spark.sql.AnalysisException: Path does not exist: file:/home/jenkins/workspace/SparkPullRequestBuilder%402/sql/core/target/scala-2.11/test-classes/text-suite.txt; at test.org.apache.spark.sql.JavaDataFrameSuite.testTextLoad(JavaDataFrameSuite.java:349) testGenericLoad(test.org.apache.spark.sql.JavaDataFrameSuite) Time elapsed: 0.322 sec <<< ERROR! org.apache.spark.sql.AnalysisException: Path does not exist: file:/home/jenkins/workspace/SparkPullRequestBuilder%402/sql/core/target/scala-2.11/test-classes/text-suite.txt; at test.org.apache.spark.sql.JavaDataFrameSuite.testGenericLoad(JavaDataFrameSuite.java:311) Running test.org.apache.spark.sql.JavaRowSuite Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in test.org.apache.spark.sql.JavaRowSuite Results : Tests in error: JavaDataFrameSuite.testGenericLoad:311 ? Analysis Path does not exist: file:/h... JavaDataFrameSuite.testTextLoad:349 ? Analysis Path does not exist: file:/home...

This is the line returned by Thread.currentThread().getContextClassLoader().getResource("text-suite.txt").toString()

file:/home/jenkins/workspace/SparkPullRequestBuilder%402/sql/core/target/scala-2.11/test-classes/text-suite.txt

SparkQA · 2016-03-24T07:57:29Z

Test build #54007 has finished for PR 11895 at commit 14d859d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-24T10:07:33Z

Test build #54008 has finished for PR 11895 at commit f5d5976.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-03-24T17:03:31Z

retest this please

zsxwing · 2016-03-24T18:52:56Z

Test build #54054 has started for PR 11895 at commit f5d5976.

Hung by DataFrameReaderWriterSuite

retest this please

zsxwing · 2016-03-24T20:20:46Z

retest this please

SparkQA · 2016-03-24T21:17:36Z

Test build #54054 has finished for PR 11895 at commit f5d5976.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-03-24T22:31:59Z

retest this please

SparkQA · 2016-03-25T02:42:29Z

Test build #54106 has finished for PR 11895 at commit f5d5976.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-03-25T03:26:24Z

retest this please

SparkQA · 2016-03-25T07:37:33Z

Test build #54140 has finished for PR 11895 at commit f5d5976.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-03-25T18:09:32Z

LGTM once we fix the maven test issues. Thanks for taking care of this!

zsxwing · 2016-03-25T20:30:45Z

retest this please

SparkQA · 2016-03-25T20:54:24Z

Test build #54208 has finished for PR 11895 at commit f5d5976.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-03-25T20:58:31Z

retest this please

SparkQA · 2016-03-25T23:25:19Z

Test build #54214 has finished for PR 11895 at commit f5d5976.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-03-25T23:45:25Z

Finally passed. Ping @rxin to take a look.

rxin · 2016-03-26T00:36:58Z

Thanks - merging in master.

Move flume back to Spark

4352499

zsxwing changed the title ~~[SPARK-14073][Streaming]Move flume back to Spark~~ [SPARK-14073][Streaming][test-maven]Move flume back to Spark Mar 23, 2016

Merge remote-tracking branch 'origin/master' into move-flume-back

cf52816

zsxwing reviewed Mar 24, 2016
View reviewed changes

Fix JavaDataFrameSuite for Maven

f5d5976

zsxwing mentioned this pull request Mar 25, 2016

[SPARK-14134] [core] [test-maven] Change the package name used for shading classes. #11941

Closed

asfgit closed this in 24587ce Mar 26, 2016

zsxwing deleted the move-flume-back branch March 26, 2016 01:26

[SPARK-14073][Streaming][test-maven]Move flume back to Spark #11895

[SPARK-14073][Streaming][test-maven]Move flume back to Spark #11895

Uh oh!

Conversation

zsxwing commented Mar 22, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

zsxwing commented Mar 22, 2016

Uh oh!

SparkQA commented Mar 22, 2016

Uh oh!

SparkQA commented Mar 22, 2016

Uh oh!

zsxwing commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

zsxwing commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

zsxwing commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

zsxwing commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

zsxwing commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

zsxwing commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

zsxwing commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

zsxwing commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

zsxwing commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 23, 2016

Uh oh!

SparkQA commented Mar 24, 2016

Uh oh!

zsxwing Mar 24, 2016

Choose a reason for hiding this comment

Uh oh!

zsxwing Mar 24, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 24, 2016

Uh oh!

SparkQA commented Mar 24, 2016

Uh oh!

zsxwing commented Mar 24, 2016

Uh oh!

zsxwing commented Mar 24, 2016

Uh oh!

zsxwing commented Mar 24, 2016

Uh oh!

SparkQA commented Mar 24, 2016

Uh oh!

zsxwing commented Mar 24, 2016

Uh oh!

SparkQA commented Mar 25, 2016

Uh oh!