Skip to content

[BEAM-374] Enable dependency-plugin and failOnWarning at global level. #516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
peihe wants to merge 3 commits into from
Closed

[BEAM-374] Enable dependency-plugin and failOnWarning at global level. #516

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
99408d9
Enable dependency-plugin at global level.
peihe Jun 21, 2016
ff5256b
Update PR to use versions in dependencyManagement
peihe Jun 23, 2016
670e301
address comments
peihe Jun 24, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 0 additions & 13 deletions examples/java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,19 +56,6 @@
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<goals><goal>analyze-only</goal></goals>
<configuration>
<failOnWarning>true</failOnWarning>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
Expand Down
14 changes: 0 additions & 14 deletions examples/java8/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,20 +66,6 @@
</configuration>
</plugin>


<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<goals><goal>analyze-only</goal></goals>
<configuration>
<failOnWarning>true</failOnWarning>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
Expand Down
25 changes: 15 additions & 10 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -860,16 +860,6 @@
</configuration>
</plugin>

<!-- Ignore runtime-only dependencies in analysis -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<configuration>
<ignoreNonCompile>true</ignoreNonCompile>
</configuration>
</plugin>

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
Expand Down Expand Up @@ -918,6 +908,21 @@
</pluginManagement>

<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<executions>
<execution>
<goals><goal>analyze-only</goal></goals>
<configuration>
<!-- Ignore runtime-only dependencies in analysis -->
<ignoreNonCompile>true</ignoreNonCompile>
<failOnWarning>true</failOnWarning>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
Expand Down
13 changes: 0 additions & 13 deletions runners/core-java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,19 +52,6 @@
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<goals><goal>analyze-only</goal></goals>
<configuration>
<failOnWarning>true</failOnWarning>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
Expand Down
13 changes: 0 additions & 13 deletions runners/direct-java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,19 +46,6 @@
<artifactId>maven-compiler-plugin</artifactId>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<goals><goal>analyze-only</goal></goals>
<configuration>
<failOnWarning>true</failOnWarning>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
Expand Down
14 changes: 14 additions & 0 deletions runners/flink/examples/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,20 @@
<artifactId>maven-source-plugin</artifactId>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<goals><goal>analyze-only</goal></goals>
<configuration>
<!-- disable for now until dependencies are cleaned up -->
<failOnWarning>false</failOnWarning>
</configuration>
</execution>
</executions>
</plugin>

<!-- Checkstyle errors for now
<plugin>
<groupId>org.apache.maven.plugins</groupId>
Expand Down
14 changes: 14 additions & 0 deletions runners/flink/runner/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,20 @@
<artifactId>maven-source-plugin</artifactId>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<goals><goal>analyze-only</goal></goals>
<configuration>
<!-- disable for now until dependencies are cleaned up -->
<failOnWarning>false</failOnWarning>
</configuration>
</execution>
</executions>
</plugin>

<!-- Checkstyle errors for now
<plugin>
<groupId>org.apache.maven.plugins</groupId>
Expand Down
13 changes: 0 additions & 13 deletions runners/google-cloud-dataflow-java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,19 +93,6 @@
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<goals><goal>analyze-only</goal></goals>
<configuration>
<failOnWarning>true</failOnWarning>
</configuration>
</execution>
</executions>
</plugin>

<!-- Run CheckStyle pass on transforms, as they are release in
source form. -->
<plugin>
Expand Down
89 changes: 86 additions & 3 deletions runners/spark/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<spark.version>1.6.1</spark.version>
<hadoop.version>2.2.0</hadoop.version>
<kafka.version>0.8.2.1</kafka.version>
</properties>

<profiles>
Expand Down Expand Up @@ -112,10 +114,34 @@
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-network-common_2.10</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.2.1</version>
<version>${kafka.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
Expand All @@ -127,11 +153,68 @@
<artifactId>auto-service</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
Copy link
Member

@amitsela amitsela Jun 23, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

${joda.version}

Copy link
Contributor Author

@peihe peihe Jun 23, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.6</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<exclusions>
<!-- Use Hadoop/Spark's backend logger -->
<!-- Use Hadoop/Spark's backend logger instead of jdk14 for tests -->
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-core-java</artifactId>
<exclusions>
<!-- Use Hadoop/Spark's backend logger instead of jdk14 for tests -->
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
Expand All @@ -142,7 +225,7 @@
<groupId>org.apache.beam</groupId>
<artifactId>beam-examples-java</artifactId>
<exclusions>
<!-- Use Hadoop/Spark's backend logger -->
<!-- Use Hadoop/Spark's backend logger instead of jdk14 for tests -->
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
Expand Down
15 changes: 0 additions & 15 deletions sdks/java/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,21 +135,6 @@
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>analyze-only</goal>
</goals>
<configuration>
<failOnWarning>true</failOnWarning>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
Expand Down
22 changes: 21 additions & 1 deletion sdks/java/io/google-cloud-platform/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,27 @@
<version>${bigtable.version}</version>
</dependency>

<!-- test -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>

<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
</dependency>

<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
</dependency>

<!-- test -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
Expand Down
29 changes: 29 additions & 0 deletions sdks/java/io/hdfs/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,42 @@
<artifactId>beam-sdks-java-core</artifactId>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>

<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.0</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.0</version>
Copy link
Contributor

@dhalperi dhalperi Jun 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extract as property ${hadoop.version}

Copy link
Contributor

@dhalperi dhalperi Jun 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amitsela -- this version conflicts with Spark's version. This might prevent users from using HadoopIO in the Spark runner, no? Suggestions for how to reconcile?

Copy link
Member

@amitsela amitsela Jun 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark can be built against a wide range of hadoops version, as stated here
Two questions:

  1. Why does HadoopIO depends specifically on 2.7.0 ?
  2. Would we want to provide different hadoop.version profiles for Beam as well ?

Copy link
Contributor

@dhalperi dhalperi Jun 24, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. My guess is that it had to depend on some version and that's the one that the dependency plugin suggested? @peihe
  2. Spark's approach seems reasonable to me, if it makes sense to those of you with more Hadoop experience. @nevillelyh? @aljoscha @jbonofre ?

Copy link
Member

@amitsela amitsela Jun 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will emphasise that Spark can be built against different hadoop versions, but the release in maven central is a 2.2.0 build (latest).
If Beam wants to support as many engines as possible (and we want it too), Beam needs to support Hadoop profiles to build against - i.e., the Spark runner will build against 2.2.0 by default.
This means that HadoopIO should probably support several hadoop version implementations.

Copy link
Contributor Author

@peihe peihe Jun 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For question 1:
I don't know why hadoop-client 2.7.0 is picked at the very beginning.
The new hadoop dependencies I added was previously pulled from hadoop-client. So, the dependency plugin suggests 2.7.0.

I agree HadoopIO should work with all runners ideally.
To track this issue, I filed https://issues.apache.org/jira/browse/BEAM-375

The version conflicts already exists, so I propose to address it separate from this PR.
It can be as easy as changing to 2.2.0 if HadoopIO works with it.

Copy link
Contributor

@mxm mxm Jun 27, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Spark's Hadoop version is shaded (classes relocated), just like Flink's Hadoop version. So Beam's Hadoop version shouldn't conflict.

<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.7.0</version>
<scope>provided</scope>
</dependency>

<!-- test dependencies -->
<dependency>
<groupId>org.apache.beam</groupId>
Expand Down
Loading