[BEAM-124] Flink and Spark running Examples WordCountIT

[markflyhigh]

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

• Make sure the PR title is formatted like:
  [BEAM-<Jira issue #>] Description of pull request
• Make sure tests pass via mvn clean verify. (Even better, enable
  Travis-CI on your fork and ensure the whole test matrix passes).
• Replace <Jira issue #> in the title with the actual Jira issue
  number, if there is one.
• If this contribution is large, please file an Apache
  Individual Contributor License Agreement.

---

+R: @dhalperi @jasonkuster

This PR is an updated version of an old PR.

Removes dependencies of Spark and Flink runner on Beam java example in order to run WordCountIT with Spark and Flink runner successfully. The following command is used for different runner:

• Spark runner:

mvn clean verify -pl examples/java -DskipITs=false -Dit.test=WordCountIT -DintegrationTestPipelineOptions='[ "--tempRoot=/tmp", "--inputFile=/tmp/kinglear.txt", "--runner=org.apache.beam.runners.spark.SparkRunner" ]'

• Flink runner:

mvn clean verify -pl examples/java -DskipITs=false -Dit.test=WordCountIT -DintegrationTestPipelineOptions='[ "--tempRoot=gs://clouddfe-testing-temp-storage", "--runner=org.apache.beam.runners.flink.FlinkRunner" ]'

• Dataflow test runner:

mvn clean verify -pl examples/java -DskipIT=false -Dit.test=WordCountIT -DintegrationTestPipelineOptions='[ "--tempRoot=gs://clouddfe-testing-temp-storage", "--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]'

[dhalperi]

R: @kennknowles has a lot of existing state here, maybe you can take a look?

[kennknowles]

These versions are managed in the root pom.xml in the pluginManagement section. But in fact it is already configured to ignoreNonCompile, isn't it?

[markflyhigh]

Yes, but actually I remove the failOnWarning setting to make it as default (false). failOnWarning makes the project build failed due to unused spark and flink runner dependency.

[kennknowles]

Yes, that is caused by them being compile dependencies. If you make them runtime it will be fine. Or, if you do reference them in the code, mark them optional.

[kennknowles]

Took a look. At a high level:

• Eliminate the dependency on examples more forcefully.
• This is related and probably conflicts with [BEAM-380] Remove Spark runner dependency on beam-examlpes-java #539.
• Get feedback from @amitsela on what integration tests are most important to keep running continuously, and which are now covered by the RunnableOnService tests.
• Is this actually running in the presubmit?

The Jenkins failure looks like it was just canceled, maybe due to a Jenkins restart.

[amitsela]

I think #539 took care of most of the changes applied to the Spark runner here, so a rebase is in place.
As for RunnableOnService coverage:

• Streaming tests are definitely not covered for obvious reasons.
• IO tests are unique as well.
  I think the rest of the tests (that should keep running continuously) are covered by RunnableOnService. There are also tests such as SerializationTest and SideEffectsTest which I'm not sure are covered by RunnableOnService but also but sure they should be running constantly.
  @kennknowles for the Spark runner in Batch mode, is there anything else missing besides Read.Bound support in order for the runner to execute all Batch tests ? I have this worked in the Spark runner 2 branch and I'm thinking of backporting this into the runner's current version so if that would help, I'll do that.

[markflyhigh]

Thanks @kennknowles @amitsela. Rebased from master. Only change is adding runtime dependencies to example/java to support Flink and Spark runner.

PTAL

[markflyhigh]

+R: @jasonkuster

[jasonkuster]

LGTM

[markflyhigh]

+R: @lukecwik. Can you take a look? Thanks.

[amitsela]

LGTM for Spark dependencies.

[lukecwik]

kinglear.tx -> kinglear.txt

[markflyhigh]

fixed

[markflyhigh]

PTAL

I don't have permission to put test file to gs://dataflow-samples/apache/LICENSE. Need to be done before merge to master.

@jasonkuster @kennknowles

[dhalperi]

If the input file is customizable, then the checksum needs to be customizable as well. Move this to the WordCountITOptions

[markflyhigh]

good idea. In fact, I'll writing another PR which create a FileChecksumMatcher to make this WordCountMatcher more general. I think I can make changes in that PR.