Import Spark Runner code#37

Merged

asfgit merged 137 commits intoapache:masterfrom

tomwhite:beam-6-import-spark-runner

Mar 10, 2016

Commits on Mar 10, 2016

Initial commit

Josh Wills
authored and
tomwhite
committed
Dumbest proof of concept possible

Josh Wills
authored and
tomwhite
committed
First bit of work to get this running against the new Dataflow API

Josh Wills
authored and
tomwhite
committed
Update version of dataflow to get new API method access

Josh Wills
authored and
tomwhite
committed
Add support for getters and a Flatten impl

Josh Wills
authored and
tomwhite
committed
Such code. Much features.

Josh Wills
authored and
tomwhite
committed
Adding some more operators: toiterable, seqdo

Josh Wills
authored and
tomwhite
committed
Support for ParDo.BoundMulti

Josh Wills
authored and
tomwhite
committed
Fix bug in deserializing side inputs

Josh Wills
authored and
tomwhite
committed
Add SparkRuntimeContext for handling shared runtime objects

Josh Wills
authored and
tomwhite
committed
First cut at aggregators

Josh Wills
authored and
tomwhite
committed
First minimally working aggregators

Josh Wills
authored and
tomwhite
committed
Updates for 141206 SDK release

Josh Wills
authored and
tomwhite
committed
Dummy impls of windowing-related ProcContext functions

Josh Wills
authored and
tomwhite
committed
Simplify pom.xml

Juliet Hougland
authored and
tomwhite
committed
Add proper coder handling to RDD retrieval

Josh Wills
authored and
tomwhite
committed
Refactor aggregation related classes.

Juliet Hougland
authored and
tomwhite
committed
Add README.md and update project version in pom.xml.

Juliet Hougland
authored and
tomwhite
committed
Adds Javadoc and Tests to project.

Juliet Hougland
authored and
tomwhite
committed
Add apache2 license and cloudera copyright.

Juliet Hougland
authored and
tomwhite
committed
Adds custom checkstyle.

Juliet Hougland
authored and
tomwhite
committed
Factor out tranform translation logic in to its own class.

Juliet Hougland
authored and
tomwhite
committed
Specify and rationalize generic types in State, CoderHelpers to start

srowen
authored and
tomwhite
committed
Add simple word count test.

Juliet Hougland
authored and
tomwhite
committed
Factor out spark pipeline options.

Juliet Hougland
authored and
tomwhite
committed
Miscellaneous inspection changes from IntelliJ

srowen
authored and
tomwhite
committed
Issue #13 : attempt to remove all generics warnings, or handle them explicitly

srowen
authored and
tomwhite
committed
Update and specify POM plugin config; Update Spark to 1.1.1, JUnit to 4.12, Spark 1.2; Add source, javadoc plugins and other info; Fix javadoc errors and a few typos

srowen
authored and
tomwhite
committed
Improve readme to explain current state of the repo, and to encourage outside contributions.

Juliet Hougland
authored and
tomwhite
committed
Update version of dataflow we depend on.

Juliet Hougland
authored and
tomwhite
committed
Fix side input loading and re-enable approxuniq test

jwills
authored and
tomwhite
committed
Add a close() method to EvaluationResult/EvaluationContext which stops the underlying JavaSparkContext (necessary for ensuring Spark EventLogs are flushed before exiting). ------------- Add BUILD r…

dennishuo
authored and
tomwhite
committed
Add tests for CombinePerKey transform and update other tests to use EvaluationResult.close() method to safely clean up SparkContext

Josh Wills
authored and
tomwhite
committed
Rename artifact to spark-dataflow; add release plugin config; add RAT check and some copyright headers; fix one compiler warning

srowen
authored and
tomwhite
committed
Release preparation: needs to begin with a -SNAPSHOT version; Javadoc 8 doesn't like <p/>

srowen
authored and
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.0.1

srowen
authored and
tomwhite
committed
[maven-release-plugin] prepare for next development iteration

srowen
authored and
tomwhite
committed
Add Cloudera repo info

srowen
authored and
tomwhite
committed
Fix formatting of cloudera repo

srowen
authored and
tomwhite
committed
Update README w/some notes on motivation. Fixes #20.

Josh Wills
authored and
tomwhite
committed
Added new test for TransformTranslator and tested TextIO.Read and TextIO.Write transforms

David Alves
authored and
tomwhite
committed
Update to latest version of Dataflow APIs and mimic

Josh Wills
authored and
tomwhite
committed
Add a test for Avro.
tomwhite
committed
Add test that uses DeDupExample from the SDK.
tomwhite
committed
Add test that uses TfIdf from the SDK and switch to use KryoSerializer.
tomwhite
committed
Make it possible to run the SDK word count example with the SparkPipelineRunner.
tomwhite
committed
Change to use SLF4J rather than java.util.logging.
tomwhite
committed
Exclude jul-to-slf4j to avoid loops.
tomwhite
committed
Exclude old Jetty version of servlet API.
tomwhite
committed
Update to dataflow 0.4.150414.
tomwhite
committed
Use DataflowAssert in tests.
tomwhite
committed
Add Travis, Jacoco / Codecov integration. Fix Spark errors due to reused test JVM.

srowen
authored and
tomwhite
committed
Enable checkstyle, remove some rules, comply with checkstyle rules, fix some compiler warnings, fix some other items from IJ inspection

srowen
authored and
tomwhite
committed
Update to Spark 1.3. Closes issue #38

srowen
authored and
tomwhite
committed
Add CONTRIBUTING.md to clarify contribution license; add build flair to README.md

srowen
authored and
tomwhite
committed
Enable RAT during verify phase and add some missing copyright headers

srowen
authored and
tomwhite
committed
Remove unnecessary build properties for mvn exec.
tomwhite
committed
Document how to run on a cluster with spark-submit.
tomwhite
committed
Support withNumShards() and withoutSharding() for TextIO output
tomwhite
committed
Implement dataflow sharding for text output. With this change the direct runner
tomwhite
committed
Fix README to reflect changes from b6e4787. Also fixes a corner case
tomwhite
committed
Shard number replacement should only be for 'S' pattern in the template, not anywhere in the filename.
tomwhite
committed
Use internal coder for side inputs, rather than trying to infer our own
tomwhite
committed
Address review feedback.
tomwhite
committed
Use Coders to convert from object-based RDDs to byte-array RDDS for all
tomwhite
committed
Switch to use Kryo serializer.
tomwhite
committed
Add a comment explaining use of Coders for serialization.
tomwhite
committed
Add HadoopIO for reading from Hadoop InputFormat classes.
tomwhite
committed
Address review feedback.
tomwhite
committed
Remove user-dependent configuration from maven-gpg-plugin.
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.1.0
tomwhite
committed
[maven-release-plugin] prepare for next development iteration
tomwhite
committed
Update README with latest release. Add brief release instructions.
tomwhite
committed
Compute PCollections that are created only for the side effects in their computation.
tomwhite
committed
Wrap failures in pipeline.run() in a RuntimeException.
tomwhite
committed
Implement DoFn.Context.getPipelineOptions().
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.1.1
tomwhite
committed
[maven-release-plugin] prepare for next development iteration
tomwhite
committed
Update README with 0.1.1 minor release.
tomwhite
committed
Allow tests to share a single Spark context.
tomwhite
committed
Update to dataflow 0.4.150602.
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.2.0
tomwhite
committed
[maven-release-plugin] prepare for next development iteration
tomwhite
committed
Update README with 0.2.0 release.
tomwhite
committed
Unwrap cause from SparkException if set.
tomwhite
committed
Specialize CombineGlobally
tomwhite
committed
Fix bug in combinePerKey where accumulator in return value is ignored.
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.2.1
tomwhite
committed
[maven-release-plugin] prepare for next development iteration
tomwhite
committed
Factor out common code from DoFnFunction and MultiDoFnFunction.
tomwhite
committed
Implement Aggregator#getCombineFn.
tomwhite
committed
Implement getAggregatorValues.
tomwhite
committed
More cleanup. View.AsSingleton is already exercised by the TfIdf test.
tomwhite
committed
Set the RDD's name from the PValue's name, to help diagnosis.
tomwhite
committed
Resolve some generics warnings with some fancier footwork

srowen
authored and
tomwhite
committed
Remove some HadoopIO.Read.Bound factory methods and fluent setters; always set key/value at creation

srowen
authored and
tomwhite
committed
Remove some HadoopIO.Read.Bound factory methods and fluent setters; always set key/value at creation

srowen
authored and
tomwhite
committed
Fix checkstyle error

srowen
authored and
tomwhite
committed
Delay converting PCollection values to bytes in case they are only used for views.
tomwhite
committed
Add a system property, dataflow.spark.directBroadcast, to allow pipelines to bypass coders for broadcasts.
tomwhite
committed
Make access of boolean system property clearer. (From Sean Owen.)
tomwhite
committed
Only accumulate outputs from one call to processContext, rather than
tomwhite
committed
Update to dataflow 0.4.150710.
tomwhite
committed
Prevent possible NPE.
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.2.2
tomwhite
committed
[maven-release-plugin] prepare for next development iteration
tomwhite
committed
Update README to latest version (0.2.2).
tomwhite
committed
Fix bug where values written to the output in DoFn#startBundle and DoFn#finishBundle
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.2.3
tomwhite
committed
[maven-release-plugin] prepare for next development iteration
tomwhite
committed
Avoid warning email by not running codecov unless it was configured; update jacoco and shade plugins

srowen
authored and
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.3.0
tomwhite
committed
[maven-release-plugin] prepare for next development iteration
tomwhite
committed
Update README to latest version (0.3.0).
tomwhite
committed
Update to dataflow 0.4.150727.
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.4.0
tomwhite
committed
[maven-release-plugin] prepare for next development iteration
tomwhite
committed
Update README to latest version (0.4.0).
tomwhite
committed
Dataflow goes GA! Update to version 1.0.0.
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.4.1
tomwhite
committed
[maven-release-plugin] prepare for next development iteration
tomwhite
committed
Update README to latest version (0.4.1).
tomwhite
committed
Correct input parameter is --inputFile

amitsela
authored and
tomwhite
committed
Add support for writes with HadoopIO. This allows Hadoop
tomwhite
committed
Add NullWritableCoder and test.
tomwhite
committed
[maven-release-plugin] prepare release spark-dataflow-0.4.2
tomwhite
committed
[maven-release-plugin] prepare for next development iteration
tomwhite
committed
Update README to latest version (0.4.2).
tomwhite
committed
Add tests for Spark 1.4 / 1.5 in Travis

srowen
authored and
tomwhite
committed
Fix a few Coverity inspection results plus more IntelliJ results

srowen
authored and
tomwhite
committed
Propagate user exceptions thrown in DoFns.
tomwhite
committed
The example needs --inputFile, not --input, to designate the input file

Roberto Congiu
authored and
tomwhite
committed
Add spark-streaming support to spark-dataflow

amitsela
authored and
tomwhite
committed
Add support for Flattenning (union) PCollections and test

Sela
authored and
tomwhite
committed
Upgrade to latest SDK version 1.3.0
tomwhite
committed
Try to clean up some build warnings, related to generics, and try to further untangle some generics issues. Update plugins. Fix some minor code issues from inspection.

srowen
authored and
tomwhite
committed
First wave of changes from feedback

srowen
authored and
tomwhite
committed