[BEAM-7] Initial Dataflow code drop#1

Merged

asfgit merged 1575 commits intoapache:masterfrom

francesperry:master

Feb 26, 2016

This pull request is big! We're only showing the most recent 250 commits

Commits on Jan 15, 2016

Fix javadoc @link warnings

peihe
authored and
lukecwik
committed
Adding "DocInclude" metadata comments to the "game" example

devin-donnelly
authored and
lukecwik
committed
Adding worker ID to the upload id logging

laraschmidt
authored and
lukecwik
committed
DataflowAssert: throw when .equals(Object) is called

dhalperi
authored and
lukecwik
committed
Generalize the 'game' example BigQuery write classes

amygdala
authored and
lukecwik
committed

Commits on Jan 16, 2016

Use .jar for staged directory packages

bchambers
authored and
davorbonaci
committed
DefaultProjectFactory: make it use new gcloud properties files

dhalperi
authored and
davorbonaci
committed
Checkstyle: support disabling specific analyzers

dhalperi
authored and
davorbonaci
committed
Add README.md for the "game" example series
davorbonaci
committed

Commits on Jan 19, 2016

Stops holding initializationStateLock while opening the reader

jkff
authored and
lukecwik
committed
Upgrade JaCoCo to 0.7.5

andreich
authored and
lukecwik
committed
Fix typo: wrong table column name

amygdala
authored and
lukecwik
committed
Fix typo in OutputTimeFn

kennknowles
authored and
lukecwik
committed
Implement typeFromId(DatabindContext,String) within CoderUtils
lukecwik
committed

Commits on Jan 22, 2016

Fixes a bug in custom unbounded readers

jkff
authored and
lukecwik
committed

Commits on Jan 24, 2016

Adapt to be able to upload to maven-central
mrunesson
committed

Commits on Jan 25, 2016

Aditional info to simplify release.
mrunesson
committed

Commits on Jan 27, 2016

Commits on Jan 30, 2016

Fix review comments.
mrunesson
committed

Commits on Jan 31, 2016

Merge pull request #104 from mrunesson/maven-central
davorbonaci
committed

Commits on Feb 2, 2016

Version management

lukecwik
authored and
davorbonaci
committed
BigQueryTableRowIterator: elide columns with null values

dhalperi
authored and
davorbonaci
committed
Rollback "BigQueryTableRowIterator: elide columns with null values"

dhalperi
authored and
davorbonaci
committed
Deterministically choose freshest aggregations in pipeline results

robertwb
authored and
davorbonaci
committed
Split streaming status pages into servlets

bchambers
authored and
davorbonaci
committed
Expose dependent realtime watermark via Windmill protos

kennknowles
authored and
davorbonaci
committed
Add ByteStringCoder, a coder for ByteStrings

dhalperi
authored and
davorbonaci
committed
CustomSources: remove dead code

dhalperi
authored and
davorbonaci
committed
Ignore Eclipse project files in root .gitignore

dhalperi
authored and
davorbonaci
committed
BigQueryTableRowIterator: elide columns with null values

dhalperi
authored and
davorbonaci
committed
Split out StatusDataProviders

bchambers
authored and
davorbonaci
committed
Proto changes for multi-worker support.

dpmills
authored and
davorbonaci
committed
Use the standard set of status pages for Batch

bchambers
authored and
davorbonaci
committed
Add FinishedTriggers abstraction with BitSet and Set implementations

kennknowles
authored and
davorbonaci
committed
CustomSources: add logs and normalize log levels

dhalperi
authored and
davorbonaci
committed
SingletonAssert: add a notEqualTo matcher

sammcveety
authored and
davorbonaci
committed
Implement Counter#merge

tgroh
authored and
davorbonaci
committed
Optimize mergeAccumulators by reusing an existing accumulator

peihe
authored and
davorbonaci
committed
Handle when Dataflow service tells us that values are sorted in GBK

lukecwik
authored and
davorbonaci
committed
Continues unifying ReaderIterator and Source.Reader

jkff
authored and
davorbonaci
committed
Fix Coder.Context equality and hashCode

lukecwik
authored and
davorbonaci
committed
Update prebuilt proto libraries for Dataflow to 2016-01-20 version

kennknowles
authored and
davorbonaci
committed
Upgrade Jackson dependency from 2.4.5 to 2.7.0

lukecwik
authored and
davorbonaci
committed
Expose event time and synchronized upstream processing time

kennknowles
authored and
davorbonaci
committed
Update Triggers to new shouldFire() based semantics

kennknowles
authored and
davorbonaci
committed
StreamingWriteFn: check if table exists before creating

dhalperi
authored and
davorbonaci
committed
Let shuffle reader and writer update counters.

mwegiel
authored and
davorbonaci
committed
Provide a reasonable default to Window.Bound#apply

tgroh
authored and
davorbonaci
committed
Allow 'game' examples to append to existing BigQuery tables.

jeffgardner
authored and
davorbonaci
committed
Create StreamingGroupAlsoByWindowsDoFnRunner

peihe
authored and
davorbonaci
committed
DatastoreIO: do not split when QuerySplitter fails

dhalperi
authored and
davorbonaci
committed
BigQueryTableInserter: retry rateLimitExceeded API calls

dhalperi
authored and
davorbonaci
committed
AvroCoder: more efficient use of Avro APIs

dhalperi
authored and
davorbonaci
committed
Modify GameStats window definitions for blog post

amygdala
authored and
davorbonaci
committed
Add new property names

lukecwik
authored and
davorbonaci
committed
Add increment support with positive infinity

lukecwik
authored and
davorbonaci
committed
BigQueryTableRowIterator: handle query failures

dhalperi
authored and
davorbonaci
committed
Add CounterSet#merge(CounterSet)

tgroh
authored and
davorbonaci
committed
Allow unflattened results from a BigQuery query-based export

sammcveety
authored and
davorbonaci
committed
Clean up GameStats session branch for blog post

amygdala
authored and
davorbonaci
committed
Account for per-window overhead in WindmillStateCache

dpmills
authored and
davorbonaci
committed
Add support to configure the shuffle client library with a property.

tudorm
authored and
davorbonaci
committed
Create worker maven module

lukecwik
authored and
davorbonaci
committed
Add explicit record type for Ism files

lukecwik
authored and
davorbonaci
committed
Upgrade protobuf runtime to version 3.0.0-beta-1

dhalperi
authored and
davorbonaci
committed
Refactor LateDataDroppingDoFnRunner

tgroh
authored and
davorbonaci
committed
BigQueryIO: remove DirectRunner dependence on Dataflow native reader

dhalperi
authored and
davorbonaci
committed
runners.CountingSource: rename to TestCountingSource

dhalperi
authored and
davorbonaci
committed
Validate DoFn.createAggregate is not called during pipeline processing

swegner
authored and
davorbonaci
committed
DataflowPipelineRunner: retry source splitting when too many bundles

dhalperi
authored and
davorbonaci
committed
DataflowWorker: elide some for loops for unused log levels

dhalperi
authored and
davorbonaci
committed
BigQueryReader: simplify and remove some redundant code

dhalperi
authored and
davorbonaci
committed
Create internal-only classifier for Dataflow SDK

lukecwik
authored and
davorbonaci
committed

Commits on Feb 4, 2016

Change BQ mode to append instead of truncate

ravwojdyla
authored and
Rafal Wojdyla
committed

Commits on Feb 10, 2016

Commits on Feb 26, 2016

Move Dataflow worker mains to worker maven module

lukecwik
authored and
davorbonaci
committed
Removes dependency on LegacyReaderIterator from more readers

jkff
authored and
davorbonaci
committed
Add support to export BigQuery files in Avro format

sammcveety
authored and
davorbonaci
committed
Move some status pages to worker module

lukecwik
authored and
davorbonaci
committed
CombineWithContext support in batch PartialGroupByKeyOperation

peihe
authored and
davorbonaci
committed
Fix TimeTrigger#isCompatible

kennknowles
authored and
davorbonaci
committed
Move StructuralByteArray to the coder package

peihe
authored and
davorbonaci
committed
AvroUtils: default missing field schema mode to NULLABLE, and test

dhalperi
authored and
davorbonaci
committed
Add ByteCount for BigQueryIO streaming write

tucosh
authored and
davorbonaci
committed
Default to exporting BigQuery files in Avro format

sammcveety
authored and
davorbonaci
committed
Add InProcessPipelineRunner, Evaluation Interfaces

tgroh
authored and
davorbonaci
committed
Upgrade maven shade plugin to avoid infinite loop

lukecwik
authored and
davorbonaci
committed
Add InMemoryWatermarkManager

tgroh
authored and
davorbonaci
committed
Have worker maven module depend on unshaded/unrelocated test jar

lukecwik
authored and
davorbonaci
committed
Separate CustomSources into worker and non-worker pieces

kennknowles
authored and
davorbonaci
committed
Most methods in com.google.common will soon have @CheckReturnValue applied to them. Adding the annotation makes it a compile time error to ignore the return value from the method. This CL fixes the…

kluever
authored and
davorbonaci
committed
Move ReaderFactory.Registry to ReaderRegistry

kennknowles
authored and
davorbonaci
committed
Add Ism file format version 2

lukecwik
authored and
davorbonaci
committed
Always explicitly pass ReaderRegistry to MapTaskExecutorFactory

kennknowles
authored and
davorbonaci
committed
Move worker tests to worker maven module

lukecwik
authored and
davorbonaci
committed
CustomSourcesTest: use a smaller byte limit to reduce test runtime

dhalperi
authored and
davorbonaci
committed
Explicitly pass ReaderFactory from worker classes to SideInputUtils

kennknowles
authored and
davorbonaci
committed
Move MapTaskExecutorFactory and system DoFns and their factories

lukecwik
authored and
davorbonaci
committed
Move several operations and dependencies to worker module

lukecwik
authored and
davorbonaci
committed
Split InMemoryBundle into Read/Write only interfaces

tgroh
authored and
davorbonaci
committed
ApiSurfaceTest: whitelist com.google.auth package

dhalperi
authored and
davorbonaci
committed
Make SinkFactory a real class; remove some reflection

kennknowles
authored and
davorbonaci
committed
Marks getAllowedTimestampSkew as deprecated.

robertwb
authored and
davorbonaci
committed
Add the transforms required to create an Ism side input

lukecwik
authored and
davorbonaci
committed
Don't wrap system DoFn exceptions.

swegner
authored and
davorbonaci
committed
Move BatchModeExecutionContext and dependents to worker module

kennknowles
authored and
davorbonaci
committed
Handle SynchronizedProcessingTime in InMemoryWatermarkManager

tgroh
authored and
davorbonaci
committed
Move streaming worker code into worker module

kennknowles
authored and
davorbonaci
committed
Move Windmill-specific code and dependents to worker module

kennknowles
authored and
davorbonaci
committed
Move Reader classes and dependents to worker module

kennknowles
authored and
davorbonaci
committed
Move over IO sink factories and dependants

lukecwik
authored and
davorbonaci
committed
Render the /threadz page using plain text.

tudorm
authored and
davorbonaci
committed
Move remainder of status servlets

lukecwik
authored and
davorbonaci
committed
Decouple StateContext from ReduceFn

kennknowles
authored and
davorbonaci
committed
Allows specifying if an OffsetBasedSource supports dynamic splitting

jkff
authored and
davorbonaci
committed
Create a top level GroupingTable interface and its factory class

peihe
authored and
davorbonaci
committed
Experimental Cloud Bigtable sink

dhalperi
authored and
davorbonaci
committed
Tighten BigtableIO's API surface exposure

dhalperi
authored and
davorbonaci
committed
Separate Timers interface from ReduceFn

kennknowles
authored and
davorbonaci
committed
Merge TimeTrigger into its only subclass

kennknowles
authored and
davorbonaci
committed
ApiSurface: add debug messages

dhalperi
authored and
davorbonaci
committed
Migrate AvroIO to internally use Read.from(AvroSource)

lukecwik
authored and
davorbonaci
committed
Upgrade to SLF4J 1.7.14 so that we can use slf4j-api MDC

lukecwik
authored and
davorbonaci
committed
Eagerly merge all window state

markshields
authored and
davorbonaci
committed
Add the Ism side input reader

lukecwik
authored and
davorbonaci
committed
Move over various worker classes to worker maven module

lukecwik
authored and
davorbonaci
committed
Move over several shuffle classes to the worker maven module

lukecwik
authored and
davorbonaci
committed
Add a DoFnRunner for StreamingGroupAlsoByWindows with side inputs

peihe
authored and
davorbonaci
committed
Handle Timers in InMemoryWatermarkManager

tgroh
authored and
davorbonaci
committed
Move over worker logging classes to worker maven module

lukecwik
authored and
davorbonaci
committed
Move worker implementation classes of aggregators to worker maven module

lukecwik
authored and
davorbonaci
committed
Remove Base64Utils which is dead code

lukecwik
authored and
davorbonaci
committed
Move over several Dataflow worker specific classes to the worker maven module

lukecwik
authored and
davorbonaci
committed
ByteKey: a key represented byte[]

dhalperi
authored and
davorbonaci
committed
Remove duplicate definition of MetricUpdate utilities

lukecwik
authored and
davorbonaci
committed
Enable new Ism side input format by default for batch Dataflow jobs

lukecwik
authored and
davorbonaci
committed
Use shaded guava from within worker maven module

lukecwik
authored and
davorbonaci
committed
ByteKey: add a static instance EMPTY for the empty key

dhalperi
authored and
davorbonaci
committed
Move miscellaneous worker classes to worker maven module

lukecwik
authored and
davorbonaci
committed
Allow StreamingDataflowWorker to run against a remote gRPC windmill

dpmills
authored and
davorbonaci
committed
Move CustomSources logic to live in more appropriate places

lukecwik
authored and
davorbonaci
committed
Add filename-based compression selection to CompressedSource

kennknowles
authored and
davorbonaci
committed
Make internal state APIs explicitly keyed

kennknowles
authored and
davorbonaci
committed
Add KeyedCombiningValueStateTag containing a KeyedCombineFn

kennknowles
authored and
davorbonaci
committed
Explicitly select GroupAlsoByWindow in default GBK

tgroh
authored and
davorbonaci
committed
Add ByteKeyRange and ByteKeyRangeTracker

dhalperi
authored and
davorbonaci
committed
Make some ReduceFnRunner implementation details private

kennknowles
authored and
davorbonaci
committed
Treat ReduceFn and Trigger exceptions as system exceptions

kennknowles
authored and
davorbonaci
committed
Decouple NonEmptyPanes from ReduceFn somewhat

kennknowles
authored and
davorbonaci
committed
Decouple TriggerRunner from ReduceFn

kennknowles
authored and
davorbonaci
committed
Support reading uncompressed files when gzip is expected

kennknowles
authored and
davorbonaci
committed
Explicitly provide StateInternals to ReduceFnRunner

kennknowles
authored and
davorbonaci
committed
Fix typo in TimerInternals comment.

charlesccychen
authored and
davorbonaci
committed
Fix propagation of streaming keys

kennknowles
authored and
davorbonaci
committed
Do not assume channel is nonempty in TextIO gzip detection

kennknowles
authored and
davorbonaci
committed
Make WatermarkHold package-private

kennknowles
authored and
davorbonaci
committed
Minor revision of ReduceFnRunner docs

kennknowles
authored and
davorbonaci
committed
Minor revision of ActiveWindowSet javadoc

kennknowles
authored and
davorbonaci
committed
Make empty flatten work in streaming

dpmills
authored and
davorbonaci
committed
Minor revision of TriggerRunner javadoc

kennknowles
authored and
davorbonaci
committed
Improve IntraBundleParallelizationTest

tgroh
authored and
davorbonaci
committed
Move specialized GroupAlsoByWindow implementations

tgroh
authored and
davorbonaci
committed
Declare SideInputs on Combine.GroupedValues

tgroh
authored and
davorbonaci
committed
Update Dataflow API version to v1b3-rev19-1.21.0

tgroh
authored and
davorbonaci
committed
Add InProcess Clock abstraction, nanoTime implementation

tgroh
authored and
davorbonaci
committed
Improve TypeDescriptor inference of SimpleFunction

bchambers
authored and
davorbonaci
committed
Fix AfterSynchronizedProcessingTime.java

dpmills
authored and
davorbonaci
committed
Remove debug statement from test

lukecwik
authored and
davorbonaci
committed
Make sure every hold is accompanied by a proximate timer

markshields
authored and
davorbonaci
committed
Add support for splitting a compressed source for uncompressed files

lukecwik
authored and
davorbonaci
committed
Add --worker_harness_container_image PipelineOption
davorbonaci
committed
Use new keyed state API to remove ReduceFn.Factory

kennknowles
authored and
davorbonaci
committed
Implement CopyOnAccessInMemoryStateInternals

tgroh
authored and
davorbonaci
committed
Handle null input bundles in InMemoryWatermarkManager

tgroh
authored and
davorbonaci
committed
Remove unneccessary type from GroupAlsoByWindowViaWindowSet

tgroh
authored and
davorbonaci
committed
Never drop late data in Reshuffle

dpmills
authored and
davorbonaci
committed
BigtableIO: add a bounded source for Google Cloud Bigtable

dhalperi
authored and
davorbonaci
committed
ApproximateUnique[Test]: improve efficiency and cleanup test

dhalperi
authored and
davorbonaci
committed
Refactor to use try-with-resources

swegner
authored and
davorbonaci
committed
StateSamplerTest: fewer repeats per reused counter name

dhalperi
authored and
davorbonaci
committed
Add InProcessSideInputContainer

tgroh
authored and
davorbonaci
committed
Update FileIOChannelFactory.expand() to only return file resources

lukecwik
authored and
davorbonaci
committed
Add validation for single objects in GcsUtil.expand

lukecwik
authored and
davorbonaci
committed
Rename StateContext to StateAccessor

peihe
authored and
davorbonaci
committed
ByteKeyRange: parameterize expensive combinatorial tests

dhalperi
authored and
davorbonaci
committed
Migrate TextIO.Read to be a custom source

lukecwik
authored and
davorbonaci
committed
Make earliest Watermark State available in CopyOnAccessState

tgroh
authored and
davorbonaci
committed
Add ForwardingPTransform

tgroh
authored and
davorbonaci
committed
Move test for post-GBK timestamp ordering to worker module

kennknowles
authored and
davorbonaci
committed
Move DataflowWorkerLoggingMDC to the worker module

kennknowles
authored and
davorbonaci
committed
Rolling back since this breaks tests

lukecwik
authored and
davorbonaci
committed
Clear empty CopyOnAccessInMemoryStateInternals

tgroh
authored and
davorbonaci
committed
Add Timers and State to InProcessTransformResult

tgroh
authored and
davorbonaci
committed
Migrate TextIO.Read to be a custom source

lukecwik
authored and
davorbonaci
committed
Remove InProcessPipelineRunner evaluator package

tgroh
authored and
davorbonaci
committed
Proto2Coder: fix static modifier check

dhalperi
authored and
davorbonaci
committed
Proto2Coder: minor cleanup

dhalperi
authored and
davorbonaci
committed
Add InProcessTimerInternals

tgroh
authored and
davorbonaci
committed
Move VerifyDynamicWorkRebalancing and dependents to worker module

kennknowles
authored and
davorbonaci
committed
Allow TransformEvaluatorFactory#forApplication to throw

tgroh
authored and
davorbonaci
committed
Add TypeDescriptors to Primitive PTransforms in Tests

tgroh
authored and
davorbonaci
committed
Remove native Dataflow text reader

lukecwik
authored and
davorbonaci
committed
Encode elements in InProcessCreate

tgroh
authored and
davorbonaci
committed
Expose base output file name on FileBasedSink

lukecwik
authored and
davorbonaci
committed
Move over Google Cloud Dataflow worker utilities to worker module

lukecwik
authored and
davorbonaci
committed
Add new test proto messages that use map fields

dhalperi
authored and
davorbonaci
committed
Migrate AvroIO.Write to a custom sink

lukecwik
authored and
davorbonaci
committed
Move Google Cloud Dataflow worker utilities to worker module

lukecwik
authored and
davorbonaci
committed
Port state to new, future-free API

kennknowles
authored and
davorbonaci
committed
Move some worker-and-example-only dependencies out of sdk

kennknowles
authored and
davorbonaci
committed
Change visibility of FileBasedSource subclass methods and fix return types for existing subclasses.

lukecwik
authored and
davorbonaci
committed
Make TestPipeline slightly less DataflowPipelineRunner-centric

kennknowles
authored and
davorbonaci
committed
Add used-but-undeclared dependency on google-http-client

kennknowles
authored and
davorbonaci
committed
Migrate TextIO.Write to a custom sink

lukecwik
authored and
davorbonaci
committed
Move Google Cloud Dataflow native sinks to worker module

lukecwik
authored and
davorbonaci
committed
Reverts "Move Google Cloud Dataflow native sinks to worker module"

sammcveety
authored and
davorbonaci
committed
Add the slf4j-jdk bridge to the integration tests

bchambers
authored and
davorbonaci
committed
Revert "Migrate TextIO.Write to a custom sink"

sammcveety
authored and
davorbonaci
committed
Add KeyedWorkItemCoder

tgroh
authored and
davorbonaci
committed
Revert "Migrate AvroIO.Write to a custom sink"

sammcveety
authored and
davorbonaci
committed
Resubmit "Migrate AvroIO.Write to a custom sink"

dhalperi
authored and
davorbonaci
committed
Add GroupByKey InProcess override

tgroh
authored and
davorbonaci
committed
Proto2Coder: recompute the extension registry when mutated

dhalperi
authored and
davorbonaci
committed
Support CombineFnWithContext in GroupAlsoByWindows

peihe
authored and
davorbonaci
committed
Add InProcess Override for CreatePCollectionView

tgroh
authored and
davorbonaci
committed
Update maven-dependency-plugin to latest

kennknowles
authored and
davorbonaci
committed
ProtoCoder: a Coder for Protocol Buffers Messages

dhalperi
authored and
davorbonaci
committed
Add used-but-undeclared findbugs JSR305 dependencies

kennknowles
authored and
davorbonaci
committed
Handle PCollectionList.empty() in FlattenEvaluatorFactory

tgroh
authored and
davorbonaci
committed
Set worker harness container image to INVALID until next release

herohde
authored and
davorbonaci
committed
Handle Undeclared Side Outputs in ParDoInProcessEvaluator

tgroh
authored and
davorbonaci
committed
Rollback revert "Migrate TextIO.Write to a custom sink"

lukecwik
authored and
davorbonaci
committed
Update Timers and State in the InProcess ParDoEvaluator

tgroh
authored and
davorbonaci
committed
Rollback reverts "Move Google Cloud Dataflow native sinks to worker module"

lukecwik
authored and
davorbonaci
committed
Use a static variable for CoderCalled in WriteTest

tgroh
authored and
davorbonaci
committed
Honor user requested shard limits for AvroIO.Write on DirectPipelineRunner

lukecwik
authored and
davorbonaci
committed
Adjust dependencies to avoid pulling in unneeded stax-api

kennknowles
authored and
davorbonaci
committed
Handle multiple requests in InProcess Read Primitives

tgroh
authored and
davorbonaci
committed
Finish Flattenning InProcess package

tgroh
authored and
davorbonaci
committed
Ensure a TypedPValue has a Coder on finishSpecifying

tgroh
authored and
davorbonaci
committed
Switch to the start state when lazily initializing

bchambers
authored and
davorbonaci
committed
Fix SDK deps and enable strict enforcement

kennknowles
authored and
davorbonaci
committed
Fix worker dependencies and turn on strict checking

kennknowles
authored and
davorbonaci
committed
Update worker harness container image
davorbonaci
committed
Revert "Add a first README.md file (at least to trigger the github mirroring)"
francesperry
committed
Dataflow code drop!
francesperry
committed
Update README for initial code drop.
francesperry
committed
Remove Google-specific contribution rules
francesperry
committed