[BEAM-53] Java-only pub/sub source and sink (streaming only)

[mshields822]

First step towards supporting pub/sub i/o in any Java runner.

Disclaimers:

• No integration tests yet.
• No unit tests for the source and sink. Propose we support mocking PubsubGrpcClient.
• Depends on grpc-pubsub-v1 which is about to be renamed.
• Only supports 'application default' credentials, and ignores any GcsOptions flags.
• Not yet wired into the 'deault' PubsubIO implementations.
• Watermark tracking is heuristic and may introduce late data.

But other than that we're ready to go.

[mshields822]

R=Ken
@kennknowles

[mshields822]

Note this has been extensively stress tested by running within a custom Google Dataflow runner.

[dhalperi]

R= @kennknowles

(Copying from #85 (comment), so my scripts work)

[mshields822]

Made PubsubGrpcClient public and refactored a bit so that it API is free of gRPC and protoc dependencies. This means it can be reused, and also will make mocking easier.

[mshields822]

Looking at the ci failure the grpc-pubsub.jar (containing all the pubsub proto-derived classes) is jdk8 only. Advice?

[mshields822]

R: @davorbonaci
for help with ci failure above

[mshields822]

The two pubsub/grpc jars this depends on have been compiled with jdk8 and thus break our jdk7 support requirement. I'm taking this up with the Google team responsible. We can review this but we can't merge until that is sorted out.

[kennknowles]

Just to be super clear - the min/max/sum are meant as examples here, right? It seems this is a generic binning/bucketing map.

[mshields822]

Those are the three functions I needed for watermark tracking and implemented in SimpleFunction. I'm trying to keep the scope as small as possible.

[kennknowles]

This is a very large PR. I am going to need to take another pass to continue to grok.

[kennknowles]

One thing you could do that would lighten the cognitive load would be to separate the source and sink into two PRs.

[mshields822]

PTAL (really need to squash history now)

• Avoid SimpleFunction, move the bucking/moving functions stuff into util
• Split off GrpcClient iface as you suggest.

So now we could proceed as
GrpcClient and friends
PubsubUnboundedSink
PubsubUnboundedSource and friends

[mshields822]

Ok, I'll save this one for just the source, and will send in pub/sub client and sink as sep prs.

[kennknowles]

Noting that the first PR peeled off from this is #120.