[BEAM-151] Work towards moving Dataflow pipeline runner to separate maven module #87
Conversation
This allows for moving the Dataflow specific portion of the test to the Dataflow runner maven module.
This allows for moving the Dataflow specific portion of the testing to the Dataflow runner maven module.
This prevents moving the Dataflow runner code to its own separate maven module.
…rface This prevents moving DataflowPipelineOptions to the Dataflow runner maven module.
This prevents moving DataflowPipelineOptions into a Dataflow runner maven module.
This prevents moving DataflowPipelineOptions into a separate maven module.
Removed DatastoreIO dependence on numWorkers from DataflowPipelineWorkerHarnessOptions. This prevented moving DataflowPipelineOptions to a separate maven module.
This prevents moving DataflowPipelineRunner and subclasses to runner specific maven module.
This prevents moving DataflowPipelineOptions to its own runner specifc maven module.
|
R: @dhalperi |
|
Random attempt: |
|
Jenkins, retest this please. |
|
Merged in apache/master to force re-run of tests |
| * Properties that can be set when using Pubsub with the Dataflow SDK. | ||
| */ | ||
| @Description("Options that are used to configure BigQuery. See " | ||
| + "https://cloud.google.com/bigquery/what-is-bigquery for details on BigQuery.") |
|
R: @davorbonaci |
| /** | ||
| * Helpers for cloud communication. | ||
| */ | ||
| public class DataflowTransport { |
There was a problem hiding this comment.
Dataflow or Gcp?
I assume these are used outside of Dataflow Service, e.g., PubSubIO on Flink.
There was a problem hiding this comment.
Oh I see you only moved Dataflow and Clouddebugger. Why only these?
There was a problem hiding this comment.
Yeah, I left all the GCP IO transports in Transport.java and moved out to the two clients which we use only with Dataflow. In the future, cloud debugger could move out to be generic for others as well.
|
No particular comments from me. |
| * {@link DataflowPipelineRunner} specific tests for TextIO Read and Write transforms. | ||
| */ | ||
| @RunWith(JUnit4.class) | ||
| public class DataflowTextIOTest { |
There was a problem hiding this comment.
Remove; duplicate exists elsewhere.
|
LGTM (self-merge). |
|
@lukecwik bumping that you should self-merge this PR when ready. |
apache#85 Move findbugs plugin execution to the process-classes phase
3 participants
Break out tests to Dataflow specific variants to make move of Dataflow pipeline runner to new maven module.
Also, swap usage of DataflowPipelineOptions for other options interfaces which were better suited. For example, creating PubsubOptions.