[BEAM-117] Validate display data registration in Dataflow Runner #325
Conversation
|
R: @bjchambers |
swegner
force-pushed
the
displaydata-dataflow
branch
from
df8b28d to
9cfa061
Compare
|
@bjchambers is out-of-office, @lukecwik can you take a look? |
|
/cc @tgroh |
| DisplayData displayData = DisplayData.from(write); | ||
|
|
||
| assertThat(displayData, hasDisplayItem("tableId", "fooTable")); | ||
| // BigtableIO adds user-agent to options; assert only on key and not value. |
|
I've addressed all feedback so far. Please take another look. @tgroh |
| <artifactId>java-sdk-all</artifactId> | ||
| </dependency> | ||
|
|
||
| <dependency> |
There was a problem hiding this comment.
I'm surprised this validation required adding dependencies on avro and datastore. Is there a reason these dependencies appeared now but not before?
There was a problem hiding this comment.
Previously there was no direct dependency on Avro/Datastore in Dataflow runner. This change adds integration tests which directly reference classes in these packages.
|
LGTM after we confirm the new dependencies and get the build passing. |
|
@bjchambers Travis is green, Jenkins failure looks like a flake. |
swegner
force-pushed
the
displaydata-dataflow
branch
from
d9ac1a6 to
6269b8f
Compare
|
@bjchambers squashed. I don't know why Jenkins is still angry, but this is rebased on top of current HEAD. |
|
Backport: GoogleCloudPlatform/DataflowJavaSDK#230 |
Implement getSplitPointsConsumed() in BigtableIO
Co-authored-by: Craig Labenz <craiglabenz@google.com>
4 participants
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
[BEAM-<Jira issue #>] Description of pull requestmvn clean verify. (Even better, enableTravis-CI on your fork and ensure the whole test matrix passes).
<Jira issue #>in the title with the actual Jira issuenumber, if there is one.
Individual Contributor License Agreement.
The DataflowRunner (like other runners) transforms pipelines during construction and sends only primitive transforms to its service. This PR validates that primitive transforms are correctly registering display data for IO transforms.