Skip to content

[Task]: Clean up Java Example PreCommit/PostCommit Dataflow #28561

@Abacn

Description

@Abacn

What needs to happen?

Currently we have multiple Example PreCommit/PostCommit on Dataflow:

  • PreCommit Example V1 (8, 11, 17) - run wordcount
  • PostCommit Example V1 (11, 17) - run wordcount and impersonate wordcount
  • PostCommit Example V2(8, 11, 17) - run full example (except wordcount)
  • PostCommit V1 - run full example (except wordcount)
  • PostCommit V2 - run full example (except wordcount)

There are a few issues

  • No Dataflow Pipeline run on runner v2 at PreCommit - this has caused regression escaped e.g. Remove TableSchema to JSON conversion. #28274
  • Wordcount never runs on runner v2 - either on Example PostCommit or the "big" PostCommit
  • Duplicate test run, e.g. both PostCommit V2 and PostCommit Example V2 runs Java example
  • Example tests are defined in two different files (runner/google-cloud-dataflow-java/build.gradle (for runner v1 tests) and runner/google-cloud-dataflow-java/example/build.gradle (for runner v2 tests))

We should put all example tests to example/build.gradle also

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions