Automatically enable Runner v2 for pipelines that use cross-language transforms.

[chamikaramj]

…transforms

This fixes #21787

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Mention the appropriate issue in your description (for example: "addresses [BEAM-121] Add DisplayData for IO transforms #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment "fixes #" instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[chamikaramj]

R: @robertwb

[chamikaramj]

Run XVR_Dataflow PostCommit

[chamikaramj]

Dataflow CrossLanguageValidatesRunner PythonUsingJava Tests

[chamikaramj]

Dataflow CrossLanguageValidatesRunner JavaUsingPython Tests

[robertwb]

Do we have to add portable_job_submission as well, or is that not automatic for all Java pipelines?

[chamikaramj]

That's automatic for all Java UW pipelines:

beam/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java

Line 1055 in b8e2e85

experiments.add("use_portable_job_submission");

[robertwb]

Note that Jenkis is currently down, see https://issues.apache.org/jira/browse/INFRA-23358

[codecov]

Codecov Report

Merging #21788 (b8e2e85) into master (f131b2d) will increase coverage by 0.14%.
The diff coverage is 84.39%.

❗ Current head b8e2e85 differs from pull request most recent head efbce54. Consider uploading reports for the commit efbce54 to get more accurate results

@@            Coverage Diff             @@
##           master   #21788      +/-   ##
==========================================
+ Coverage   74.00%   74.15%   +0.14%     
==========================================
  Files         698      698              
  Lines       92224    92411     +187     
==========================================
+ Hits        68253    68530     +277     
+ Misses      22720    22630      -90     
  Partials     1251     1251              

Flag	Coverage Δ
go	50.93% <ø> (ø)
python	83.77% <84.39%> (+0.18%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...thon/apache_beam/ml/inference/pytorch_inference.py	0.00% <0.00%> (ø)
sdks/python/apache_beam/utils/counters.py	85.39% <66.66%> (-1.36%)	⬇️
...on/apache_beam/runners/direct/sdf_direct_runner.py	35.53% <75.00%> (ø)
...eam/transforms/py_dataflow_distribution_counter.py	91.42% <75.00%> (-4.87%)	⬇️
sdks/python/apache_beam/transforms/cy_combiners.py	62.90% <76.00%> (+4.50%)	⬆️
sdks/python/apache_beam/runners/common.py	88.68% <83.33%> (+0.74%)	⬆️
sdks/python/apache_beam/transforms/core.py	92.38% <90.47%> (+0.23%)	⬆️
sdks/python/apache_beam/typehints/batch.py	87.09% <94.11%> (+0.86%)	⬆️
sdks/python/apache_beam/ml/inference/base.py	95.06% <96.66%> (+1.36%)	⬆️
...thon/apache_beam/ml/inference/sklearn_inference.py	93.33% <97.05%> (+0.83%)	⬆️
... and 27 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f131b2d...efbce54. Read the comment docs.

[chamikaramj]

Retest this please

[chamikaramj]

Run Java PreCommit

[chamikaramj]

Dataflow CrossLanguageValidatesRunner PythonUsingJava Tests

[chamikaramj]

Dataflow CrossLanguageValidatesRunner JavaUsingPython Tests

[chamikaramj]

Run Java PreCommit

[chamikaramj]

Run Java PostCommit

[chamikaramj]

Run Java PreCommit

[chamikaramj]

(failures are unrelated)

[chamikaramj]

Run Java PreCommit

[chamikaramj]

Only failing test is "org.apache.beam.runners.flink.batch.ReshuffleTest.testEqualDistributionOnReshuffleAcrossMultipleStages" which is completed unrelated.

https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/5235/