[BEAM-13806] Adding test suite for Go x-lang tests on Dataflow.

[youngoli]

Following the pattern of the other Dataflow X-Lang test suites, this one adds Go using Java expansion services. This needs to be tested on Jenkins.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---					---
Java
Python	---					---
XLang						---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[youngoli]

Run Seed Job

[youngoli]

Run XVR_GoUsingJava_Dataflow PostCommit

[youngoli]

Run Seed Job

[youngoli]

Run XVR_GoUsingJava_Dataflow PostCommit

[aaltay]

@youngoli - Is this ready for a review?

[aaltay]

@youngoli - could you add reviewers?

[youngoli]

Sorry I haven't been checking this. I want to review it after I get confirmation that this works with Jenkins. Last time I tried Jenkins was completely down and even now it's still being a bit difficult (last I saw SeedJob was failing, and I'm trying now and the site isn't loading).

[aaltay]

Run Java PreCommit

[youngoli]

Run Seed Job

[youngoli]

Run XVR_GoUsingJava_Dataflow PostCommit

[youngoli]

Run XVR_GoUsingJava_Dataflow PostCommit

[youngoli]

Run XVR_GoUsingJava_Dataflow PostCommit

[youngoli]

Run Seed Job

[youngoli]

Run XVR_GoUsingJava_Dataflow PostCommit

[youngoli]

Run Seed Job

[youngoli]

Run XVR_GoUsingJava_Dataflow PostCommit

[youngoli]

R: @lostluck

This is the run I actually got to work: https://ci-beam.apache.org/job/beam_PostCommit_XVR_GoUsingJava_Dataflow_PR/5/

The errors look like legit errors as far as I can tell, although I haven't exactly been able to diagnose what's happening. Personally I'm in favor of just merging this as-is, and then triaging and creating Jiras like we usually do when one of our test suites starts failing.

[youngoli]

R: @ihji
Since you've been the main dev working on the Dataflow xlang tests.

[aaltay]

@ihji - could you please take a look at this change?

[ihji]

Thanks, looks good to me.

[youngoli]

Just to clarify why I think this should be submitted despite the test being red (since I've been getting some confusion about it offline):

From what I investigated, the test framework and Jenkins configuration itself, which is what this PR is changing, seems to be working just fine. It runs the test suite no problem, the failure doesn't seem to be due to the test environment, and all the tests except one are passing.

This indicates to me that the redness is due to an actual failure that was already happening but unnoticed because it wasn't being tested. The next steps once this is merged is to immediately create a Jira for this failure and start triaging and fixing it, like we do when existing test suites turn red.

[y1chi]

this is perma-red https://ci-beam.apache.org/job/beam_PostCommit_XVR_GoUsingJava_Dataflow/, also could we move this suite off the java runner harness, perhaps use ULR instead?

[lostluck]

All Go SDK jobs on Dataflow should be using UW, and not the Java Runner Harness. By ULR, do you mean the Python Portable Runner? Wouldn't that be a different suite?

…

On Thu, Apr 28, 2022, 5:35 PM Yichi Zhang ***@***.***> wrote: this is perma-red https://ci-beam.apache.org/job/beam_PostCommit_XVR_GoUsingJava_Dataflow/, also could we move this suite off the java runner harness, perhaps use ULR instead? — Reply to this email directly, view it on GitHub <#16819 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKDOFLQSW66IOJCGCIZSHTVHMVDXANCNFSM5OC6UMVQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[y1chi]

All Go SDK jobs on Dataflow should be using UW, and not the Java Runner Harness. By ULR, do you mean the Python Portable Runner? Wouldn't that be a different suite?
…
On Thu, Apr 28, 2022, 5:35 PM Yichi Zhang @.> wrote: this is perma-red https://ci-beam.apache.org/job/beam_PostCommit_XVR_GoUsingJava_Dataflow/, also could we move this suite off the java runner harness, perhaps use ULR instead? — Reply to this email directly, view it on GitHub <#16819 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADKDOFLQSW66IOJCGCIZSHTVHMVDXANCNFSM5OC6UMVQ . You are receiving this because you were mentioned.Message ID: @.>

if this is using UW I see no reason for it to depend on ":runners:google-cloud-dataflow-java:worker:shadowJar", perhaps I misunderstood something.