[Failing Test]: Wordcounts in preCommitLegacyWorkerImpersonate portion of beam_PreCommit_Java_Examples_Dataflow are permared

[tvalentyn]

What happened?

java.io.IOException: Error matching file spec gs://apache-beam-samples/shakespeare/sonnets.txt: status ERROR
	at org.apache.beam.sdk.io.FileSystems.matchSingleFileSpec(FileSystems.java:196)
	at org.apache.beam.sdk.util.ExplicitShardedFile.<init>(ExplicitShardedFile.java:58)
	at org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountPipeline(WindowedWordCountIT.java:174)
	at org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInStreamingStaticSharding(WindowedWordCountIT.java:113)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)

checked that gs://apache-beam-samples/shakespeare/sonnets.txt exists

checked last PR that was merged after which the suite started failing on master on
https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml?query=is%3Afailure+branch%3Amaster

Presubmits passed on that PR: #30172

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[shunping]

Presubmits passed on that PR: #30172

I think the presubmit for that PR failed actually.
https://github.com/apache/beam/actions/runs/9244426284

[tvalentyn]

That apache/beam/actions/runs/9244426284 run was 'Triggered via push 2 days ago' (from the metadata), that means, GHA started the suite after the merge of the pr into master branch.

The run on the PR itself was this one: https://github.com/apache/beam/actions/runs/7732101964/job/21081256349

[tvalentyn]

Actually, the PR was merged after a long pause in reviews, it is likely that the suite passed on the PR a while ago, but by the time we got around to merge it, it wasn't passing anymore. Let's verify on a rollback: #31423

[tvalentyn]

Rollback doesn't seem to help. Also saw:

java.lang.IllegalArgumentException: DataflowRunner requires gcpTempLocation, but failed to retrieve a value from PipelineOptions
	at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:291)
	at org.apache.beam.runners.dataflow.TestDataflowRunner.<init>(TestDataflowRunner.java:71)
	at org.apache.beam.runners.dataflow.TestDataflowRunner.fromOptions(TestDataflowRunner.java:90)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

[shunping]

I had two PRs (#31358, #31324) related to tempLocation last week, but they were at least five days before the test started to fail.

[VineetRamuk]

I had similar failure for #31413. Can it be access issue?

https://github.com/apache/beam/pull/31413/checks?check_run_id=25470855044

Caused by: java.io.IOException: Error requesting access token
at com.google.auth.oauth2.ImpersonatedCredentials.refreshAccessToken(ImpersonatedCredentials.java:516)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:270)
at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)

[Abacn]

The failed task is ':runners:google-cloud-dataflow-java:examples:preCommitLegacyWorkerImpersonate'. It uses a dedicate "impersonate" service account to interact with GCP. Local run on your own credential is fine

The actual error message:

Caused by: com.google.api.client.http.HttpResponseException: 403 Forbidden
POST https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/allows-impersonation@apache-beam-testing.iam.gserviceaccount.com:generateAccessToken
{
  "error": {
    "code": 403,
    "message": "Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).",
    "errors": [
      {
        "message": "Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).",
        "domain": "global",
        "reason": "forbidden"
      }
    ],
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "IAM_PERMISSION_DENIED",
        "domain": "iam.googleapis.com",
        "metadata": {
          "permission": "iam.serviceAccounts.getAccessToken"
        }
      }
    ]
  }
}

Either permission of the service account "allows-impersonation@apache-beam-testing.iam.gserviceaccount.com" is changed unexpectedly, or there are some recent breaking change on GCP side