[Bug]: Number of Python examples are failing for Flink and Spark on 2.43.0 release branch

[chamikaramj]

What happened?

Seems like following Python examples are failing for Flink and Spark on 2.43.0 release branch.

• bigquery_tornadoes
• hourly_team_score
• filters_test

For example,
https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark_PR/14/
https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink_PR/12/

For bigquery_tornadoes, the error is following.

RuntimeError: Pipeline BeamApp-jenkins-1028002928-4325f8c5_d8258ff0-6727-4335-867f-56bc846d9f3e failed in state FAILED: java.lang.IllegalArgumentException: PCollectionNodes [PCollectionNode{id=ref_PCollection_PCollection_52, PCollection=unique_name: "61Write/BigQueryBatchFileLoads/TriggerLoadJobsWithoutTempTables.None"
coder_id: "ref_Coder_FastPrimitivesCoder_3"
is_bounded: BOUNDED
windowing_strategy_id: "ref_Windowing_Windowing_1"
}] were consumed but never produced

I found #21300 that probably explain some example failures but this doesn't explain all failures above.

Valentyn, are there any known issues that explain these failures ?

Issue Priority

Priority: 0

Issue Component

Component: examples-python

[chamikaramj]

P0 since this is blocking the ongoing Beam release.

[tvalentyn]

Did these pass on previous release? If so, errors should be bisectable.

[Abacn]

Looks like failed tests are all BigQuery tests. Both have been failed after Sept 14th. Possible related changes on the Day of 14th: #23122 and/or #23012

[tvalentyn]

Thanks, @Abacn . The first change affects go sdk only. #23012 looks suspicious. @ahmedabu98 could you please take a look?

[chamikaramj]

Confirmed that this:

Passes on 2.42.0 branch
Fails on 2.43.0 branch
Fails on master

Will also try a revert.

[chamikaramj]

Fails for commit ac37784

Passes for commit 2d4f61c (one before above).

So the culprit seems to be ac37784

@ahmedabu98 can you please take a look ? We can either do a forward fix in the release branch or revert this if we can revert cleanly.

To reproduce locally, you can run the command below.

./gradlew :sdks:python:test-suites:portable:py37:flinkExamples

[ahmedabu98]

Seeing a lot of ValueError: Unable to parse jar URL "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Examples_Spark/src/runners/spark/2/job-server/build/libs/beam-runners-spark-3-job-server-2.44.0-SNAPSHOT.jar". If using a full URL, make sure the scheme is specified. If using a local file path, make sure the file exists; you may have to first build the job server using './gradlew runners:spark:3:job-server:shadowJar'.
Is this familiar to anyone?