2.42.0 Release Branch: Flink Examples integration test failing

[lostluck]

What happened?

Release Validation branch #23200 has failures for the Flink Example post commits:

https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Flink_PR/8/

from the beam branch: ./gradlew :runners:flink:1.13:examplesIntegrationTest replicates the same failure.

Errors are all the following InvalidProgramException

08:52:14 org.apache.beam.examples.snippets.transforms.io.gcp.bigquery.BigQuerySamplesIT > testTableIO FAILED
08:52:14     org.apache.beam.sdk.Pipeline$PipelineExecutionException at BigQuerySamplesIT.java:108
08:52:14         Caused by: org.apache.flink.api.common.InvalidProgramException at BigQuerySamplesIT.java:108

Other flink validation is passing so it appears related to gradle configuration or similar. This needs to be determined and or resolved before RC2 is built.

Issue Priority

Priority: 1

Issue Component

Component: runner-flink

[lostluck]

Confirmed this is still happening at Master Head, and was working at the time of the 2.41.0 release, so this is a regression in configuration at minimum.

[lostluck]

git bisect blames f973acc
[Website] add Python to KinesisIO in connectors #22845 (#22841)
looking into it...

[lostluck]

git bisect log
git bisect start
# bad: [135877e04a0d4e0349c48b65ebd974a20d764a41] Fix IllegalStateException in StorageApiWriteUnshardedRecords error handling. (#23233)
git bisect bad 135877e04a0d4e0349c48b65ebd974a20d764a41
# good: [7ef51bc8733c6bcba874d18a77e83762e9c4567f] Merge pull request #22759 from kileys/releasebranch
git bisect good 7ef51bc8733c6bcba874d18a77e83762e9c4567f
# good: [cd5f88abbb571618b633b6a0068ef4f14fac55d3] [22369] Default Metrics for Executable Stages in Samza Runner (#22370)
git bisect good cd5f88abbb571618b633b6a0068ef4f14fac55d3
# good: [f921a2f1996cf906d994a9d62aeb6978bab09dd5] Fix lint issues (#22800)
git bisect good f921a2f1996cf906d994a9d62aeb6978bab09dd5
# bad: [149ed074428ff9b5344169da7d54e8ee271aaba1] 22805: Upgrade Jackson version from 2.13.0 to 2.13.3 (#22806)
git bisect bad 149ed074428ff9b5344169da7d54e8ee271aaba1
# bad: [b09a8c7010b191cc788364bfdf223b201d0b9b08] Preserve state on SDK switch (#22430) (#22735)
git bisect bad b09a8c7010b191cc788364bfdf223b201d0b9b08
# good: [754de4635f1844735924d2d37c89e5ec7ee16245] [BEAM-22723] Yield BatchElement batches at end of window.
git bisect good 754de4635f1844735924d2d37c89e5ec7ee16245
# good: [63653aec39db58982108a651ee6e56a76e508d5e] Bump google.golang.org/grpc from 1.48.0 to 1.49.0 in /sdks (#22838)
git bisect good 63653aec39db58982108a651ee6e56a76e508d5e
# bad: [f973acc47101bff6313cebcdbc9a511e499cb8fd] [Website] add Python to KinesisIO in connectors #22845 (#22841)
git bisect bad f973acc47101bff6313cebcdbc9a511e499cb8fd
# good: [d47438a29775598910b48da89fd9c394627165aa] Update Dataflow fnapi_container-version (#22852)
git bisect good d47438a29775598910b48da89fd9c394627165aa
# good: [42b1640a25d5dbdea08ae2feaa0d3e81f6278575] unskip sklearn IT test (#22825)
git bisect good 42b1640a25d5dbdea08ae2feaa0d3e81f6278575
# first bad commit: [f973acc47101bff6313cebcdbc9a511e499cb8fd] [Website] add Python to KinesisIO in connectors #22845 (#22841)

[lostluck]

But that seems rather unrelated. Going to run it again, but it's due to flakes.

[lostluck]

Second pass seems more readinable:

git bisect start
# good: [8347b9e1d36cb8c2a1d863909d2d27a00a3efdaa] Merge pull request #22814 from cushon/bb
git bisect good 8347b9e1d36cb8c2a1d863909d2d27a00a3efdaa
# bad: [1526ca8c4cc6d58b3c28d816fc2597e51603d75f] Improvements to SchemaTransform implementations for BQ and Kafka (#23045)
git bisect bad 1526ca8c4cc6d58b3c28d816fc2597e51603d75f
# bad: [fe297c365356c4b4f7dc294206d64300c6a09c3a] Add run-inference component for autolabeling (#22971)
git bisect bad fe297c365356c4b4f7dc294206d64300c6a09c3a
# bad: [149ed074428ff9b5344169da7d54e8ee271aaba1] 22805: Upgrade Jackson version from 2.13.0 to 2.13.3 (#22806)
git bisect bad 149ed074428ff9b5344169da7d54e8ee271aaba1
# bad: [7cc48e9dc3f8c44b0cc5dcfd2cd4eca51096ecdc] Handle stateful windows correctly + integration test (#22918)
git bisect bad 7cc48e9dc3f8c44b0cc5dcfd2cd4eca51096ecdc
# bad: [9ca1f7acd5cc6f59017aa3961192bd17ed6dcb2e] Go Map State integration test (#22898)
git bisect bad 9ca1f7acd5cc6f59017aa3961192bd17ed6dcb2e
# bad: [39967a7ba3dbb391db0757c7567a7e8c45f85fe8] Update wordcount_minimal.py by removing pipeline_args.extend (#22786)
git bisect bad 39967a7ba3dbb391db0757c7567a7e8c45f85fe8
# bad: [172bd31d01bcc1e87a5aa1e37379a669c81b46bf] [Go SDK] Stream decode values in single iterations (#22904)
git bisect bad 172bd31d01bcc1e87a5aa1e37379a669c81b46bf
# bad: [3217017aee8e4a0a95f0153a541c6626ca1c1d4a] Pass user specified destination type to UpdateSchemaDestination  (#22624) fixing #22543
git bisect bad 3217017aee8e4a0a95f0153a541c6626ca1c1d4a
# first bad commit: [3217017aee8e4a0a95f0153a541c6626ca1c1d4a] Pass user specified destination type to UpdateSchemaDestination  (#22624) fixing #22543

[lostluck]

@ahmedabu98 your PR #22624 seems to have broken the Flink Examples using big query.

Could you please investigate and resolve?

[ahmedabu98]

.take-issue