[Failing Test]: Jenkins job beam_PostCommit_Java_Nexmark_Flink not being finished

[aromanenko-dev]

What happened?

A Jenkins job beam_PostCommit_Java_Nexmark_Flink keeps running forever and not finished properly (aborted after a limit of 4 hours). So it wastes the resources of apache-beam-jenkins-16 machine.

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

Same for GItHub Action. Start timing out on Oct 28th: https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Nexmark_Flink.yml?page=4&query=event%3Aschedule

In particular, the streaming queries all timed out

[Abacn]

The breakage happens between 7b38f74 (inclusive) and 7b38f74 (exclusive), #28614 is the only related change CC: @jto @tvalentyn

[Abacn]

Comparing the log it seems the pipeline is executed differently.

Before:

INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (1/2) (b5360562850a3be1374b8798dfacfdce) switched from DEPLOYING to INITIALIZING.
Oct 27, 2023 4:25:11 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper initializeState
INFO: No restore state for UnboundedSourceWrapper.
Oct 27, 2023 4:25:11 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper initializeState
INFO: No restore state for UnboundedSourceWrapper.
Oct 27, 2023 4:25:11 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper open
INFO: Unbounded Flink Source 1/2 is reading from sources: [UnboundedEventSource(0, 1000), UnboundedEventSource(2000, 3000), UnboundedEventSource(4000, 5000), UnboundedEventSource(6000, 7000), UnboundedEventSource(8000, 9000), UnboundedEventSource(10000, 11000), UnboundedEventSource(12000, 13000), UnboundedEventSource(14000, 15000), ...]
INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (2/2)#0 (6cb95a324b37d5272f769a25e93e5906) switched from INITIALIZING to RUNNING.

after:

INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (2/2) (eee384163299676ced9b2a87bf24bbca) switched from DEPLOYING to INITIALIZING.
Oct 27, 2023 10:22:48 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.unbounded.FlinkUnboundedSourceReader start
WARNING: AutoWatermarkInterval is not set, watermarks won't be emitted.
...
INFO: Source: Query0.ReadUnbounded -> Flat Map -> Query0/Query0.Monitor/ParMultiDo(Monitor) -> Query0/Query0.Snoop/ParMultiDo(Anonymous) -> Query0/Query0/Query0.Serialize/ParMultiDo(Anonymous) -> Query0/Query0.Debug/ParMultiDo(Monitor) -> Query0/Query0.Stamp/ParMultiDo(Anonymous) -> Query0.Format/ParMultiDo(Anonymous) -> Query0.DevNull/ParMultiDo(Anonymous) (2/2) (eee384163299676ced9b2a87bf24bbca) switched from INITIALIZING to RUNNING.
Oct 27, 2023 10:22:48 PM org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceSplitEnumerator assignSplitsAndLog
INFO: Assigned splits [[SplitIndex: 0, BeamSource: UnboundedEventSource(0, 1000)], [SplitIndex: 2, BeamSource: UnboundedEventSource(2000, 3000)], [SplitIndex: 4, BeamSource: UnboundedEventSource(4000, 5000)], [SplitIndex: 6, BeamSource: UnboundedEventSource(6000, 7000)], [SplitIndex: 8, BeamSource: UnboundedEventSource(8000, 9000)], [SplitIndex: 10, BeamSource: UnboundedEventSource(10000, 11000)], [SplitIndex: 12, BeamSource: UnboundedEventSource(12000, 13000)], [SplitIndex: 14, BeamSource: 

In particular, before, UnboundedEventSource splits at INITIALIZING, log from org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper open

After, UnboundedEventSource split at RUNNING, log from org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceSplitEnumerator assignSplitsAndLog

From #28614, the PR should only change batch, but it's breaking streaming. @jto could you please investigate? Thanks!

[jto]

Hey! I may have an idea but I'd need a way to run the specific test that's blocked to confirm. How can I do that ?

[Abacn]

Under your PR, comment "Run Flink Runner Nexmark Tests" then Jenkins will pick up the PR and run the test

[jto]

I meant I'd like to run a specific blocked test locally on my machine so I can quickly iterate on a fix.

[Abacn]

The equivalent gradle command is

./gradlew :sdks:java:testing:nexmark:run -Pnexmark.runner=:runners:flink:1.15 -Pnexmark.args="--manageResources=false --monitorJobs=true --resourceNameMode=QUERY_RUNNER_AND_MODE  --exportSummaryToInfluxDB=false  --influxRetentionPolicy=forever --suite=SMOKE  --streamTimeout=60  --runner=FlinkRunner --streaming=true --queryLanguage=sql"

see https://beam.apache.org/documentation/sdks/java/testing/nexmark/

I was able to run this command on release-2.51.0 branch ("BUILD SUCCESSFUL in 6m") but on master it stucks at same log

Nov 16, 2023 11:34:30 AM org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.FlinkSourceReaderBase notifyNoMoreSplits
INFO: Received NoMoreSplits signal from enumerator.

[Abacn]

just a note, if compile locally it's good to be based on the latest master or cherrypick #29446, otherwise the compile is slow due to reason stated in that PR