[Bug]: Dataflow worker container resolved to legacy runner label if not explicitly disable/enable runner v2 in 2.54.0+.dev

[Abacn]

What happened?

Run a batch job on Dataflow runner use SNAPSHOT version, the Dataflow job now stucks, with error

Container image gcr.io/cloud-dataflow/v1beta3/beam_java8_sdk:beam-master-20240306 not downloaded yet.

This is seen in #30505 (comment)

It does not affect release as both dataflowLegacyContainerVersion and dataflowFnapiContainerVersion are identical

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

The logic of determining contaner label is here:

beam/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java

Line 2516 in ef919e2

static String getDefaultContainerImageUrl(DataflowPipelineOptions options) {

calls

beam/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java

Line 2542 in ef919e2

static String getDefaultContainerVersion(DataflowPipelineOptions options) {

calls

beam/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java

Line 2554 in ef919e2

static boolean useUnifiedWorker(DataflowPipelineOptions options) {

As for batch pipeline, it should still resolve to `gcr.io/cloud-dataflow/v1beta3/beam-java-batch:beam-master-20240306'. Does the replace happen on the Dataflow side ?

[Abacn]

This also affects

beam/it/google-cloud-platform/src/main/java/org/apache/beam/it/gcp/dataflow/AbstractPipelineLauncher.java

Lines 261 to 267 in a3e5ac8

	String runner = LEGACY_RUNNER;
	Environment environment = job.getEnvironment();
	if (environment != null
	&& environment.getExperiments() != null
	&& environment.getExperiments().contains("use_runner_v2")) {
	runner = RUNNER_V2;
	}

the throughput metrics are not correctly reported in IOLoadTests for read pipelines

[kennknowles]

Seems like we should first explicitly make the v2 image and the v1 image use entirely different code paths, since those images are incompatible types (in the sense of type systems). I know there were places where we used the same flag or config field and we should remove that universally.

[kennknowles]

Specifically, the fact that this function exists shows that we have misunderstood the use of these containers

beam/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java

Lines 2528 to 2537 in ef919e2

	static String getDefaultContainerImageNameForJob(DataflowPipelineOptions options) {
	Environments.JavaVersion javaVersion = Environments.getJavaVersion();
	if (useUnifiedWorker(options)) {
	return String.format("beam_%s_sdk", javaVersion.name());
	} else if (options.isStreaming()) {
	return String.format("beam-%s-streaming", javaVersion.legacyName());
	} else {
	return String.format("beam-%s-batch", javaVersion.legacyName());
	}
	}

[Abacn]

Does the replace happen on the Dataflow side ?

This is the case. There is an internal overwrites on service side, causing client and Dataflow have different understanding whether runner v2 enabled.

It is not straightforward to change Beam side logic. Currently the auto runner v2 has a fallback mechanism. For a pipeline not specifying runner v2, when it detects feature there are Beam features not supported by runner v2, it falls back to legacy runner.