[Failing Test]: testBatchGroupIntoBatchesOverrideBytes and testBatchGroupIntoBatchesOverrideCount failed when running on Dataflow Batch mode

[shunping]

What happened?

See #37363 (comment)

Looks like these tests are for batch mode only; they are ignored in streaming mode.

beam/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java

Lines 2303 to 2304 in e7c80cd

	// Ignore this test for streaming pipelines.
	assumeFalse(pipeline.getOptions().as(StreamingOptions.class).isStreaming());

beam/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java

Lines 2295 to 2296 in e7c80cd

	// Ignore this test for streaming pipelines.
	assumeFalse(pipeline.getOptions().as(StreamingOptions.class).isStreaming());

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[ATHARVA262005]

"I've been looking at the DataflowRunnerTest.java in my environment. I noticed that while some tests have the assumeFalse guard, these specific methods are failing in the Batch environment where that guard evaluates to true anyway. I'm currently looking into the GroupIntoBatches logic to see if there's a regression in how sharded keys are handled in the Batch runner.

[ATHARVA262005]

Hey @shunping

I have updated the PR (#37402) to correctly use the TestPipeline options for the streaming guard. This ensures the tests are properly skipped when the runner is in streaming mode. Could a maintainer please assign this issue to me?

[shunping]

I think there may be some misunderstanding here.

Basically, these tests are successfully skipped in streaming mode because of those guards.

They are failing on batch mode currently. So I think we should investigate why the true assertion (not the guard) in these tests failed.

[ATHARVA262005]

Thank you for the guidance, @shunping. I appreciate the correction!

As this is my first PR in the Beam community and I'm still familiarizing myself with the Dataflow Runner’s internal logic, I mistakenly focused on the environment guards rather than the underlying execution failure.

I’ve reverted the guards. I’m now investigating why the GroupIntoBatches assertion fails specifically in Batch mode. If you have any pointers on which part of the DataflowRunner translation handles these overrides for Batch, I'd love to dig deeper!