[Failing Test]: AttributeError for MaybeReshuffle in Python SDK

[apraga]

What happened?

When packaging apache beam 2.62 in nix ( NixOS/nixpkgs#379201 ), around 70 tests fail with a single error AttributeError: 'MaybeReshuffle' object has no attribute 'side_inputs'.

The list is below. Thanks for your help

apache_beam/examples/snippets/transforms/aggregation/approximatequantiles_test.py
apache_beam/examples/snippets/transforms/aggregation/approximateunique_test.py
apache_beam/examples/snippets/transforms/aggregation/batchelements_test.py
apache_beam/examples/snippets/transforms/aggregation/cogroupbykey_test.py
apache_beam/examples/snippets/transforms/aggregation/combineglobally_test.py
apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py
apache_beam/examples/snippets/transforms/aggregation/combinevalues_test.py
apache_beam/examples/snippets/transforms/aggregation/count_test.py
apache_beam/examples/snippets/transforms/aggregation/distinct_test.py
apache_beam/examples/snippets/transforms/aggregation/groupbykey_test.py
apache_beam/examples/snippets/transforms/aggregation/groupintobatches_test.py
apache_beam/examples/snippets/transforms/aggregation/latest_test.py
apache_beam/examples/snippets/transforms/aggregation/max_test.py
apache_beam/examples/snippets/transforms/aggregation/mean_test.py
apache_beam/examples/snippets/transforms/aggregation/min_test.py
apache_beam/examples/snippets/transforms/aggregation/sample_test.py
apache_beam/examples/snippets/transforms/aggregation/sum_test.py
apache_beam/examples/snippets/transforms/aggregation/tolist_test.py
apache_beam/examples/snippets/transforms/aggregation/top_test.py
apache_beam/examples/snippets/transforms/elementwise/filter_test.py
apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py
apache_beam/examples/snippets/transforms/elementwise/keys_test.py
apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py
apache_beam/examples/snippets/transforms/elementwise/map_test.py
apache_beam/examples/snippets/transforms/elementwise/pardo_test.py
apache_beam/examples/snippets/transforms/elementwise/partition_test.py
apache_beam/examples/snippets/transforms/elementwise/regex_test.py
apache_beam/examples/snippets/transforms/elementwise/tostring_test.py
apache_beam/examples/snippets/transforms/elementwise/values_test.py
apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py
apache_beam/examples/snippets/transforms/other/create_test.py
apache_beam/examples/snippets/transforms/other/flatten_test.py
apache_beam/examples/snippets/transforms/other/window_test.py
apache_beam/examples/snippets/util_test.py
apache_beam/io/avroio_test.py
apache_beam/io/concat_source_test.py
apache_beam/io/filebasedsink_test.py
apache_beam/io/filebasedsource_test.py
apache_beam/io/fileio_test.py
apache_beam/io/mongodbio_test.py
apache_beam/io/parquetio_test.py
apache_beam/io/sources_test.py
apache_beam/io/textio_test.py
apache_beam/io/tfrecordio_test.py
apache_beam/metrics/metric_test.py
apache_beam/ml/inference/base_test.py
apache_beam/ml/inference/sklearn_inference_test.py
apache_beam/ml/inference/utils_test.py
apache_beam/ml/rag/chunking/base_test.py
apache_beam/pipeline_test.py
apache_beam/runners/direct/direct_runner_test.py
apache_beam/runners/direct/sdf_direct_runner_test.py
apache_beam/runners/interactive/interactive_beam_test.py
apache_beam/runners/interactive/interactive_runner_test.py
apache_beam/runners/interactive/non_interactive_runner_test.py
apache_beam/runners/interactive/recording_manager_test.py
apache_beam/runners/portability/fn_api_runner/translations_test.py
apache_beam/runners/portability/fn_api_runner/trigger_manager_test.py
apache_beam/runners/portability/stager_test.py
apache_beam/testing/synthetic_pipeline_test.py
apache_beam/testing/test_stream_test.py
apache_beam/testing/util_test.py
apache_beam/transforms/combiners_test.py
apache_beam/transforms/core_test.py
apache_beam/transforms/create_test.py
apache_beam/transforms/deduplicate_test.py
apache_beam/transforms/periodicsequence_test.py
apache_beam/transforms/ptransform_test.py
apache_beam/transforms/sideinputs_test.py
apache_beam/transforms/stats_test.py
apache_beam/transforms/transforms_keyword_only_args_test.py
apache_beam/transforms/trigger_test.py
apache_beam/transforms/userstate_test.py
apache_beam/transforms/util_test.py
apache_beam/transforms/write_ptransform_test.py

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[TanuSharma2511]

One potential reason for this error is dependencies conflict or not proper installation of any dependency. You should see more information in stack trace. In my case, pyarrow was creating issue. So, I re-install it: pip install pyarrow and error got resolved.

Please check the stack trace. You will get the idea which dependency is creating issue.

[smblsm]

Wondering if there's any more info on this, because I'm running into it to when upgrading from 2.60.0 to 2.64.0. Any test using TestPipeline fails with AttributeError: 'MaybeReshuffle' object has no attribute 'side_inputs'.

[apraga]

Hi,

Error log is attached if that can help.

error.log

[TanuSharma2511]

Can you please try clearing your dependency cache and reinstalling beam + dependencies ? If it does not work, try with some different python version. (For example 3.9.2)

[GaetanLepage]

Hello,
This issue was opened in the context of apache-beam packaging in the nixpkgs software repository.
We (package maintainers) are observing that these tests fail and we have thus disabled them. We run the test suite of all python packages to ensure that we have packaged them correctly.

Can you please try clearing your dependency cache and reinstalling beam + dependencies ?

Due to the nature of the nix builder itself, there is no polluting dependency cache to clear. The environment in which the package is tested only contains the dependencies that we have explicitly listed.

If it does not work, try with some different python version. (For example 3.9.2)

We currently support the python 3.12 and python 3.13 interpreters and are ensuring that all python packages function correctly against these versions.

[shunping]

I happened to see this issue (and a few other related ones mentioned in NixOS/nixpkgs#379201) while scanning through our backlog. Thanks for bring this up.

We would like to help but we are not familiar with the packaging system/process in nix. If you can provide steps to reproduce (e.g. maybe running your tests in a docker environment), then I can help to take a look.