[tests] run one test but got 2 test results

[faaany]

System Info

• transformers version: 4.47.0.dev0
• Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.26.5
• Safetensors version: 0.4.5
• Accelerate version: 1.1.0.dev0
• Accelerate config: - compute_environment: LOCAL_MACHINE
  - distributed_type: MULTI_GPU
  - mixed_precision: bf16
  - use_cpu: False
  - debug: False
  - num_processes: 2
  - machine_rank: 0
  - num_machines: 1
  - gpu_ids: all
  - rdzv_backend: static
  - same_network: True
  - main_training_function: main
  - enable_cpu_affinity: False
  - downcast_bf16: no
  - tpu_use_cluster: False
  - tpu_use_sudo: False
  - tpu_env: []
• PyTorch version (GPU?): 2.5.1+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:
• Using GPU in script?:
• GPU type: NVIDIA A100 80GB PCIe

Who can help?

@ydshieh

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

with the following test command:

pytest -rA tests/models/mra/test_modeling_mra.py::MraModelTest::test_load_with_mismatched_shapes

I will get 2 test results:

======================================================== short test summary info =========================================================
PASSED tests/models/mra/test_modeling_mra.py::MraModelTest::test_load_with_mismatched_shapes
[Testing <class 'transformers.models.mra.modeling_mra.MraForSequenceClassification'>] SUBFAIL tests/models/mra/test_modeling_mra.py::MraModelTest::test_load_with_mismatched_shapes - ValueError: sequence length must be divisible by the block_size.
================================================ 1 failed, 1 passed, 3 warnings in 6.27s =================================================

Expected behavior

I expect that this command gives_ me only one test result rather than two.

Below are the experiments I tried:

• with pytest-subtests installed, comment out "with self.subTest(msg=f"Testing {model_class}"):", test pass with 1 test case
• with pytest-subtests uninstalled, comment out "with self.subTest(msg=f"Testing {model_class}"):", test pass with 1 test case
• with pytest-subtests uninstalled, test fails with 1 test case

I also saw your other PR #34806. Maybe this is another issue with pytest-subtests?

[ydshieh]

Hi @faaany !

pytest-subtests is designed to report multiple states (regarding subtests) even for a single test method.
So far in transformers, the CI env. doesn't have pytest-subtests installed (yet).
Also pytest-subtests has some issue regarding the reporting, see this.

transformers might install pytest-subtests in the CI env. soon and we definitely need to work on some issues that haven't been observed so far. But reporting multiple states for a single test is from pytest-subtests and not something transformers could change.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.