GH-35728: [CI][Python] Move test_total_bytes_allocated to a subprocess to improve reliability #36355

raulcd · 2023-06-28T13:49:44Z

Rationale for this change

Currently some conda and wheels jobs are failing due to this error:

 =================================== FAILURES ===================================
__________________________ test_total_bytes_allocated __________________________

    def test_total_bytes_allocated():
>       assert pa.total_allocated_bytes() == 0
E       assert 1216 == 0
E        +  where 1216 = <built-in function total_allocated_bytes>()
E        +    where <built-in function total_allocated_bytes> = pa.total_allocated_bytes

What changes are included in this PR?

As suggested on the issue trying to move this test to a subprocess to see if it improves reliabilit.

Are these changes tested?

Archery and CI

Are there any user-facing changes?

No

Closes: [Python] test_total_bytes_allocated failing on CI #35728

…process to improve reliability

github-actions · 2023-06-28T13:50:15Z

⚠️ GitHub issue #35728 has been automatically assigned in GitHub to PR creator.

raulcd · 2023-06-28T13:50:51Z

@github-actions crossbow submit wheel-manylinux-*

github-actions · 2023-06-28T13:54:11Z

Revision: 0806a80

Submitted crossbow builds: ursacomputing/crossbow @ actions-020f7043d9

Task	Status
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64

raulcd · 2023-06-28T14:42:54Z

@github-actions crossbow submit wheel-manylinux-2014-*-arm64

github-actions · 2023-06-28T14:45:31Z

Revision: 4d3d57e

Submitted crossbow builds: ursacomputing/crossbow @ actions-8d319d3cbd

Task	Status
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-arm64

raulcd · 2023-06-28T15:21:34Z

@github-actions crossbow submit wheel-manylinux-2014-cp310-arm64

github-actions · 2023-06-28T15:23:57Z

Revision: 4d3d57e

Submitted crossbow builds: ursacomputing/crossbow @ actions-5976a03430

Task	Status
wheel-manylinux-2014-cp310-arm64

raulcd · 2023-06-28T15:25:30Z

the flaky failures seem unrelated to this test now but to s3

 Timeout (0:05:00)!
Thread 0x0000ffff8fcca420 (most recent call first):
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pyarrow/tests/test_fs.py", line 1122 in test_s3_options
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/python.py", line 1788 in runtest
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/runner.py", line 341 in from_call
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/main.py", line 324 in _main
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/config/__init__.py", line 166 in main
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/_pytest/config/__init__.py", line 189 in console_main
  File "/tmp/arrow-HEAD.1rhW8/venv-wheel-3.8-manylinux_2_17_aarch64.manylinux2014_aarch64/lib64/python3.8/site-packages/pytest/__main__.py", line 5 in <module>
  File "/usr/lib64/python3.8/runpy.py", line 87 in _run_code
  File "/usr/lib64/python3.8/runpy.py", line 194 in _run_module_as_main

pitrou · 2023-06-28T15:32:47Z

@github-actions crossbow submit -g python

github-actions · 2023-06-28T15:36:12Z

Revision: 4d3d57e

Submitted crossbow builds: ursacomputing/crossbow @ actions-b168c1dadc

Task	Status
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-master
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.1.2
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-conda-python-3.9-spark-v3.2.0
test-cuda-python
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3

raulcd · 2023-06-29T14:07:13Z

@pitrou are you ok merging this? All the failures are known unrelated nightly failures

pitrou · 2023-06-29T14:39:24Z

@raulcd Yes, please merge!

conbench-apache-arrow · 2023-07-04T15:55:07Z

Conbench analyzed the 6 benchmark runs on commit fa4e27a6.

There were 3 benchmark results indicating a performance regression:

Commit Run on ursa-thinkcentre-m75q at 2023-07-02 19:11:16Z
- params=<Repetition::REQUIRED, Compression::LZ4>/65536/1024, source=cpp-micro, suite=parquet-column-io-benchmark
Commit Run on arm64-m6g-linux-compute at 2023-06-29 16:29:48Z
- params=<STATIC_VECTOR(std::string)>, source=cpp-micro, suite=arrow-small-vector-benchmark
and 1 more (see the report linked below)

The full Conbench report has more details.

apacheGH-35728: [CI][Python] Move test_total_bytes_allocated to a sub…

0806a80

…process to improve reliability

github-actions bot added Component: Python awaiting committer review Awaiting committer review labels Jun 28, 2023

Fix linter by removing unnecessary f-string

4d3d57e

pitrou approved these changes Jun 28, 2023

View reviewed changes

raulcd marked this pull request as ready for review June 28, 2023 15:36

raulcd requested a review from AlenkaF as a code owner June 28, 2023 15:36

raulcd added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Jun 28, 2023

raulcd merged commit fa4e27a into apache:main Jun 29, 2023

raulcd removed the awaiting merge Awaiting merge label Jun 29, 2023

GH-35728: [CI][Python] Move test_total_bytes_allocated to a subprocess to improve reliability #36355

GH-35728: [CI][Python] Move test_total_bytes_allocated to a subprocess to improve reliability #36355

Uh oh!

Conversation

raulcd commented Jun 28, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

raulcd commented Jun 28, 2023

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

raulcd commented Jun 28, 2023

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

raulcd commented Jun 28, 2023

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

raulcd commented Jun 28, 2023

Uh oh!

pitrou commented Jun 28, 2023

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

raulcd commented Jun 29, 2023

Uh oh!

pitrou commented Jun 29, 2023

Uh oh!

conbench-apache-arrow bot commented Jul 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

raulcd commented Jun 28, 2023 •

edited by github-actions bot

Loading