[Python] ResourceWarnings raised by s3 related tests

[asfimport]

Running the python tests give a lot of the following warnings:

opt/conda/envs/arrow/lib/python3.9/site-packages/pyarrow/tests/test_fs.py::test_s3fs_limited_permissions_create_bucket
  /opt/conda/envs/arrow/lib/python3.9/site-packages/pyarrow/tests/util.py:439: ResourceWarning: unclosed file <_io.TextIOWrapper name=29 encoding='utf-8'>
    _run_mc_command(mcdir, 'admin', 'policy', 'add',
  Enable tracemalloc to get traceback where the object was allocated.
  See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.

Ideally we should ensure the tests don't give such warnings (it also makes other warning we should notice less visible)

Reporter: Joris Van den Bossche / @jorisvandenbossche

Related issues:

• [Python] Reduce warnings during tests (is a child of)

_{Note: This issue was originally created as ARROW-17651. Please see the migration documentation for further details.}

[Shaheer-Ahmd]

I would like to contribute to this, but have no idea where to start my work.
Any help will be really appreciated!

[westonpace]

@Shaheer-Ahmd

The first step would be to clone the repo and ensure that you can build the C++ and python implementations. This guide should have the information you need. If you run into troubles you can ask here (this issue or a new question issue) or on the mailing list.

Once built I would suggest following the instructions suggested in the diagnostic and Enable tracemalloc to get traceback where the object was allocated.

Once you've figured out the offending resource it would then be a matter of figuring out how to close it. If that is not obvious then reply back here and share what you got from tracemalloc. We can likely give you suggestions how to proceed from there.

[Shaheer-Ahmd]

Feb 2, 2023

Hey @westonpace !
Thank you for giving me a direction.
I have cloned the repo and installed the requirements with vcpkg as explained in the guide. I have downloaded cmake from the internet but, when I pass the command cmake .. --preset ninja-debug-basic within the arrow/cpp/build directory, the following error arises:

[jorisvandenbossche]

@Shaheer-Ahmd the user mailing list (see https://arrow.apache.org/community/) or a separate issue might be a better place to ask about the building error, since this is not related to the actual issue here.

[westonpace]

#34029 looks related. I suspect you need to install ninja-build. Sorry, it was my suggestion to report problems here but Joris is right the ML or a separate issue might be easiest.

[Shaheer-Ahmd]

@jorisvandenbossche @westonpace

When I send my query to dev@arrow.apache.org, it fails to send the mail. Is there a specific format that we have to follow to get our mails through? My mail only contains text.

Any Help would be appreciated!

[westonpace]

@jorisvandenbossche @westonpace

When I send my query to dev@arrow.apache.org, it fails to send the mail. Is there a specific format that we have to follow to get our mails through? My mail only contains text.

Any Help would be appreciated!

You will need to subscribe to the mailing list first.

Follow the instructions here and send a message to dev-subscribe@arrow.apache.org

[Shaheer-Ahmd]

I am already subscribed to the mailing list. I have attached the error its giving.

 ezmlm-send: fatal: Sorry, after removing unacceptable MIME parts from your message I was left with nothing (#5.7.0)
ezmlm-gate: fatal: fatal error from child

[westonpace]

I'm afraid I have not seen that message before and I'm not sure what the next steps for debugging the mailing list issue would be. Perhaps someone else will know. In the meantime you are still welcome to open a dedicated issue on the topic (however, I think the fix for your build problem is to install ninja (on Ubuntu this would be apt-get install ninja-build)

[Shaheer-Ahmd]

I'm afraid I have not seen that message before and I'm not sure what the next steps for debugging the mailing list issue would be. Perhaps someone else will know. In the meantime you are still welcome to open a dedicated issue on the topic (however, I think the fix for your build problem is to install ninja (on Ubuntu this would be apt-get install ninja-build)

Okay, I am opening a new issue for this.
Thank you!

[jorisvandenbossche]

This was fixed by @danepitkin in #37018

Closed by #37018