GH-20127: [Python][CI] Remove legacy hdfs tests from hdfs and hypothesis setup

[jorisvandenbossche]

Rationale for this change

Small follow-up on #39825, which removed the test_hdfs.py file itself, but didn't remove it from the hypothesis script

• GitHub Issue: [Python] Remove the deprecated pyarrow.filesystem legacy implementations #20127

[jorisvandenbossche]

@github-actions crossbow submit test-conda-python-3.11-hypothesis

[github-actions]

Revision: 6e84caa

Submitted crossbow builds: ursacomputing/crossbow @ actions-3636eb2740

Task	Status
test-conda-python-3.11-hypothesis

[pitrou]

It would be nice to also fix the HDFS builds themselves (failures seem related?).
https://github.com/ursacomputing/crossbow/actions/runs/8148998591/job/22272924163

[pitrou]

@github-actions crossbow submit hdfs

[github-actions]

Revision: 6e84caa

Submitted crossbow builds: ursacomputing/crossbow @ actions-4e7114f791

Task	Status
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1

[jorisvandenbossche]

It would be nice to also fix the HDFS builds themselves (failures seem related?).

Ah yes, sorry I should have checked that when merging the PR. But indeed that should be fixed now as well

[jorisvandenbossche]

The hypothesis tests are still failing (but now actual test failures instead of erroring on test collection for the missing file). the output is quite incomprehensible though, so not sure what's the issue from a quick look

[jorisvandenbossche]

HDFS build is green now!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 3d467ac.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.