[C++][Python] Sporadic asof_join failures in PyArrow

[pitrou]

Describe the bug, including details regarding any error messages, version, and platform.

We see sporadic CI failures in test_dataset_join_asof_empty_by. Example:
https://github.com/ursacomputing/crossbow/actions/runs/8597009571/job/23554761859#step:10:787

=================================== FAILURES ===================================
_______________________ test_dataset_join_asof_empty_by ________________________

tempdir = PosixPath('/tmp/pytest-of-root/pytest-0/test_dataset_join_asof_empty_b0')

    @pytest.mark.dataset
    def test_dataset_join_asof_empty_by(tempdir):
        t1 = pa.table({
            "on": [1, 2, 3],
        })
        ds.write_dataset(t1, tempdir / "t1", format="ipc")
        ds1 = ds.dataset(tempdir / "t1", format="ipc")
    
        t2 = pa.table({
            "colVals": ["Z", "B", "A"],
            "on": [2, 3, 4],
        })
        ds.write_dataset(t2, tempdir / "t2", format="ipc")
        ds2 = ds.dataset(tempdir / "t2", format="ipc")
    
        result = ds1.join_asof(
            ds2, on="on", by=[], tolerance=1
        )
>       assert result.to_table() == pa.table({
            "on": [1, 2, 3],
            "colVals": ["Z", "Z", "B"],
        })
E       assert pyarrow.Table...null,"Z","B"]] == pyarrow.Table...["Z","Z","B"]]
E         
E         Use -v to get more diff

usr/local/lib/python3.9/site-packages/pyarrow/tests/test_dataset.py:5154: AssertionError

The useful information is unfortunately truncated by stupid pytest, but we can still see that there is a null value somewhere in the colVals result, which is probably unexpected.

Component(s)

C++, Python

[pitrou]

cc @judahrand @ZhangHuiGui @JerAguilon @rtpsw

[pitrou]

Also @raulcd

[pitrou]

The tests are new, so this is not necessarily a regression.

[jorisvandenbossche]

@pitrou you already opened an issue for this a few weeks ago, I think: #40675

(although my feeling is that it happens more often the last days than before, but maybe I am just paying more attention)

[pitrou]

It's not the same test that's failing, though the underlying cause might be the same.

[jorisvandenbossche]

Ah, I didn't actually look at the test output you posted here, sorry ;)
The ones that I have been noticing the last days was the one with the nulls in the output, so the one described in #40675

[judahrand]

From playing around with this locally a bit I can reproduce the flake. However, if I force use_threads=False in the Python code I can't. To me this suggests the failure is threading related and on the C++ side rather than the Python side.

[pitrou]

Yes, it's certainly on the C++ side since PyArrow is a wrapper.