[CI][Python] `pyarrow.lib` fails loading on AppVeyor

[pitrou]

Describe the bug, including details regarding any error messages, version, and platform.

Recently the AppVeyor CI jobs have started failing with this error:

(arrow) C:\projects\arrow\python>py.test -r sxX --durations=15 --pyargs pyarrow.tests   || exit /B 
ERROR: module or package not found: pyarrow.tests (missing __init__.py?)
============================= test session starts =============================
platform win32 -- Python 3.10.16, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\projects\arrow\python
configfile: setup.cfg
plugins: hypothesis-6.122.3
collected 0 items
============================ no tests ran in 1.43s ============================

A bit of debugging shows that its pyarrow.lib that fails loading, even though the corresponding DLL lib.cp310-win_amd64.pyd exists:

________________________ ERROR collecting test session ________________________
C:\Miniconda38-x64\envs\arrow\lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:992: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:688: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:883: in exec_module
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
C:\Miniconda38-x64\envs\arrow\lib\site-packages\pyarrow\__init__.py:65: in <module>
    import pyarrow.lib as _lib
E   ImportError: DLL load failed while importing lib: The specified module could not be found.

When one compares a successful build from 4 days before and a failing build, the only relevant differences seem to be a CMake version bump and a NumPy version bump. However, neither downgrading NumPy nor downgrading CMake fixes the problem. No change in the source tree seems like it could have caused the issue.

(I also tried downgrading LZ4, which also failed)

Component(s)

Continuous Integration, Python

[pitrou]

cc @raulcd @jorisvandenbossche @kou I'm not sure how to further debug this issue. Perhaps by remotely logging into the AppVeyor build worker and trying to find out why the DLL refuses loading?

[raulcd]

Not strictly related but there's something really wrong on the Appveyor build with the pyarrow version generated:

Creating c:\miniconda38-x64\envs\arrow\lib\site-packages\pyarrow.egg-link (link to .)
Adding pyarrow 17.0.0.dev1169+gd7fb17edc to easy-install.pth file
Installed c:\projects\arrow\python
Processing dependencies for pyarrow==17.0.0.dev1169+gd7fb17edc
Finished processing dependencies for pyarrow==17.0.0.dev1169+gd7fb17edc
(arrow) C:\projects\arrow\python>set PYTHONDEVMODE=1 

[raulcd]

Well, the previous successful build had the correct version, so it might be related:

Creating c:\miniconda38-x64\envs\arrow\lib\site-packages\pyarrow.egg-link (link to .)
Adding pyarrow 19.0.0.dev160+gea0d6f714 to easy-install.pth file
Installed c:\projects\arrow\python
Processing dependencies for pyarrow==19.0.0.dev160+gea0d6f714
Finished processing dependencies for pyarrow==19.0.0.dev160+gea0d6f714
(arrow) C:\projects\arrow\python>set PYTHONDEVMODE=1 

edit: This probably has to do with the remote fork not having the corresponding development tags, we should probably fix the version generation if tags are not present: #44803

[kou]

In general, this happens when one or more symbols are missing in linked DLLs.
Windows error message doesn't show which symbols are missing...
But Windows tell which symbols are missing by using https://en.wikipedia.org/wiki/Run_command .

BTW, LZ4 may be related.

The success build has the following log:

https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/51120420?fullLog=true#L4559

CMake Warning at cmake_modules/Findlz4Alt.cmake:29 (find_package):
  By not providing "Findlz4.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "lz4", but
  CMake did not find one.
  Could not find a package configuration file provided by "lz4" with any of
  the following names:
    lz4Config.cmake
    lz4-config.cmake
  Add the installation prefix of "lz4" to CMAKE_PREFIX_PATH or set "lz4_DIR"
  to a directory containing one of the above files.  If "lz4" provides a
  separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:313 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:2644 (resolve_dependency)
  CMakeLists.txt:546 (include)
-- Checking for module 'liblz4'
--   No package 'liblz4' found
-- Found lz4Alt: C:/Miniconda38-x64/envs/arrow/Library/lib/liblz4.lib
-- Providing CMake module for Findlz4Alt as part of Arrow CMake package

But the downgrading LZ4 log doesn't have it:

https://ci.appveyor.com/project/pitrou/arrow/builds/51141234?fullLog=true#L4364

-- Providing CMake module for Findlz4Alt as part of Arrow CMake package

lz4-c 1.10.0 may be still used.

Note that lz4-c 1.9.4 doesn't have CMake package. So we should have logs like in the success log.

[pitrou]

Not strictly related but there's something really wrong on the Appveyor build with the pyarrow version generated:

I think that's just because my fork doesn't have its tags up to date.

[pitrou]

BTW, LZ4 may be related.

The C++ tests run fine, so that sounds unlikely.

[jorisvandenbossche]

Not that this explains why it would have started failing now, but we should probably also "modernize" the way we install pyarrow here.
It's still using python setup.py develop, and then testing the installed package (i.e. pytest --pyargs pyarrow.tests and not pytest pyarrow/tests) but from within the arrow/python directory, where it might also pick up the inplace build? And @pitrou you show that the lib exists in C:\Miniconda38-x64\envs\arrow\lib\site-packages\pyarrow, but so that is not inplace? (or does the dir cmd follow the pyarrow.egg-link?)

[pitrou]

Not that this explains why it would have started failing now, but we should probably also "modernize" the way we install pyarrow here.

I can push a PR for that.