[R][Python] Unable to load pyarrow package through reticulate in R on windows

[aquasync]

Describe the bug, including details regarding any error messages, version, and platform.

I'm trying to use arrow to share large data frames between python and R. If I load the R packages then try to import the python module this fails with an ImportError from python:

> library(arrow)
The tzdb package is not installed. Timezones will not be available to Arrow compute functions.

Attaching package: ‘arrow’

The following object is masked from ‘package:utils’:

    timestamp

> library(reticulate)
> py_discover_config()
...
version:        3.9.18 | packaged by conda-forge | (main, Dec 23 2023, 16:29:04) [MSC v.1929 64 bit (AMD64)]
Architecture:   64bit
...
> pa <- import("pyarrow")
Error in py_module_import(module, convert = convert) : 
  ImportError: DLL load failed while importing lib: The specified procedure could not be found.
Run `reticulate::py_last_error()` for details.

Note that I can load the pyarrow in a vanilla python REPL just fine.

If I ensure to import pyarrow before loading the R package then it works, however this is difficult to guarantee (other packages may have already loaded the R arrow package), and causes failures during package development.

> library(reticulate)
> import('pyarrow')
Module(pyarrow)
> library(arrow)
The tzdb package is not installed. Timezones will not be available to Arrow compute functions.

Attaching package: ‘arrow’

The following object is masked from ‘package:utils’:

    timestamp

Because it seems to be driven by some sort of incompatibility between the R and python arrow libraries, I tried various different versions to see if I could get anything that works. Initially I was using R 4.2.1, but could also reproduce with a fresh install of R 4.3.2. I tried various python versions including fresh installs of the latest (3.11.5), and also the conda version as installed by reticulate::install_miniconda (3.9.18 above). I tried using the latest version of the pyarrow package (15.0.0), and also downgrading to 14.0.0 to match the latest available R package.

Component(s)

Python, R

[aquasync]

I've attempted to diagnose it by loading the dll directly as follows:

> library(arrow)
...
> reticulate::repl_python()
Python 3.12.2 (C:/Program Files/Python312/python.exe)
Reticulate 1.35.0 REPL -- A Python interpreter in R.
Enter 'exit' or 'quit' to exit the REPL and return to R.
>>> import ctypes as c
>>> dll = c.WinDLL(r'C:\Program Files\Python312\Lib\site-packages\pyarrow\lib.cp312-win_amd64.pyd')
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\PROGRA~1\PYTHON~2\Lib\ctypes\__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 127] The specified procedure could not be found

That produces a pop up with the mangled C++ symbol name that caused the issues, in this case eg: "The procedure entry point ?type@StringBuilder@arrow@@UEBA?AV?$shared_ptr@VDataType@arrow@@@std@@XZ could not be located in the dynamic link library C:\Program Files\Python312\Lib\site-packages\pyarrow\lib.cp312-win_amd64.pyd".

If I repeat the above without first loading the R arrow package, it will load the python libs just fine. Is this problem caused by both the R and python packages having different "arrow.dll" files that they're attempting to load into the same address space?

[aquasync]

After some more digging, I think that the cause is indeed the two "arrow.dll" files. Loading the python package first works because the R "arrow.dll" is loaded directly, whereas the pyarrow version is loaded indirectly by being dependencies of the .pyd files.

I tried a hacky solution of renaming the python one to "xrrow.dll", and running sed -i.bak 's/arrow.dll/xrrow.dll/g' on the .pyd files in the directory, and it now "works" when loading the R package first:

> library(arrow)
> library(reticulate)
...
> pa = import('pyarrow')
> pa
Module(pyarrow)

Perhaps the easiest solution then is for the R library to use a different name for its dll, though I'm not sure if it has to match the package name or if that is just a convention? Alternatively perhaps dll manifests can be used on the python side.

[aquasync]

I was able to get a manifest solution to work, following https://devblogs.microsoft.com/oldnewthing/20171011-00/?p=97195.

First I created "force_local_arrow.manifest" with the following:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
    <file name="arrow.dll" />
</assembly>

Then I patched each of "arrow_flight.dll", "arrow_python.dll", "lib.cp312-win_amd64.pyd", and "_hdfsio.cp312-win_amd64.pyd" (probably incomplete, just enough to load the package) as follows:

> mt.exe -manifest force_local_arrow.manifest -outputresource:lib.cp312-win_amd64.pyd;#2

Perhaps this approach is better than just renaming the R dll; by amending the build process for the windows python wheel to include the necessary flags to embed the manifest and avoid the problem altogether.

[amoeba]

Hi @aquasync, thanks for the report and your work getting to a cause and fix. I can reproduce what you saw in your original report on my Windows ARM machine so I can offer that as a start here. I'll work on getting some more eyes on this.

[annakrystalli]

If it's of any use/related, this message just started showing up in some of our R CMD CHECK github actions of an R package that has arrow as a dependency. In tests of a function that uses arrow to read csv files (without loading the package) we now get a new message:

 Message
 The tzdb package is not installed. Timezones will not be available to Arrow compute functions.

only on windows, causing comparison against our previous snapshots to fail because of the message: https://github.com/Infectious-Disease-Modeling-Hubs/hubValidations/actions/runs/8083994354/job/22088275404?pr=74

The last time the action was run and passed was Feb 13.

Let me know if you think this should be moved into a separate issue

[amoeba]

Hi @annakrystalli, I don't think that's related but feel free to file a new issue and tag me. It doesn't look like anything related to that code has changed recently so my guess is that something in your environment may have changed. tzdb is a Suggests and getting this warning is expected on Windows.

[amoeba]

Hey @aquasync, I did a little bit more research on this today and have a few notes:

I found this older Jira ticket that seemed related. That led me to try installing R arrow from conda instead of CRAN and I was able to get things to work:

$ conda create -n reticulate-test -c conda-forge python==3.12 r-reticulate r-arrow pyarrow
$ conda activate reticulate-test
$ R
> library(arrow)
> library(reticulate)
> import("pyarrow")
Module(pyarrow)

So that's possibly a workaround in case you need one right now.

As for next steps, it seems like we can modify either the Python build or the R build here and my preference would be to modify the R build as it would be relatively low impact and we know it already works because of our conda package. And we'd only have to modify it for Windows. @assignUser do you have any immediate reaction or insights into this idea?

Before I figured the above out, I filed an issue over on the reticulate repo to see if they have any ideas so I'll be curious to see if they have any thoughts.

[aquasync]

Ah ok, interesting, thanks for investigating. I did try a conda install but that was just for python & python packages, not R.

I'm just using a modified wheel in the interim with the manifest attached to all the dlls, but if the solution of renaming the R dll is already being used in the conda context it makes sense to just use that for default builds.