[Python] Build 0.16.0 wheel install for Windows + Python 3.5 and publish to PyPI

[asfimport]

Hi,

 

I'd like to help getting pyarrow==0.16.0 released for Windows Python 3.5.
I have seen the issue here: https://ci.appveyor.com/project/Ursa-Labs/crossbow/builds/27848124
I'm not familiar with your release build procedure, but I have been trying to follow https://github.com/apache/arrow/blob/master/docs/source/developers/python.rst#building-on-windows
Note that I have replaced:
{{conda create -y -n pyarrow-dev -c conda-forge ^
--file arrow\ci\conda_env_cpp.yml ^
--file arrow\ci\conda_env_python.yml ^
--file arrow\ci\conda_env_gandiva.yml ^
python=3.7}}
with
{{conda create -y -n pyarrow-dev -c conda-forge ^
--file arrow\ci\conda_env_cpp.yml ^
--file arrow\ci\conda_env_python.yml ^
--file arrow\ci\conda_env_gandiva.yml ^
python=3.5}}
I also had to update Cython manually and that python setup.py build_ext --inplace seemed to do nothing, so I tried doing pip wheel -w dist . and this successfully built a wheel file for me.

I understand that ursabot is down, but when it comes back could you retry building the missing wheel? It is possible that this dependency issue is already resolved.

If I am wrong about the build procedure please let me know, I'm really unfamiliar with Windows development and your build procedure.

Reporter: Mark Keller
Assignee: Wes McKinney / @wesm

Related issues:

• [Python] Add wheel build script and Crossbow configuration for Windows on Python 3.5 (is related to)

_{Note: This issue was originally created as ARROW-8015. Please see the migration documentation for further details.}

[asfimport]

Mark Keller:
Note: I have opened this issue based on: #5569

[asfimport]

Wes McKinney / @wesm:
We don't need ursabot to build the wheel. It would be good to run all the unit tests on Windows on Python 3.5 to make sure something isn't broken there. I can't look at it today but if someone else with a properly configured Windows VM or machine can kick the tires and provide a wheel someone with write access to PyPI can upload it

[asfimport]

Mark Keller:
I can't run the tests, the wheel that I have created is missing

pyarrow.lib

However; the external build is still not doing anything, I will look at this more later, but if someone could let me know why this could be that would be appreciated.

[asfimport]

Wes McKinney / @wesm:
Try cleaning out the Python directory (git clean -fdx .). See our Windows build script here

https://github.com/apache/arrow/blob/master/dev/tasks/python-wheels/win-build.bat

The ideal situation would be to build the wheel using Crossbow so we're using the exact same configuration as for the other wheels. @kszucs do you want to try?

[asfimport]

Mark Keller:
Nvm, I figured it out the compilation process and it still fails.

[asfimport]

Krisztian Szucs / @kszucs:
As far as i can remember the windows build had package conflicts since conda doesn't maintain 3.5 packages anymore.

I doubt that we can use conda-forge in building py35 windows wheels and nut sure what would be the easy workaround.

[asfimport]

Wes McKinney / @wesm:
I'll try to build the wheel locally on my Windows box

[asfimport]

Lucas Pickup:
@wesm  Did you make any progress on this? Not having a Python 3.5 Windows wheel for pyarrow is causing azureml-sdk some issues. Cheers.

[asfimport]

Wes McKinney / @wesm:
Sorry, I'm sick right now (just a cold, not covid19). I'll try to give the build a try and report back

[asfimport]

Lucas Pickup:
No worries, just wanted to know if it was gonna get uploaded imminently or if we need to switch back to depending on 0.15.*. We tried to bump to 0.16.0 for Python 3.8 support.

[asfimport]

Wes McKinney / @wesm:
I'm attempting a bundled build of all the dependencies (because conda-forge stopped supporting Python 3.5 – we had been using that to create the wheels).

[asfimport]

Lucas Pickup:
Thank you! Sounds mighty annoying to get working.

[asfimport]

Wes McKinney / @wesm:
Here is a candidate 3.5 wheel that I built locally

https://drive.google.com/open?id=1_hvgnMafWOZSgPCldNjp6iIiW3tCLwvX

It doesn't have:

• Flight

• Gandiva

• Brotli, LZ4, or ZSTD (it has Snappy and ZLIB though)

  I had to comment out some lines about bundling zlib.dll because with -DZLIB_SOURCE=BUNDLED, zlib is statically linked with arrow.dll.

  I tested out the wheel locally by running the Python test suite and it looks good to me. I'll wait about 24 hours for feedback and then upload to PyPI is all looks good

[asfimport]

Lucas Pickup:
We are mitigating the issue on our end by depending on pyarrow==0.15.\* for python<3.6.

[asfimport]

Lucas Pickup:
@wesm  What validation can I run that will help with getting the wheel to pypi? Is there a doc describing how to run the tests you ran locally?