nosetests not running on windows

[flothesof]

I've been trying to get the tests running on Windows (Python 3.6 conda environment for tofu testing).
I've seen two problems so far.

Path to data files used by tests

It appears that the way the path to the data files is handled breaks on my setup.
If I replace _root = _here[:_here.rfind('/tofu')] in tofu/geom/utils.py by _path_testcases = os.path.join(os.path.split(_here)[0], 'inputs'), tests manage to find the correct path to data files.

Cython problems (?)

However, I then break on what I think are Cython related errors:

Error
Traceback (most recent call last):
  File "C:\Anaconda3\envs\test_tofu\lib\unittest\case.py", line 59, in testPartExecutor
    yield
  File "C:\Anaconda3\envs\test_tofu\lib\unittest\case.py", line 605, in run
    testMethod()
  File "C:\Anaconda3\envs\test_tofu\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\Users\FL232714\Documents\Python workspace\Scripts\2019_conda packaging\tofu\tofu\tests\tests01_geom\tests01_GG.py", line 882, in test13_LOS_PInOut
    ves_type='Lin', test=True)
  File "tofu\geom\_GG03.pyx", line 1821, in tofu.geom._GG03.LOS_Calc_PInOut_VesStruct
ValueError: Buffer dtype mismatch, expected 'long' but got 'long long'

I'll investigate to see if I can go further (and then propose a PR).

[Didou09]

Thanks a lot !

[flothesof]

It seems that other people have had the same problem.

Here's a link to how they fixed it:
mdtraj/mdtraj@46bf143

Looks like this needs some more investigation.

[flothesof]

One of the functions that raises the error is _Ves_Smesh_Lin_SubFromInd_cython, found in _GG03.pyx.

I've tried to put a long long type for the ind variable in the signature of

def _Ves_Smesh_Lin_SubFromInd_cython(double[::1] XMinMax, double dL, double dX,
                                     double[:,::1] VPoly, np.ndarray[long,ndim=1] ind,
                                     double DIn=0., VIn=None, double margin=_VSMALL):

but every time I run python setup.py build_ext --inplace, the code gets replaced by its original signature.
Any hints on why I can't manage to modify the cython?

[lasofivec]

You have to change it in the _GG02.pyx file, the _GG03.pyx is only a copy of the GG02 to ensure that the python2 and python3 versions remain the same :)

[lasofivec]

However, I think it should be better to force the type of all longs and/or ints at the beginning of the file (as done in the solution @flothesof linked to) is better than changing the signatures of the function (?)

[flothesof]

Thanks for explaining the copy mechanism. I'll see if I can get it to work following the solution mdtraj used and I'll report back here.

[Didou09]

Yes, I had forgotten to explain, thanks Laura.
_GG03.pyx is a copy of _GG02.pyx produced when the setup.py is run, never edit _GG03.pyx, the changes would be lost at the next copy.

I agree that forcing all types at the beginning of the file is a good options
Thanks Flo, waiting for your changes

(if you create a local branch from devel, please also push it here on Github so we'll keep track of it and keep a common trace after it is deleted).

[flothesof]

After trying the brute force solution (replacing all longs by int64), I am now able to run 85/114 tests on my test environment.
However, there are still errors related to this problem. I'm still trying to understand the problems. One observation I have is based on code where the variable is first declared and then initialized with a numpy array of integer datatype (from _Ves_meshCross_FromD here):

cdef np.ndarray[int64_t,ndim=1] ind
[...]
ind = np.empty((n1*n2,),dtype=int)

This leads to errors of the type Buffer dtype mismatch, expected 'int64_t' but got 'long'.

So brute force replacement is not enough...

[lasofivec]

From this SO answer:

Windows for both 32 and 64 bit uses 4 bytes (32 bits) for int and long, 8 bytes (64 bits) for long long. The difference between Win32 and Win64 is size of pointer (32 bits for Win32 and 64 bits for Win64). (See Data Type Ranges] from MSDN).

Linux uses another model: int is 32 bits for both linux-32 and linux-64, long long is always 64-bit. long and pointers are vary: 32 bits on linux-32 and 64 bits on linux-64.

I think we could be using long long everywhere where we use long, I created a branch to test this: long2longlong as I dont have a Windows machine, I cannot try it, would you mind testing it for me ?

[flothesof]

Hi all,
now retrying with Laura's long2longlong branch.
Here's the log of actions I performed.

• conda create -n py36tofu python=3.6 pip cython numpy
• conda activate py36tofu
• git clone https://github.com/ToFuProject/tofu.git
• git checkout long2longlong
• pip install -e .[test] with some modifications of setup.py (the openmp check on line 70 fails, so I had to avoid it and write not_openmp_installed = True)
• nosetests from the current directory and having fixed _path_testcases = os.path.join(os.path.split(_here)[0], 'inputs')

I get the following

----------------------------------------------------------------------
Ran 115 tests in 55.602s

FAILED (errors=22, failures=8)

I'm attaching the nosetest log to the issue for further analysis.
test_report.txt

I believe some of the errors are related to the longlong changes.

What do you think we should do?