class SliceBox initialization panic with PyO3 0.7

[rth]

After I upgraded my project in rth/vtext#56 to use rust-numpy 0.6.0 (and PyO3 0.7), I started to see panics in python tests with the following message,

thread '<unnamed>' panicked at 'An error occurred while initializing class SliceBox'

that is raised in pyo3 here

full backtrace below,

Details

tests/test_vectorize.py::test_count_vectorizer thread '<unnamed>' panicked at 'An error occurred while initializing class SliceBox', C:\Users\Administrator\.cargo\registry\src\github.com-1ecc6299db9ec823\pyo3-0.7.0\src\type_object.rs:260:17
stack backtrace:
   0: std::sys::windows::backtrace::set_frames
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\sys\windows\backtrace\mod.rs:94
   1: std::sys::windows::backtrace::unwind_backtrace
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\sys\windows\backtrace\mod.rs:81
   2: std::sys_common::backtrace::_print
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\sys_common\backtrace.rs:70
   3: std::sys_common::backtrace::print
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\sys_common\backtrace.rs:58
   4: std::panicking::default_hook::{{closure}}
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\panicking.rs:200
   5: std::panicking::default_hook
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\panicking.rs:215
   6: std::panicking::rust_panic_with_hook
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\panicking.rs:478
   7: std::panicking::continue_panic_fmt
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\panicking.rs:385
   8: std::panicking::begin_panic_fmt
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\panicking.rs:340
   9: <T as pyo3::type_object::PyTypeObject>::init_type::{{closure}}
  10: <numpy::slice_box::SliceBox<T>>::new
  11: <numpy::array::PyArray<T, D>>::from_boxed_slice
  12: <ndarray::ArrayBase<ndarray::OwnedRepr<A>, D> as numpy::convert::IntoPyArray>::into_pyarray
  13: <T as pyo3::type_object::PyTypeObject>::init_type::{{closure}}
  14: <T as pyo3::type_object::PyTypeObject>::init_type::{{closure}}
  15: PyMethodDef_RawFastCallKeywords
  16: PyMethodDef_RawFastCallKeywords
  17: PyEval_EvalFrameDefault
  18: PyMethodDef_RawFastCallKeywords
  19: PyEval_EvalFrameDefault
  20: PyFunction_FastCallDict
  21: PySlice_New
  22: PyEval_EvalFrameDefault
  23: PyEval_EvalCodeWithName
  24: PyFunction_FastCallDict
  25: PySlice_New
  26: PyEval_EvalFrameDefault
  27: PyEval_EvalCodeWithName
  28: PyMethodDef_RawFastCallKeywords
  29: PyEval_EvalFrameDefault
  30: PyEval_EvalCodeWithName
  31: PyMethodDef_RawFastCallKeywords
  32: PyEval_EvalFrameDefault
  33: PyMethodDef_RawFastCallKeywords
  34: PyEval_EvalFrameDefault
  35: PyEval_EvalCodeWithName
  36: PyFunction_FastCallDict
  37: PyObject_Call_Prepend
  38: PyType_FromSpecWithBases
  39: PyObject_FastCallKeywords
  40: PyMethodDef_RawFastCallKeywords
  41: PyEval_EvalFrameDefault
  42: PyMethodDef_RawFastCallKeywords
  43: PyEval_EvalFrameDefault
  44: PyFunction_FastCallDict
  45: PySlice_New
  46: PyEval_EvalFrameDefault
  47: PyEval_EvalCodeWithName
  48: PyMethodDef_RawFastCallKeywords
  49: PyEval_EvalFrameDefault
  50: PyEval_EvalCodeWithName
  51: PyMethodDef_RawFastCallKeywords
  52: PyEval_EvalFrameDefault
  53: PyMethodDef_RawFastCallKeywords
  54: PyEval_EvalFrameDefault
  55: PyEval_EvalCodeWithName
  56: PyFunction_FastCallDict
  57: PyObject_Call_Prepend
  58: PyType_FromSpecWithBases
  59: PySlice_New
  60: PyEval_EvalFrameDefault
  61: PyEval_EvalCodeWithName
  62: PyMethodDef_RawFastCallKeywords
  63: PyEval_EvalFrameDefault
  64: PyEval_EvalCodeWithName
  65: PyMethodDef_RawFastCallKeywords
  66: PyEval_EvalFrameDefault
  67: PyEval_EvalCodeWithName
  68: PyFunction_FastCallDict
  69: PySlice_New
  70: PyEval_EvalFrameDefault
  71: PyEval_EvalCodeWithName
  72: PyMethodDef_RawFastCallKeywords
  73: PyEval_EvalFrameDefault
  74: PyEval_EvalCodeWithName
  75: PyMethodDef_RawFastCallKeywords
  76: PyEval_EvalFrameDefault
  77: PyFunction_FastCallDict
  78: PySlice_New
  79: PyEval_EvalFrameDefault
  80: PyEval_EvalCodeWithName
  81: PyMethodDef_RawFastCallKeywords
  82: PyEval_EvalFrameDefault
  83: PyEval_EvalCodeWithName
  84: PyMethodDef_RawFastCallKeywords
  85: PyEval_EvalFrameDefault
  86: PyMethodDef_RawFastCallKeywords
  87: PyEval_EvalFrameDefault
  88: PyEval_EvalCodeWithName
  89: PyFunction_FastCallDict
  90: PyObject_Call_Prepend
  91: PyType_FromSpecWithBases
  92: PyObject_FastCallKeywords
  93: PyMethodDef_RawFastCallKeywords
  94: PyEval_EvalFrameDefault
  95: PyFunction_FastCallDict
  96: PySlice_New
  97: PyEval_EvalFrameDefault
  98: PyEval_EvalCodeWithName
  99: PyMethodDef_RawFastCallKeywords
Windows fatal exception: code 0xc000001d

I think it must be happening in this function that mostly does something along the lines of,

ndarray::arr1(..).mapv(|elem| elem as i32).into_pyarray(py).to_owned()

(sorry for not providing a more self-contained example).

Interestingly this happens somewhat erratically. A week or so ago, this error was consistently happening on Windows builds while Linux and MacOS builds were fine. Now Windows builds pass, and Linux build fails.

I managed to reproduce this on a separate windows machine, but only when building wheels (i.e. this error did not happen when running python setup.py develop).

I'm not sure if this is an issue with rust-numpy, Pyo3 or if I'm doing something wrong.

[kngwyu]

Thanks for reporting!
On the branch you referred, I executed

python setup.py develop
pytest -sv --pyargs vtext

and couldn't reproduce the error.

Could you provide an instruction to reproduce the panic?
I'm usually on Linux(Ubuntu 18.04 or Arch) but can use Windows for debugging.

Edited: I used Ubuntu18 and Python3.6 for the test.

[rth]

Thanks for checking @kngwyu !

The issue doesn't seem to happen when using python setup.py develop but can be reproduced with,

python setup.py bdist_wheel
pip install --pre --force-reinstall dist/*whl
cd ..
pytest -sv --pyargs vtext

I can reproduce it also on Linux in this Docker container.

(I find that installing pytest-faulthandler is quite useful for debugging )

Edit: actually, I can also reproduce it with simply python setup.py install. I have seen this happening on python 3.5 to 3.7, so I don't think the python version matter.

[kngwyu]

Hoops I'm still failing to reproduce...

Test script that I run in the container is

rustup default nightly
pip3 install setuptools-rust pytest pytest-faulthandler
cd /src/python
python3 setup.py bdist_wheel
pip3 install --pre --force-reinstall dist/*whl
cd ..
pytest -sv --pyargs vtext

[rth]

Previously I was using nightly-2019-02-28, and upgrading to the latest nightly-2019-06-04 I can indeed no longer reproduce it. So maybe something not quite stable in the compiler itself. Closing.

Thanks for investigating @kngwyu !

[clbarnes]

I am getting this error again (pattonw/rust-pyn5#93), using nightly-2019-09-17. It only breaks in release builds, not development builds. The workaround was to use an older nightly compiler (I went with the oldest pyo3 is compatible with). It looks like this is something which may show up again until pyo3 no longer requires a nightly compiler ( PyO3/pyo3#5 , rust-lang/rust#31844 ).

[kngwyu]

Thank you, it's terrible 😱

[NotSqrt]

Hello,

If it helps, I had the same exception with 1.40 nightly when compiling in release mode.
Simply adding a println of the array before .into_pyarray(py) mitigated the issue...

I've just ran rustup toolchain update nightly and it seems the current nightly version 1.41.0-nightly (ded5ee001 2019-11-13) does not exhibit the problem.

[clbarnes]

Thanks! Good to keep track of which compilers work as new ones come out.

[amb]

Trying to build the example project for rust-numpy and run the Python test file (Write a Python module in Rust). Setting to various nightly versions didn't work for me on Windows 10.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 49: invalid continuation byte
thread '' panicked at 'An error occurred while initializing class SliceBox', C:\Users\hyppa.cargo\registry\src\github.com-1ecc6299db9ec823\pyo3-0.8.5\src\type_object.rs:264:17

Setting [profile.release] opt-level=1 works, opt-level=2 does not. Might be able to live with it or maybe someone can figure out what particular optimization triggers the bug.