PyEval_InitThreads is not called in `#[pymodule]` for Python3.6

[andylokandy]

🐛 Bug Reports

I was writing an extension module (python call rust) using pyo3 and recently upgraded the pyo3 dependency from 0.11 to 0.14.

Then once thePython::with_gil() was called, the pyo3 would panic:

thread 'tokio-runtime-worker' panicked at 'assertion failed: `(left != right)`
  left: `0`,
 right: `0`: Python threading is not initalized and the `auto-initialize` feature is not enabled.

Consider calling `pyo3::prepare_freethreaded_python()` before attempting to use Python APIs.', /home/andylokandy/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.14.4/src/gil.rs:232:21

Then I added the auto-initialize feature flag and also tried to add pyo3::prepare_freethreaded_python() to the first line in #[pymodule] function without auto-initialize feature flag, it still panicked:

thread '<unnamed>' panicked at 'assertion failed: `(left != right)`
  left: `0`,
 right: `0`', /home/andylokandy/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.14.4/src/gil.rs:84:13

After investigating a little bit, I believe that the latter panic in gil.rs:84:13 is caused by the uninitialized threading while the interpreter is already initialized by #[pymodule].

The PyEval_InitThreads is called in #[pymodule] before #1630, in which @davidhewitt commented that PyEval_InitThreads is not necessary which I think is not true because PyEval_InitThreads is required to be called before Python3.7 (https://docs.python.org/3/c-api/init.html#c.PyEval_InitThreads).

After manually calling PyEval_InitThreads in #[pymodule], pyo3 no longer panics.
https://github.com/tikv/client-py/blob/6225c2ffcafbbed70dde3cf9efe3460a5e34d84e/src/lib.rs#L14-L16

🌍 Environment

• Your operating system and version: Ubuntu 2004
• Your python version: 3.6.9
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: apt, yes
• Your Rust version (rustc --version): rustc 1.53.0-nightly (16bf626a3 2021-04-14)
• Your PyO3 version: 0.14.4
• Have you tried using latest PyO3 main (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?: not yet

💥 Reproducing

Not minimized the project yet but I've found the root cause.

[davidhewitt]

Thanks for the report.

I defend that it is correct to not initialize threads as part of #[pymodule], however I have opened #1874 to fix the panic in with_gil when using auto-initialize or prepare_freethreaded_python.

[andylokandy]

This comment may need to be updated:

pyo3/src/gil.rs

Lines 55 to 58 in 8ffdbd5

	/// # Panics
	/// - If the Python interpreter is initialized but Python threading is not, a panic occurs.
	/// It is not possible to safely access the Python runtime unless the main thread (the thread
	/// which originally initialized Python) also initializes threading.

[davidhewitt]

👍 good spot, thanks!

[Congyuwang]

Just got the same problem in python 3.6. Is there any expectation when this fix might appear in crate release? A lot of thanks.

[Congyuwang]

Also solved by manually calling PyEval_InitThreads for now:

#[pymodule]
 fn xxxx(_py: Python, m: &PyModule) -> PyResult<()> {
     unsafe {
         pyo3::ffi::PyEval_InitThreads();
     }
     m.add_class::<XXXX>()?;
}

[davidhewitt]

Just got the same problem in python 3.6. Is there any expectation when this fix might appear in crate release? A lot of thanks.

#1921