Accessing the module object in `pyfunction`s

[birkenfeld]

In the C API, module-level functions get passed their registered module object as the self parameter. This is very nice to interact with other module level API.

Can we (optionally?) make the module reference available in pyfunctions too?

[davidhewitt]

I'm nominating this for 0.12.

[sebpuetz]

I'm taking a dive into this, although I have no idea what I'm getting into. I'll let you know if I'm getting stuck or if I don't think I'm the right person for this.

[davidhewitt]

❤️ Many thanks! I see two options:

• You can probably do a similar thing as this function we use to detect Python arguments:

  pyo3/pyo3-derive-backend/src/utils.rs

  Line 11 in 4a05f27

  pub fn if_type_is_python(ty: &syn::Type) -> bool {

  (I guess here you'll be detecting &PyModule.)

• Alternatively, maybe it's more predictable if this becomes an option to the pyfunction macro instead of trying to auto-detect:

  #[pyfunction(pass_module)] 
  fn with_module(module: &PyModule) { }
  
  // or
  #[pyfunction]
  #[pyo3(pass_module)] 
  fn with_module(module: &PyModule) { }
  

I think I slightly prefer the #[pyfunction(pass_module)] design. All opinions welcome.

[davidhewitt]

Note: you might also need to consider #[pyfn] for the same functionality. One day I hope we merge those two things!

[sebpuetz]

I had a first go at the implementation, and we need to make some changes to the ffi-calls for function creation.

Currently the slf parameter in #[pyfunction]s is always a null-pointer. This happens because the second parameter passed to PyCFunction_NewEx is always null:

pyo3::ffi::PyCFunction_New(
                    Box::into_raw(Box::new(_def.as_method_def())),
                    ::std::ptr::null_mut(),
                ),

[...]
pub unsafe fn PyCFunction_New(ml: *mut PyMethodDef, slf: *mut PyObject) -> *mut PyObject {
    #[cfg_attr(PyPy, link_name = "PyPyCFunction_NewEx")]
    PyCFunction_NewEx(ml, slf, ptr::null_mut())
}

This is the signature of PyCFunction_NewEx:

PyObject *
PyCFunction_NewEx(PyMethodDef *ml, PyObject *self, PyObject *module)
{
    return PyCMethod_New(ml, self, module, NULL);
}

So we're setting self explicitly to NULL for pyfunction/pyfn. In order to change this, we need to get a pointer to the module object into the fn __pyo3_get_function_X. Steps required to do this are:

1. add pub fn add_function(&self, wrapper: &impl Fn(Python, &Self) -> PyObject) -> PyResult<()> on PyModule
   • add_wrapped could probably also be amended this way, so we wouldn't need to introduce a new function.
2. change code-generation for get_function to also pass the module (Might as well eliminate the Python param at this point)
   • new signature for wrapped fns: fn __pyo3_get_function_NAME(py: Python, m: &PyModule) -> pyo3::PyObject
3. replace std::ptr::null_mut() with m.as_ptr()
4. extract module from _slf arg in wrap: let _mod = _py.from_borrowed_ptr::<pyo3::types::PyModule>(_slf);
5. call the pyfunction with the additional module parameter

---

PyFunction_NewEx also has a module parameter that seems to generally be set to a string with the module name, it gets exposed as the __module__ attribute on functions and is part of the repr of function objects.

The above steps also provide the facilities to set this attribute, all that's left at that point is to get the module name from the self pointer in the get_function wrapper:

let name = unsafe { pyo3::ffi::PyModule_GetNameObject(m.as_ptr()) };

And passing the method_def, the module pointer and the name object to PyCFunction_NewEx instead of PyCFunction_New.

---

I have only implemented this manually based on expanded code, I haven't touched the proc-macros yet. I hope to find some time to start writing a PR later today or tomorrow!

[sebpuetz]

It looks like a lot of tests rely on "free" functions, i.e. not associated with any PyModule. In that case we can't pass a PyModule to the get_function wrapper. I feel like such functions are not the norm and mostly show up in special circumstances like tests. I'm not sure how we'd even be able to call a function in Python that's never been added to a module.

So, what do you think about adding a free attribute on pyfunction to toggle the PyModule param?

[sebpuetz]

I don't think that we can apply the same heuristics used for Python to detect when a &PyModule is wanted. Python can never be an actual argument that's passed to a pyfunction, &PyModule on the other hand is a regular Python type that can be passed around.

To fix this is, we could require that the first argument on a non-free pyfunction is always &PyModule. That would add an unused argument for almost all pyfunctions, so I'm not sure how popular that idea would be. On the other hand, it'd actually resemble how things are implemented in CPython.

The other way I see is to look for a clean way to communicate which &PyModule is the one that's requested inside the fn body. A naive implementation could just be something like pyfunction(module=moduleParam). But I think this can cause collisions with the way fn-args are currently parsed. This could be solved by moving to a nested variant:

• pyfunction(module(moduleParam)) to signal that the argument moduleParam is the one that should be passed as the modle
• pyfunction(free) to signal that the function does not belong to any module
• pyfunction for the default where no module gets passed.

I'll probably get around to open a WIP PR tonight which includes the FFI & proc-macro changes up to this point. The implementation of passing the PyModule to the fn is pretty much all that's missing.