Expose PyCapsule API

[davidhewitt]

The capsule API (https://docs.python.org/3.8/c-api/capsule.html) can be useful for safely storing Rust data in a way that's not usable by Python without leaking it. This is possible because the capsule creation takes a destructor function.

Today it came up on Gitter that for rust-numpy it may be useful to do zero-copy transfer of arrays from Rust to Python, by stashing the backing storage (e.g. vec) in a capsule which is set as the base object to the numpy array.

[kngwyu]

Today it came up on Gitter that for rust-numpy it may be useful to do zero-copy transfer of arrays from Rust to Python, by stashing the backing storage (e.g. vec) in a capsule which is set as the base object to the numpy array.

But isn't casule for exposing C-APIs, just like numpy does so?

[birkenfeld]

That's one use, but it's more general (from the docstring):

Capsule objects let you wrap a C \"void *\" pointer in a Python
object.  They're a way of passing data through the Python interpreter
without creating your own custom type.

[davidhewitt]

Yep exactly. Numpy uses it to share pointers to api functions; we can stash whatever we want in capsules 😄

[messense]

The cpython crate has already exposed the PyCapsule API, we can take inspiration from it: https://docs.rs/cpython/0.6.0/cpython/struct.PyCapsule.html

[davidhewitt]

Perhaps. Some things we might be able to extend about that API:

• If we want to use it to give Python ownership of a value, we probably want to use Box<T> and Box::into_raw. But we need to be a little careful with trait objects, because *mut dyn Any is actually a fat pointer, whereas capsules only store a single pointer.
• It would be nice to be able to customise the destructors, as a way to make a callback when a Python object is destroyed.

I was imagining we'd want an api roughly like this:

impl PyCapsule {
    pub fn new(py: Python, value: Box<T>, name: CString) -> &Self;
    
    pub fn new_with_static(py: Python, value: &'static T, name: CString) -> &Self;
    
    pub fn new_with_destructor(py: Python, value: Box<T>, name: CString, destructor: impl FnOnce(Box<T>)) -> &Self;

    pub fn name(&self) -> CString;
    
    pub fn pointer(&self, name: &CStr) -> *mut std::os::raw::c_void;
}

Usage of &CStr / CString can be a little annoying for the user but it's the most direct translation of the underlying APIs. Maybe we can be elegant and work with &str / String but I suspect that may add additional overhead. (Unsure if that matters.)

[davidhewitt]

For reference I've also used the pybind11 capsule API in the past:

https://github.com/pybind/pybind11/blob/b5357d1fa8e91ddbfbc2ad057b9a1ccb74becbba/include/pybind11/pytypes.h#L1278