Confusing APIs around PyObject

[vadimcn]

I started learning PyO3 a couple of days ago, and for the longest time I was confused by the fact that PyObject represents a "free-floating" reference to a Python object, whereas the test of Py<thing>'s represent concrete Python types that are only accessible with 'py lifetime.

It might be too late now to change all that, but... it would have been much easier to grok PyO3's API structure if PyObject and PyObjectRef meanings were swapped. Furthermore, I would suggest moving PyObject's convenience methods to Py and making PyObjectRef just an alias for Py<PyObject>.

At the very least, all this should have been explained somewhere close to the top of the user guide.

[kngwyu]

Thanks for reporting!
I think you misunderstand Py.
Please see the guide about it.

However, I completely agree with things around PyObject, PyObjectRef and so on are really confusing(in addition, now we have PyRef types!).
So anyway I think we should simplify our APIs.
And it sounds good to make PyObjectRef just an alias.

[kngwyu]

I changed your issue title but please feel free to change it again if you dislike it.

[konstin]

I agree that PyObject is very confusing. Unfortunately it's difficult to at least rename PyObject to something more reasonable because there's ffi::PyObject (which make the situation even more confusing).

Afaik we could replace PyObject with Py<PyObjectRef> and maybe add alias like PyStored for convenience, but that would be a huge refactoring.

[konstin]

I think we can avoid much of the confusion without changing the api by explaining the types properly in the guide.

[vadimcn]

Well, the crate is still pre-1.0, but I agree, renaming PyObject now would break everybody's code.
Still, I would suggest to at least consider renaming PyObjectRef to something like PyBaseObject.

Regarding PyRef: the distinction between built-in and user-defined Python objects seems like another wart in crate API. Why not put native objects behind PyRef as well? Also, calling it PyBox would be more intuitive, IMO.

[jothan]

With the latest 0.6.0-alpha.4, I'm having trouble with storing references to other Python instances from within a pyclass. What is the proper type to use inside structs now ? With the current situation, I'm having trouble upgrading &self or &mut self to Py<Self>. Does the new API account for this ?

Here is the code that broke.

[kngwyu]

@jothan
Thanks for your feedback.
It looks we need to provide some more APIs.
Please wait for a few days...

[jothan]

Going forward, all types used to reference Python objects should probably be reduced or documented thoroughly.

From memory I can name Py, PyRef, PyRefMut, PyObjectRef and PyObject. This is too much for me. Working with PyO3, I found that I spent most of my time figuring out how to convert object references.

Once this is sorted out, I would like to update the docs with a matrix stating:

• Each type and its precise purpose.
• How each type behaves with regards to the GIL and lifetimes.
• How #[pyclass] structs fit into all this.
• How to convert a type into all the others when applicable.

[pganssle]

I can't agree with this more strongly, I'd love to see a re-working of the API to be more clear about this and I'd be happy to chime in.

See for example this StackOverflow question, which is characteristic of my experience with PyO3. From the Rust side, I think a lot of this is clearer, but from the Python side there seem to be many, many ways to return values to Python, all of which seem to work from the Python side, but where I am not clear on the differences.

[pganssle]

Also, as I noted in this comment, but I guess never created an issue for, the RAII approach for GILGuard is very misleading, because of the way the GIL works.

It's probably worth taking a look at that in a separate issue, but essentially the lifetime of GILGuard is not the lifetime of the GIL, which can and probably is released and re-acquired many times over the course of a given Python operation.

[konstin]

I agree that the api of pyo3 is overly complex and poorly documented. That being said I'd be very happy about pull request improving the api, the guide and/or the docstrings. E.g. having a matrix explaining all the different types and their usage would be great. The wasm-bindgen book should also give some inspiration.

Something that's related to the wrapper types are the conversions between them. Most of pyo3's current conversions traits ([In]ToPyObject, [In]ToPyPointer, FromPyObject) could be replaced by a generic FromPy<T>/IntoPy<T> pair that works like std's From<T>/Into<T>. I've added IntoPy in an earlier pull request and will add FromPy in #367, so if anyone interested it should be possible to remove one of those traits with medium effort.

[vadimcn]

That being said I'd be very happy about pull request improving the api, the guide and/or the docstrings.

I'm sorry, but conceptual API design is the job of crate owners. If you allow pull requests from random strangers to decide that for you, nothing good will come out of your project.

---

If you want my opinion, I would suggest deciding between:

1. Representing "Python object with GIL held" as some wrapper type, say PyGIL<T>, and exposing all object methods of all types (including Python built-ins) on that wrapper.
2. Somehow ensuring that user-defined types cannot be constructed outside your control, if which case you can make sure no one can obtain &UserDefinedClass without locking the GIL. Perhaps this could be done by injecting a non-user-constructible marker field as a part of #[pyclass] expansion.

Then stick to that approach. Using both at the same time is unpalatable.

[pganssle]

I'm sorry, but conceptual API design is the job of crate owners. If you allow pull requests from random strangers to decide that for you, nothing good will come out of your project.

Obviously it's the crate owner's job to make decisions about the final outcome, but I think it's a good idea to encourage contributors to make their opinions known, and as specifically as possible, and to work to implement their ideas.

I very frequently will make pull requests that I have no intention of getting merged, just as a proof of concept of what my proposed API would look like in situ.

Personally, if I understood the current API well enough to change it, I'd be happy to make a PR to improve the API.