Skip to content

store Bound<T> inside PyRef and PyRefMut#3860

Merged
davidhewitt merged 4 commits intoPyO3:mainfrom
davidhewitt:pyref-bound-internals
Feb 27, 2024
Merged

store Bound<T> inside PyRef and PyRefMut#3860
davidhewitt merged 4 commits intoPyO3:mainfrom
davidhewitt:pyref-bound-internals

Conversation

@davidhewitt
Copy link
Member

Part of #3684, split from #3606

This first commit in this PR rewrites the internals of PyRef and PyRefMut so that they store Bound<T> instead of the gil-ref &PyCell<T>. This incurs a necessary reference counting overhead where there wasn't before.

This is necessary step to get lifetimes to wire up successfully for FromPyObject implementations for PyRef and PyRefMut to use the Bound APIs. In extract_bound, while we only have the one lifetime on FromPyObject (for compatibility reasons) we cannot borrow from the input &Bound. So we are forced to take ownership of a new reference. This could be removed again in future when we add the lifetime to FromPyObject.

The second commit makes the changes to the FromPyObject implementations, which required loosening of lifetimes on Bound::borrow and similar methods.

Overall I don't love where this gets to in terms of implementation, so suggestions are welcome and I'll be glad to clean this up as soon as there's a better solution.

@davidhewitt davidhewitt added the CI-skip-changelog Skip checking changelog entry label Feb 18, 2024
}

fn get_cell(&'py self) -> &'py PyCell<T> {
pub(crate) fn get_cell(&'py self) -> &'py PyCell<T> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be &'py self or could it be &self?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh, so it could probably just be &self and things would work but the problem is that because PyCell<T> is a GIL Ref, I want to respect that still and keep the 'py very formal for now. This is exactly part of what I mean by the implementation is a bit messy.

I think that to solve this we might need to split PyCell into the public GIL Ref type, and a private inner PyCell (or will a different name like PyCellObject) and change all this stuff to work off the private inner type. But I'm not completely decided on that.

@codspeed-hq
Copy link

codspeed-hq bot commented Feb 18, 2024

CodSpeed Performance Report

Merging #3860 will not alter performance

Comparing davidhewitt:pyref-bound-internals (05208aa) with main (cd1c0db)

Summary

✅ 67 untouched benchmarks

Copy link
Member

@Icxolu Icxolu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've played around with this a bit yesterday. I agree with you that we would need the second lifetime to avoid the ownership requirement. That's a bit sad, but I guess it has to do for now.

@davidhewitt
Copy link
Member Author

Thanks @LilyFoote @Icxolu for the reviews. I've finally come back to this and tidied it up.

I stalled for a bit both because I was busy and because I was wondering about going ahead with the greater refactor of splitting PyCell up into internals and public GIL Ref as per #3860 (comment)

For now I left that; it won't block us getting 0.21 shipped and would be a great tidy up to do on the way to 0.22, perhaps.

Copy link
Contributor

@LilyFirefly LilyFirefly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still looks good to me!

Copy link
Member

@Icxolu Icxolu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@davidhewitt
Copy link
Member Author

🎉 thanks both!

@davidhewitt davidhewitt added this pull request to the merge queue Feb 27, 2024
Merged via the queue into PyO3:main with commit 9370404 Feb 27, 2024
@davidhewitt davidhewitt deleted the pyref-bound-internals branch February 27, 2024 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI-skip-changelog Skip checking changelog entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants