Object protocols without specialization by kngwyu · Pull Request #961 · PyO3/pyo3

kngwyu · 2020-06-06T08:51:25Z

For #210.
Now all pyclass has a method registory for protocols:

impl pyo3::class::proto_methods::HasProtoRegistry for MyClass {
    fn registory() -> &'static pyo3::class::proto_methods::PyProtoRegistry {
        static REGISTRY: PyProtoRegistry = PyProtoRegistry::new();
        &REGISTRY
    }
}

, where PyProtoRegistory is a set of method tables:

#[doc(hidden)]
pub struct PyProtoRegistry {
    async_methods: AtomicPtr<PyAsyncMethods>,
    basic_methods: AtomicPtr<PyObjectMethods>,
    ...
}

.

Using the ctor crate, we insert some initialization codes for this registry.
For example, when we write this code:

#[pyproto]
impl PyObjectProtocol for MyClass {
    fn __hash__(&self) -> PyResult<isize> {
        Ok(self.val as isize)
    }
}

, (roughly) the following code is generated:

#[ctor]
fn __init_Object_2817873() {
    let mut table = pyo3::class::PyObjectMethods::default();
    table.set_hash::<MyClass>();
    <MyClass as pyo3::class::proto_methods::HasProtoRegistry>::registory().set_basic_methods(table);
}

One disadvantage of this solution is it needs redundant Box::new for some protocols(basic, iter, and so on). However, to keep PyProtoRegistry simple and small, I chose this design.

Unfortunately, some protocols are not yet tested.

TODO

Add tests for async protocols
- Maybe we need to change the trait definition to make it work?
Add tests for descr protocols
Maybe need some more doc comments?

kngwyu · 2020-06-06T09:05:39Z

I forgot to mention that this PR has a breaking change. nb_bool is now moved under PyNumberProtocol from PyObjectProtocol.

davidhewitt

Thanks so much for this - it's excellent to see that this code is shorter than what was there before and should unblock us for compiling pyo3 on stable! 🎉

There's a couple things I'd like to see changed:

Typo registory -> registry in several places
Let's fix the Send + Sync bounds in #948 rather than adding in this PR.

I forgot to mention that this PR has a breaking change. nb_bool is now moved under PyNumberProtocol from PyObjectProtocol.

I understand why you moved this, but I am not sure if it is the right change from a user design.

The related C API is listed in the Object Protocol: https://docs.python.org/3/c-api/object.html?highlight=pyobject_istrue

Does it make the implementation much more complicated if we can keep this in the PyObjectProtocol?

pyo3-derive-backend/src/pyclass.rs

src/class/proto_methods.rs

davidhewitt · 2020-06-07T10:06:20Z

src/class/proto_methods.rs

+        self.async_methods
+            .store(Box::into_raw(Box::new(methods)), Ordering::SeqCst)


Technically all of these could leak memory if called for a second time.

Yes, but the current code does the same. See https://github.com/PyO3/pyo3/blob/v0.10.1/src/pyclass.rs#L151.
The difference is now we cause memory leak for all protocols(e.g., iter and basic).
We can avoid this by changing the pub struct PyProtoRegistry like the below:

pub struct PyProtoRegistry { /// Protocols that needs specific table types async_methods: AtomicPtr<ffi::PyAsyncMethods>, buffer_methods: AtomicPtr<ffi::PyBufferProcs>, mapping_methods: AtomicPtr<ffi::PyMappingMethods>, number_methods: AtomicPtr<ffi::PyNumberMethods>, sequence_methods: AtomicPtr<ffi::PySequenceMethods>, /// Other methods. /// AtomicUsize is used instead of `Atomic<Option<ffi::func>>` or so because /// we can't use function pointer in const function. tp_str: AtomicUsize, tp_repr: AtomicUsize. .... }

.

I chose this design first but abandoned it due to its ugliness.

Cool, I think your design choice is fine. Can we maybe just add a docstring for now to warn calling these twice will leak?

src/class/pyasync.rs

src/instance.rs

src/pycell.rs

kngwyu · 2020-06-07T13:19:41Z

Thank you for the review!

Let's fix the Send + Sync bounds in #948 rather than adding in this PR.

I accidentally pushed these changes when I was playing with them. I'll revert them.

Does it make the implementation much more complicated if we can keep this in the PyObjectProtocol?

I don't have any good solution now, unfortunately.

davidhewitt · 2020-06-07T20:26:27Z

I don't have any good solution now, unfortunately.

👍 ok. I have a half-baked idea but it's hard to write down as I haven't properly thought about it yet. I'll try to find time tomorrow evening to experiment with it whether it can help and explain it better then.

kngwyu · 2020-06-08T00:12:25Z

I have a half-baked idea but it's hard to write down as I haven't properly thought about it yet.

Thank you but wait for a bit...
I don't think most users know that __bool__ is classified as a part of Object protocols.
Thus I don't think it's worth doing if we need more codes to do so.

EDITED:
I just tried to move nb_bool under ObjectProtocol again but the resulting code is not so smart 🤔

    // nb_bool is a part of PyObjectProtocol, but should be placed under tp_as_number
    let mut nb_bool = None;
    // basic methods
    if let Some(basic) = T::basic_methods() {
        unsafe { basic.as_ref() }.update_typeobj(type_object);
        nb_bool = unsafe { basic.as_ref() }.nb_bool;
    }

    // number methods
    type_object.tp_as_number = T::number_methods()
        .map(|mut p| {
            unsafe { p.as_mut() }.nb_bool = nb_bool;
            p.as_ptr()
        })
        .unwrap_or_else(|| {
            if nb_bool.is_some() {
                let mut number_methods = ffi::PyNumberMethods::default();
                number_methods.nb_bool = nb_bool;
                Box::into_raw(Box::new(number_methods))
            } else {
                ptr::null_mut()
            }
        });

Still not sure we should do this...

kngwyu · 2020-06-08T03:06:05Z

At last, I decided to move nb_bool under PyObjectProtocol again.
I don't think the code is very ugly but please check it.

davidhewitt · 2020-06-08T18:16:42Z

pyo3-derive-backend/src/pyproto.rs

+    // E.g., if we have `__add__`, we need to skip `set_radd`.
+    let mut skipped_setters = HashSet::new();


Do you think that we still need skipped_setters once we implement a solution for #844?

pyo3-derive-backend/src/pyproto.rs

src/class/pyasync.rs

davidhewitt

At last, I decided to move nb_bool under PyObjectProtocol again.
I don't think the code is very ugly but please check it.

I think what you've done is nice. It's good to avoid the unnecessary breaking change 😄

This branch is looking good, I'm very excited by it! As well as your TODO for docs and tests I would be interested to hear what you think about how we could solve #844 with this new implementation.

src/class/macros.rs

src/pyclass.rs

davidhewitt · 2020-06-08T18:32:23Z

It would also be good to add a test for #940 in this PR - if I read your implementation correctly I think that should already be solved by what you've done!

kngwyu · 2020-06-13T10:44:19Z

Added a test for __await__.
I was really confused by Python's async/await and noticed some things:

Returning Future from Rust is hard.
To return some value, we have to make StopIterException have the value.

But for now, I think this is sufficient for testing proc-macros.

kngwyu · 2020-06-13T11:22:27Z

OMG...

(Windows + Python3.8)

tests/test_dunder.rs

madhavajay · 2020-06-15T07:14:01Z

🍿🤓

kngwyu · 2020-06-15T14:42:24Z

I would be interested to hear what you think about how we could solve #844 with this new implementation.

I think we can reuse the same trick as __set/delitem__ for that, but maybe it should be done in another PR 🤔

davidhewitt · 2020-06-15T19:18:25Z

I think we can reuse the same trick as set/delitem for that, but maybe it should be done in another PR 🤔

I'm happy with that - this PR is already quite large 😄

kngwyu · 2020-06-17T03:46:55Z

Now I think this PR is ready.
Though I added more comments than usual, please point out what you think is unclear.

davidhewitt

This is looking fantastic to me! +1,373 / -2,245 - nice job 🚀

davidhewitt · 2020-06-17T08:04:26Z

pyo3-derive-backend/src/pyproto.rs

                    quote!(0)
                };
-                // TODO(kngwyu): doc
+                // TODO(kngwyu): Set ml_doc


Let's create an issue for this so that we remember to do it?

konstin · 2020-06-17T08:39:22Z

The test_await was failing for me locally (python3.6 from apt in virtualenv), this is the workaround I used to make them pass.

Traceback (most recent call last):
  File "<string>", line 12, in <module>
  File "/usr/lib/python3.6/asyncio/events.py", line 694, in get_event_loop
    return get_event_loop_policy().get_event_loop()
  File "/usr/lib/python3.6/asyncio/events.py", line 602, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Dummy-1'.
test test_await ... FAILED

diff --git a/tests/test_dunder.rs b/tests/test_dunder.rs
index 67a66fb49..47f22fc91 100644
--- a/tests/test_dunder.rs
+++ b/tests/test_dunder.rs
@@ -612,7 +612,11 @@ async def main():
 # but we see still errors on Github actions...
 if sys.platform == "win32" and sys.version_info >= (3, 8, 0):
     asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
-loop = asyncio.get_event_loop()
+try:
+    loop = asyncio.get_event_loop()
+except RuntimeError: # RuntimeError: There is no current event loop in thread 'Dummy-1'.
+    loop = asyncio.new_event_loop()
+    asyncio.set_event_loop(loop)
 assert loop.run_until_complete(main()) is None
 loop.close()
 "#

kngwyu · 2020-06-17T13:17:58Z

@konstin
Thanks, I pushed the change based on your diff.

davidhewitt · 2020-06-18T08:20:35Z

🎉

davidhewitt requested changes Jun 7, 2020

View reviewed changes

davidhewitt mentioned this pull request Jun 7, 2020

Add nightly-optimizations feature #927

Closed

1 task

davidhewitt reviewed Jun 8, 2020

View reviewed changes

davidhewitt approved these changes Jun 8, 2020

View reviewed changes

src/class/macros.rs Show resolved Hide resolved

src/pyclass.rs Show resolved Hide resolved

konstin mentioned this pull request Jun 9, 2020

Stable Rust #969

Merged

kngwyu added 6 commits June 13, 2020 15:24

Remove specialization from basic/buffer/descr/iter protocols

7967874

Remove specialization from gc/mapping/number protocols

0d08296

Remove specialization from async/sequence methods

d678093

Avoid function definition in setters for readability

05effe2

Fix typo and revert unintended changes

ac2c51c

Move nb_bool under PyObjectProtocol again

f322771

kngwyu force-pushed the slot-provider branch 2 times, most recently from 1d6295c to ed31c70 Compare June 13, 2020 10:42

kngwyu force-pushed the slot-provider branch from ed31c70 to 280ce0b Compare June 13, 2020 11:05

kngwyu force-pushed the slot-provider branch from 280ce0b to 78577c6 Compare June 13, 2020 12:19

davidhewitt reviewed Jun 14, 2020

View reviewed changes

tests/test_dunder.rs Show resolved Hide resolved

tests/test_dunder.rs Outdated Show resolved Hide resolved

tests/test_dunder.rs Outdated Show resolved Hide resolved

kngwyu force-pushed the slot-provider branch from 78577c6 to c8fae71 Compare June 14, 2020 13:40

Async methods now takes Receiver

8c4162d

kngwyu force-pushed the slot-provider branch from c8fae71 to 8c4162d Compare June 14, 2020 13:43

Descr get/set now take receiver

2e1ed2c

kngwyu added 2 commits June 17, 2020 02:17

Use Relaxed ordering since setters are called by a single thread

f7fa008

Mark protocols setters as doc(hidden)

2a69367

Add more comments for #[pyproto] related parts

fc3fa70

kngwyu force-pushed the slot-provider branch from 245dc53 to fc3fa70 Compare June 17, 2020 04:21

davidhewitt approved these changes Jun 17, 2020

View reviewed changes

Make test_await more robust

1207c2d

kngwyu force-pushed the slot-provider branch from da381e5 to 1207c2d Compare June 17, 2020 16:58

kngwyu added 2 commits June 18, 2020 15:49

Merge branch 'master' into slot-provider

a044f3c

Add a CHANGELOG entry for 961

8a0c9bf

kngwyu merged commit 390ff5f into PyO3:master Jun 18, 2020

kngwyu deleted the slot-provider branch June 18, 2020 08:19

kngwyu mentioned this pull request Jun 23, 2020

Cannot return option from PySequenceProtocol __getitem__ #798

Closed

		self.async_methods
		.store(Box::into_raw(Box::new(methods)), Ordering::SeqCst)

		// E.g., if we have `__add__`, we need to skip `set_radd`.
		let mut skipped_setters = HashSet::new();

Conversation

kngwyu commented Jun 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

kngwyu commented Jun 6, 2020

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidhewitt Jun 7, 2020

Choose a reason for hiding this comment

Uh oh!

kngwyu Jun 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidhewitt Jun 7, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kngwyu commented Jun 7, 2020

Uh oh!

davidhewitt commented Jun 7, 2020

Uh oh!

kngwyu commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kngwyu commented Jun 8, 2020

Uh oh!

davidhewitt Jun 8, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

davidhewitt commented Jun 8, 2020

Uh oh!

kngwyu commented Jun 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kngwyu commented Jun 13, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

madhavajay commented Jun 15, 2020

Uh oh!

kngwyu commented Jun 15, 2020

Uh oh!

davidhewitt commented Jun 15, 2020

Uh oh!

kngwyu commented Jun 17, 2020

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

davidhewitt Jun 17, 2020

Choose a reason for hiding this comment

Uh oh!

konstin commented Jun 17, 2020

Uh oh!

kngwyu commented Jun 17, 2020

Uh oh!

davidhewitt commented Jun 18, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

kngwyu commented Jun 6, 2020 •

edited

Loading

kngwyu Jun 7, 2020 •

edited

Loading

kngwyu commented Jun 8, 2020 •

edited

Loading

kngwyu commented Jun 13, 2020 •

edited

Loading