add bound dict constructors & py.run variants

[davidhewitt]

This is based on top of #3711

The second commit adds PyDict bound constructors. At the same time I also added Python::eval_bound, Python::run_bound, and py_run_bound!, because these APIs consume dictionaries and it's extremely churn heavy to change all of these. So to at least do the churn in a single hop, I've pushed it as a single commit.

[codspeed-hq]

CodSpeed Performance Report

Merging #3716 will degrade performances by 14.16%

_{Comparing davidhewitt:dict-constructors (8cd3133) with main (c54d897)}

Summary

⚡ 1 improvements
❌ 6 regressions
✅ 73 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	davidhewitt:dict-constructors	Change
❌	extract_int_downcast_fail	266.1 ns	297.2 ns	-10.47%
❌	extract_str_downcast_fail	266.1 ns	297.2 ns	-10.47%
❌	extract_str_extract_fail	2.3 µs	2.6 µs	-11.78%
❌	extract_float_downcast_success	418.9 ns	474.4 ns	-11.71%
❌	list_via_extract	336.7 ns	392.2 ns	-14.16%
❌	extract_float_extract_success	408.9 ns	464.4 ns	-11.96%
⚡	test_simple_py	34.7 µs	31.4 µs	+10.67%

[davidhewitt]

The performance "regressions" here are because the benchmarks are now eagerly dropping the containers rather than leaving them on the pool. I'll try to adjust accordingly.

The iter_dict improvement of ~25% is pretty consistent with the sort of gains I've seen from this new API! 🎉

[davidhewitt]

Ok, this one is now good to review. The performance regressions still reported are on the ns scale and I expect are sensitive to changes in inlining due to benchmark changes. I'm not bothered by those as they might get change again later, e.g. after #3706

The iter_dict performance gain, I'm pretty sure is a genuine improvement of using the new Bound API instead of the pool.

[adamreichold]

99 changed files in the diff, 99 changed files,
review one, mark it as viewed, 98 changed files in the diff.

[davidhewitt]

Yes, I agree, it's a horrible diff. If you've got any suggestions on ways to split it to help review, I'm very open to trying. The problem I found is that adding these constructors basically immediately forced either adding locals.as_gil_ref() or #[allow(deprecated)] on every of these py.run sites anyway, so this diff wasn't much larger and ultimately seemed clearer imo.

[davidhewitt]

... maybe I can try to see what happens if I just do py.run and friends first, and save the dict constructors for a second review? Let me know if you'd like me to try that.

[davidhewitt]

(And for what it's worth, I have a feeling that this is one of the largest single diff PRs that we might face. These ones are used in so many tests.)

[davidhewitt]

In terms of why some tests have this blanket applied to mod tests, I did this for files where the GIL Ref API is tested in some detail in the contained tests, mostly inside src/types directory. I think we want to keep the GIL Ref tests for coverage anyway, so my feelings were to just do this now as one way to keep churn down, and revisit these after 0.21 is pushed and we're heading towards 0.22.

[adamreichold]

Please don't worry. Sometimes we'll just have to pull through. I will finish the review eventually. I also understand that it is mostly tests and the changes there are mechanical so I can skim those files. I guess I will focus on the "source" changes in the any case.

[davidhewitt]

I've just rebased this, sorry if that disrupts a partial review.

I'm going to push a separate PR in a second which adds PyDict::new_bound without the deprecation, as that'll allow us to use that in other PRs and also maybe allow us to incrementally cut away at this diff.

[davidhewitt]

@Icxolu FYI I just rebased this; it contains py.run_bound and py.eval_bound as well as deprecation change for PyDict::new_bound . This is probably still too much to be reviewable in one PR. I don't know whether you want to chop this PR up, start fresh, or just review this one. Up to you 👍

[Icxolu]

Thanks for the ping, I must have overlooked this one. I give it a try to see if this can reasonably be split, otherwise we might have to review it like this.

[davidhewitt]

I think one option is to merge the rest of the new _bound API additions here without adding deprecations, like I already did for PyDict::new_bound. That way the rest of the changes could be done one API at a time, or group of files at a time, etc...

[Icxolu]

So I have played around with this a little. I could split off Python::eval and Python::run each into their separate PR (each with ~200 lines changed) leaving this PR to switch the PyDict/IntoPyDict constructors. @davidhewitt What do you think about this?

[davidhewitt]

I think that would be excellent, yes please! I can either take responsibility for rebasing this one or leave it to you to eventually replace this one too.

[davidhewitt]

Closing with thanks to @Icxolu for splitting this up and making a much better end result 🚀