Python 3.{8,9}: `NameError: __build_class__ not found` when globals are provided to `py.run()`

[GoldsteinE]

Bug Description

Passing Some(_) as globals to py.run() breaks Python builtins under Python 3.8 and 3.9 (and maybe others, but not 3.10).

Steps to Reproduce

Run the following code (linking with Python 3.8 or Python 3.9):

fn main() -> eyre::Result<()> {
    let script_text = r#"
def trivial(s):
    return s
class Foo:
    def __init__(self):
        print(trivial('Hello, world!'))
    "#;

    pyo3::Python::with_gil(|py| {
        let module = pyo3::types::PyDict::new(py);
        py.run(script_text, Some(module), Some(module))?;
        let foo_class = module.get_item("Foo").unwrap();
        let _foo = foo_class.call0()?;
        Ok(())
    })
}

Observe the following error:

NameError: __build_class__ not found

Your operating system and version

NixOS

Your Python version (python --version)

Python 3.8.17, Python 3.9.17

Your Rust version (rustc --version)

rustc 1.73.0-nightly (8131b9774 2023-08-02)

Your PyO3 version

0.19.2

How did you install python? Did you use a virtualenv?

Nix, no virtualenv

Additional Info

Possibly related to #3371, #2891

[GoldsteinE]

Pure C repro of this one:

#include <stdlib.h>
#include <Python.h>

PyObject* must(const char* desc, PyObject* ptr) {
    if (!ptr) {
        dprintf(2, "falied to %s\nPython exception:\n", desc);
        PyErr_Print();
        exit(1);
    }
    return ptr;
}

const char* CODE = "\
def trivial(s):\n\
    return s\n\
class Foo:\n\
    def __init__(self):\n\
        print(trivial('Hello, world!'))\n\
";

int main(void) {
    Py_Initialize();
    PyObject* main_module = must("import __main__", PyImport_AddModule("__main__"));
    PyObject* locals = must("create an empty dict", PyDict_New());
    PyObject* code_obj = must("make a code object", Py_CompileString(CODE, "<string>", Py_file_input));
    PyObject* ok = must("evaluate the code object", PyEval_EvalCode(code_obj, locals, locals));
    PyObject* Class = must("get class from locals", PyDict_GetItemString(locals, "Foo"));
    PyObject* instance = must("create an instance", PyObject_CallNoArgs(Class));
}

[GoldsteinE]

That’s this one CPython issue: python/cpython#24564

PyFrame_New(), PyEval_EvalCode(), PyEval_EvalCodeEx(), PyFunction_New() and PyFunction_NewWithQualName() now inherits the current builtins namespace if the globals dictionary has no __builtins__ key.

While technically not a bug in PyO3, I propose (and I’m willing to send a PR) to always (or maybe when on Python <3.10) add __builtins__ if they are not present in globals dictionary to improve consistency.

[davidhewitt]

Thanks for the report and the investigation. I think that adding __builtins__ to the globals dictionary would be a good consistency improvement, I think we should do it on all versions. A PR would be very much appreciated.