Python 3.{8,9,10}: functions are not saved when globals are not provided to `py.run()`

[GoldsteinE]

Bug Description

Passing None as a globals argument to py.run() leads to newly defined functions seemingly not being saved anywhere at least under Python 3.8, 3.9 and 3.10.

Steps to Reproduce

Run the following code:

fn main() -> eyre::Result<()> {
    let script_text = r#"
def trivial(s):
    return s
class Foo:
    def __init__(self):
        print(trivial('Hello, world!'))
    "#;

    pyo3::Python::with_gil(|py| {
        let module = pyo3::types::PyDict::new(py);
        py.run(script_text, None, Some(module))?;
        let foo_class = module.get_item("Foo").unwrap();
        let _foo = foo_class.call0()?;
        Ok(())
    })
}

Observe the following error:

NameError: name 'trivial' is not defined

Your operating system and version

NixOS

Your Python version (python --version)

Python 3.8.17, Python 3.9.17, Python 3.10.12

Your Rust version (rustc --version)

rustc 1.73.0-nightly (8131b9774 2023-08-02)

Your PyO3 version

0.19.2

How did you install python? Did you use a virtualenv?

Nix, no virtualenv

Additional Info

Possibly related to #3370, #2891

[GoldsteinE]

A pure C repro of this one:

#include <stdlib.h>
#include <Python.h>

PyObject* must(const char* desc, PyObject* ptr) {
    if (!ptr) {
        dprintf(2, "falied to %s\nPython exception:\n", desc);
        PyErr_Print();
        exit(1);
    }
    return ptr;
}

const char* CODE = "\
def trivial(s):\n\
    return s\n\
class Foo:\n\
    def __init__(self):\n\
        print(trivial('Hello, world!'))\n\
";

int main(void) {
    Py_Initialize();
    PyObject* main_module = must("import __main__", PyImport_AddModule("__main__"));
    PyObject* globals = must("get __main__'s dict", PyModule_GetDict(main_module));
    PyObject* locals = must("create an empty dict", PyDict_New());
    PyObject* code_obj = must("make a code object", Py_CompileString(CODE, "<string>", Py_file_input));
    PyObject* ok = must("evaluate the code object", PyEval_EvalCode(code_obj, globals, locals));
    PyObject* Class = must("get class from locals", PyDict_GetItemString(locals, "Foo"));
    PyObject* instance = must("create an instance", PyObject_CallNoArgs(Class));
}

[GoldsteinE]

Pure Python repro:

import __main__
gs = __main__.__dict__
ls = dict()
code_s = """
def trivial(s):
    return s
class Foo:
    def __init__(self):
        print(trivial('Hello, world!'))
"""
c = compile(code_s, '<string>', 'exec')
eval(c, gs, ls)
ls['Foo']()

I guess the problem is obvious now: actually calling Foo() occurs in a different context than defining it, so it’s not a bug. Not sure what workaround could be though.

[davidhewitt]

Thanks for the investigation. One solution seems to be to add the values from ls into globals(), but that seems like pure evil. Maybe with some investigation of the internals of Python frames we could work out how to inject trivial back into the frame of Foo.__init__, but that also seems pretty evil!

[GoldsteinE]

Thanks for the investigation. One solution seems to be to add the values from ls into globals(), but that seems like pure evil. Maybe with some investigation of the internals of Python frames we could work out how to inject trivial back into the frame of Foo.init, but that also seems pretty evil!

I closed this because I don’t see a non-evil solution, yes. This problem is Python’s quirk that’s reproducible even in pure Python, so there’re few things pyo3 can do here, except maybe document this pitfall.