PEP 558: Note compatibility constraints on locals(), other updates (#1069)

- new design discussion section to cover the requirement that the
  semantics of locals() itself at function scope be left alone
- propose a C level API that exactly matches Python level
  frame.f_locals semantics
- other minor text formatting and wording updates

Author: ncoghlan

pep-0558.rst

@@ -206,12 +206,11 @@ may not affect the values of local and free variables used by the interpreter."
 This PEP proposes to change that text to instead say:
 
     At function scope (including for generators and coroutines), [this function]
-    returns a
-    dynamic snapshot of the function's local variables and any nonlocal cell
-    references. In this case, changes made via the snapshot are *not* written
-    back to the corresponding local variables or nonlocal cell references, and
-    any such changes to the snapshot will be overwritten if the snapshot is
-    subsequently refreshed (e.g. by another call to ``locals()``).
+    returns a dynamic snapshot of the function's local variables and any
+    nonlocal cell references. In this case, changes made via the snapshot are
+    *not* written back to the corresponding local variables or nonlocal cell
+    references, and any such changes to the snapshot will be overwritten if the
+    snapshot is subsequently refreshed (e.g. by another call to ``locals()``).
 
     CPython implementation detail: the dynamic snapshot for the currently
     executing frame will be implicitly refreshed before each call to the trace
@@ -281,13 +280,15 @@ return a reference to the dynamic snapshot rather than to the write-through
 proxy.
 
 At the C API layer, ``PyEval_GetLocals()`` will implement the same semantics
-as the Python level ``locals()`` builtin, and a new ``PyFrame_GetLocals(frame)``
-accessor API will be provided to allow the proxy bypass logic to be encapsulated
-entirely inside the frame implementation. The C level equivalent of accessing
-``pyframe.f_locals`` in Python will be to access ``cframe->f_locals`` directly
-(the one difference is that accessing ``pyframe.f_locals`` will continue to
-implicitly refresh the dynamic snapshot, whereas C code will need to explicitly
-call ``PyFrame_GetLocals(frame)`` to refresh the snapshot).
+as the Python level ``locals()`` builtin, and a new
+``PyFrame_GetPyLocals(frame)`` accessor API will be provided to allow the
+function level proxy bypass logic to be encapsulated entirely inside the frame
+implementation.
+
+The C level equivalent of accessing ``pyframe.f_locals`` in Python will be a
+new ``PyFrame_GetLocalsAttr(frame)`` API. Like the Python level descriptor, the
+new API will implicitly refresh the dynamic snapshot at function scope before
+returning a reference to the write-through proxy.
 
 The ``PyFrame_LocalsToFast()`` function will be changed to always emit
 ``RuntimeError``, explaining that it is no longer a supported operation, and
@@ -324,7 +325,59 @@ The proposal in this PEP aims to retain the first two properties (to maintain
 backwards compatibility with as much code as possible) while ensuring that
 simply installing a trace hook can't enable rebinding of function locals via
 the ``locals()`` builtin (whereas enabling rebinding via
-``inspect.currentframe().f_locals`` is fully intended).
+``frame.f_locals`` inside the tracehook implementation is fully intended).
+
+
+Keeping ``locals()`` as a dynamic snapshot at function scope
+------------------------------------------------------------
+
+It would theoretically be possible to change the semantics of the ``locals()``
+builtin to return the write-through proxy at function scope, rather than
+continuing to return a dynamic snapshot.
+
+This PEP doesn't (and won't) propose this as it's a backwards incompatible
+change in practice, even though code that relies on the current behaviour is
+technically operating in an undefined area of the language specification.
+
+Consider the following code snippet::
+
+    def example():
+        x = 1
+        locals()["x"] = 2
+        print(x)
+
+Even with a trace hook installed, that function will consistently print ``1``
+on the current reference interpreter implementation::
+
+    >>> example()
+    1
+    >>> import sys
+    >>> def basic_hook(*args):
+    ...     return basic_hook
+    ...
+    >>> sys.settrace(basic_hook)
+    >>> example()
+    1
+
+Similarly, ``locals()`` can be passed to the ``exec()`` and ``eval()`` builtins
+at function scope without risking unexpected rebinding of local variables.
+
+Provoking the reference interpreter into incorrectly mutating the local variable
+state requires a more complex setup where a nested function closes over a
+variable being rebound in the outer function, and due to the use of either
+threads, generators, or coroutines, it's possible for a trace function to start
+running for the nested function before the rebinding operation in the outer
+function, but finish running after the rebinding operation has taken place (in
+which case the rebinding will be reverted, which is the bug reported in [1]_).
+
+In addition to preserving the de facto semantics which have been in place since
+PEP 227 introduced nested scopes in Python 2.1, the other benefit of restricting
+the write-through proxy support to the implementation-defined frame object API
+is that it means that only interpreter implementations which emulate the full
+frame API need to offer the write-through capability at all, and that
+JIT-compiled implementations only need to enable it when a frame introspection
+API is invoked, or a trace hook is installed, not whenever ``locals()`` is
+accessed at function scope.
 
 
 What happens with the default args for ``eval()`` and ``exec()``?
@@ -333,7 +386,7 @@ What happens with the default args for ``eval()`` and ``exec()``?
 These are formally defined as inheriting ``globals()`` and ``locals()`` from
 the calling scope by default.
 
-There doesn't seem to be any reason for the PEP to change this.
+There isn't any need for the PEP to change these defaults, so it doesn't.
 
 
 Changing the frame API semantics in regular operation