From 62b8d1d0fd98f7e50a3140c6feda5b885786b879 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 2 Jul 2024 10:17:12 +0200 Subject: [PATCH 1/5] Rewrite PEP 743 --- peps/pep-0743.rst | 310 +++++++++++++++++++++++++++------------------- 1 file changed, 186 insertions(+), 124 deletions(-) diff --git a/peps/pep-0743.rst b/peps/pep-0743.rst index 39d9039eb1a..30b293798c9 100644 --- a/peps/pep-0743.rst +++ b/peps/pep-0743.rst @@ -1,10 +1,11 @@ PEP: 743 Title: Add Py_COMPAT_API_VERSION to the Python C API -Author: Victor Stinner +Author: Victor Stinner , + Petr Viktorin , Status: Draft Type: Standards Track Created: 11-Mar-2024 -Python-Version: 3.13 +Python-Version: 3.14 .. highlight:: c @@ -12,177 +13,235 @@ Python-Version: 3.13 Abstract ======== -Add ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` macros -to opt-in for planned incompatible C API changes in a C extension. -Maintainers can decide when they make their C extension compatible -and also decide which future Python version they want to be compatible -with. +Add ``Py_COMPAT_API_VERSION`` C macro that hides some deprecated and +soft-deprecated symbols, allowing users to opt out of using API with known +issues that other API solves. +The macro is versioned, allowing users to update (or not) on their own pace. +Add convenience macros ``Py_PACK_VERSION`` and ``Py_PACK_VER`` to +make it easier to express versions in C code. -Rationale -========= -Python releases enforce C API changes -------------------------------------- +Motivation +========== -Every Python 3.x release has a long list of C API changes, including -incompatible changes. C extensions have to be updated to work on the -newly released Python. +Some of Python's C API has flaws that are only obvious in hindsight. -Some incompatible changes are driven by new features: they cannot be -avoided, unless we decide to not add these features. Other reasons: +For API that prevents adding features, or presents a serious security risk +or maintenance burden, we deprecate and remove as described in :pep:`387`. -* Remove deprecated API (see :pep:`387`). -* Ease the implementation of another change. -* Change or remove error-prone API. +However, this leaves us with some API that has “sharp edges”, which works fine +for its current users, but should be avoided in new code. +For example: -Currently, there is no middle ground between "not change the C API" and -"incompatible C API changes impact everybody". Either a C extension is -updated or the new Python version cannot be used. Such all-or-nothing -deal does not satisfy C extension maintainers nor C extensions users. +.. (These are examples, not categories. They overlap.) +- API that is easy to misuse or subtly dangerous, but doesn't have a scary + name; +- API that cannot signal an exception, so failures are either ignored or + exit the process with a fatal error; +- API that is not thread-safe (for example by borrowing references from + mutable objects, or exposing unfinished mutable objects); +- API with names that don't use the ``Py``/``_Py`` prefix, and so can clash + with other code. For example: ``setter``. -Limited C API -------------- +XXX: add examples -The limited C API is versioned: the ``Py_LIMITED_API`` macro can be set -to a Python version to select which API is available. On the Python -side, it allows introducing incompatible changes at a specific -``Py_LIMITED_API`` version. For example, if ``Py_LIMITED_API`` is set to -Python 3.11 or newer, the ```` is no longer included by -``Python.h``, whereas C extensions targeting Python 3.10 are not -affected. +It is important to note that despite such flaws, it's usually possible +to use the API correctly. For example, in a single-threaded environment, +thread safety is not an issue. +We do not want to break working code, even if it uses API that would be wrong +in some -- or even *most* -- other contexts. -The difference here is that upgrading Python does not change if -```` is included or not, but updating ``Py_LIMITED_API`` does. -Updating ``Py_LIMITED_API`` is an deliberate action made by the C -extension maintainer. It gives more freedom to decide **when** the -maintainer is ready to deal with the latest batch of incompatible -changes. +But, it's in everyone's best interest to avoid such “undesirable” API in *new* +code, especially if a safer alternative exists. -A similar version can be used with the regular (non-limited) C API. +Rationale +========= -Deprecation and compiler warnings ---------------------------------- +We want to allow an easy way for users to avoid “undesirable” API if they +choose to do so. + +It might be be sufficient to leave this to third-party linters. +For that we'd need a good way to expose a list of (soft-)deprecated +API to such linters. +While adding that, we can -- rather easily -- avoid an extra tool for simple +cases, and do the linter's job directly in CPython headers. + +Unlike Python, C makes it rather easy to limit available API -- for a whole +project or for each individual source file -- by defining a macro. + +Also, for API that lacks the ``Py`` prefix (e.g. ``setter``), hiding the name +entirely means that it cannot clash with user or library code. +A linter cannot do this. +(Of course, we could use another mechanism for just this issue). + +We already do something similar with ``Py_LIMITED_API``, which limits the +available API to a subset that compiles to stable ABI. (In hindsight, we should +have used a different macro name for that particular kind of limiting, but it's +too late to change that now.) + +To prevent working code from breaking as we identify more “undesirable” API +and add safer alternatives to it, the opt-in macro should be *versioned*. +Users can choose a version they need based on their compatibility requirements, +and update it at their own pace. + +To be clear, this mechanism is *not* a replacement for deprecation. +Deprecation is for API that prevents new features or optimizations, or +presents a security risk or maintenance burden. +This mechanism, on the other hand, is meant for cases where “we found +a slightly better way of doing things” -- perhaps one that's harder to misuse, +or just has a less misleading name. +(On a lighter note: the mechanism is meant for the kind of people who +configure a code quality checker to shout at them about the number of blank +lines between functions.) + +The proposed macro does not *change* any API definitions; it only *hides* them. +So, if code compiles with the macro, it'll also compile without it, with +identical behaviour. +This has implications for core devs: to deal with undesirable behaviour, +we'll need to introduce new, better API, and discourage the old one. +We should look at an individual API and fix all its known issues at once, +rather than do codebase-wide sweeps for a single kind of issue, +which would result in multiple renames of the same function. -Deprecated functions are marked with ``Py_DEPRECATED()``. Using a -deprecated function emits a compiler warning. -The problem is that ``pip`` and ``build`` tools hide compiler logs by -default, unless a build fails. Moreover, it's easy to miss a single -warning in the middle of hundred lines of logs. +Specification +============= -Schedule changes ----------------- +We introduce a ``Py_COMPAT_API_VERSION`` macro. +If this macro is defined before ``#include ``, some API definitions +-- as described below -- will be omitted from the Python header files. -Currently, there is no way to schedule a C API change: announce it but -also provide a way to maintainers to test their C extensions with the -change. Either a change is not made, or everybody must update their code -if they want to update Python. +The macro only hides top-level definitions exposed from ````. +Other things (the ABI, structure definitions, macro expansions, static inline +function bodies, etc.) are not affected. +The C API working group (:pep:`731`) has authority over the set of omitted +definitions. -Specification -============= +The set of omitted definitions will be tied to a particular feature release +of CPython, and is finalized in each 3.x.0 Beta 1 release. +In rare cases, entries can be removed (i.e. made available for use) at any +time. + +The macro should be defined to a version in the format used by +``PY_VERSION_HEX``, with the “micro”, “release” and “serial” fields do not +matter and should be set to zero. +For example, to omit API deemed undesirable in 3.14.0b1, users should define +``Py_COMPAT_API_VERSION`` to ``0x030e0000``. -New macros ----------- -Add new ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` -macros. They can be set to test if a C extension is prepared for future -C API changes: compatible with future Python versions. +Requirements for omitted API +---------------------------- -The ``Py_COMPAT_API_VERSION`` macro can be set to a specific Python -version. For example, ``Py_COMPAT_API_VERSION=0x030e0000`` tests C API -changes scheduled in Python 3.14. +An API that is omitted with ``Py_COMPAT_API_VERSION`` must be deprecated +(see :pep:`387`), or: -If the ``Py_COMPAT_API_VERSION`` macro is set to -``Py_COMPAT_API_VERSION_MAX``, all scheduled C API changes are tested at -once. +- be soft-deprecated (see :pep:`387`); +- for all known use cases of the API, have a documented alternative + or workaround; +- have tests to ensure it keeps working (except for 1:1 renames using + ``#define`` or ``typedef``); +- be documented (except if it was never mentioned in previous versions of the + documentation); and +- be approved by the C API working group. -If the ``Py_COMPAT_API_VERSION`` macro is not set, it is to -``PY_VERSION_HEX`` by default. +The mechanism is meant for API that can be trivially replaced by a better +alternative. +API without a replacement should generally be deprecated instead. -The ``Py_COMPAT_API_VERSION`` macro can be set in a single C file or for -a whole project in compiler flags. The macro does not affected other -projects or Python itself. +Location +-------- -Example in Python ------------------ +All API definitions omitted by ``Py_COMPAT_API_VERSION`` will be moved to +a new header, ``Include/legacy.h``. -For example, the ``PyImport_ImportModuleNoBlock()`` function is -deprecated in Python 3.13 and scheduled for removal in Python 3.15. The -function can be declared in the Python C API with the following -declaration: +This is meant to help linter authors compile lists, so they can flag the API +with warnings rather than errors. -.. code-block:: c +Note that for simple renaming of source-only constructs (macros, types), we +expect names to be omitted in the same version -- or the same PR -- that adds +a replacement. +This means that the original definition will be renamed, and a ``typedef`` +or ``#define`` for the old name added to ``Include/legacy.h``. + + +Documentation +------------- - #if Py_COMPAT_API_VERSION < 0x030f0000 - Py_DEPRECATED(3.13) PyAPI_FUNC(PyObject *) PyImport_ImportModuleNoBlock( - const char *name /* UTF-8 encoded string */ - ); - #endif +Documentation for omitted API should generally appear after the recommended +replacement, reference it (e.g. “Similar to X, but…”), and focus on differences +from the replacement and migration advice. -If ``if Py_COMPAT_API_VERSION`` is equal to or greater than Python 3.15 -(``0x030f0000``), the ``PyImport_ImportModuleNoBlock()`` function is not -declared, and so using it fails with a build error. -Goals ------ +Initial set +----------- -* Reduce the number of C API changes affecting C extensions when - updating Python. -* When testing C extensions (for example, optional CI test), - ``Py_COMPAT_API_VERSION`` can be set to ``Py_COMPAT_API_VERSION_MAX`` - to detect future incompatibilities. For mandatory tests, it is - recommended to set ``Py_COMPAT_API_VERSION`` to a specific Python - version. -* For core developers, make sure that the C API can still evolve - without being afraid of breaking an unknown number of C extensions. +The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to +``0x030e0000`` (3.14) or greater: -Non-goals ---------- +- ``PyList_GetItem``, omitted in favour of the existing ``PyList_GetItemRef`` -* Freeze the API forever: this is not the stable ABI. For example, - deprecated functions will continue to be removed on a regular basis. -* C extensions maintainers not using ``Py_COMPAT_API_VERSION`` will - still be affected by C API changes when updating Python. -* Provide a stable ABI: the macro only impacts the regular (non-limited) - API. -* Silver bullet solving all C API issues. +- XXX (Victor, do you have a list? :) +- Typedefs without the ``Py``/``_Py`` prefix + (``getter``, ``setter``, ``allocfunc``, …), omitted in favour of *new* ones + that add the prefix (``Py_getter`` , etc.) -Examples of ``Py_COMPAT_API_VERSION`` usages -============================================ +- Macros without the ``Py``/``_Py`` prefix + (``METH_O``, ``CO_COROUTINE``, ``FUTURE_ANNOTATIONS``, ``WAIT_LOCK``, …), + omitted in favour of *new* ones that add the prefix (``Py_METH_O`` , etc.). -* Remove deprecated functions. -* Remove deprecated structure members, such as - ``PyBytesObject.ob_shash``. -* Remove a standard ``#include``, such as ``#include ``, - from ````. -* Change the behavior of a function or a macro. For example, calling - ``PyObject_SetAttr(obj, name, NULL)`` can fail, to enforce the usage - of the ``PyObject_DelAttr()`` function instead to delete an attribute. + - Replacements for macros generated by ``configure`` + (``HAVE_*``, ``WITH_*``, ``ALIGNOF_*``, ``SIZEOF_*``, and several without + a common prefix) may not be ready for 3.14; if so they'll be omitted + in later versions. + +- Any others approved by the C API workgroup + +Helper macros +------------- + +To make it easier to set and test values for ``Py_COMPAT_API_VERSION``, +``Py_LIMITED_API``, and other versions, we will add two helper macros: + +- ``Py_PACK_VERSION(3, y, z, level, serial)`` packs the version + as an integer in the format used by ``PY_VERSION_HEX``. + For example, ``Py_PACK_VERSION(3, 4, 1, 0xA, 2)`` gives ``0x030401a2``. + +- ``Py_PACK_VER(3, y)`` is shorthand for ``Py_PACK_VERSION(3, y, 0, 0, 0)``, + useful because the first two version components often determine ABI + compatibility. + +Library functions with the sames and functionality will be exported, for use in +auto-generated wrappers for non-C languages. +(The macro-style naming means that we encourage implementing them as +compile-time constructs, rather than library calls -- but that's harder to +auto-generate.) Implementation ============== -* `Issue gh-116587 `_ -* PR: `Add Py_COMPAT_API_VERSION and Py_COMPAT_API_VERSION_MAX macros - `_ +TBD + + +Open issues +=========== + +The name ``Py_COMPAT_API_VERSION`` was taken from the earlier PEP; +it doesn't fit this version. Backwards Compatibility ======================= -There is no impact on backward compatibility. - -Adding ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` -macros has no effect on backward compatibility. Only developers setting -the ``Py_COMPAT_API_VERSION`` macro in their project will be impacted by -effects of this macro which is the expected behavior. +The macro is backwards compatible. +Developers can introduce and update the macro on their own pace, potentially +for one source file at a time. Discussions @@ -195,6 +254,9 @@ Discussions with no known issues `_ (June 2023) +* Finishing the Great Renaming + `_ + (May 2024) Prior Art From 784e9157f9cd185cd82ce6e10f5b368f6c3a9760 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 2 Jul 2024 16:40:05 +0200 Subject: [PATCH 2/5] PEP 737: List Omitted APIs --- peps/pep-0743.rst | 114 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 111 insertions(+), 3 deletions(-) diff --git a/peps/pep-0743.rst b/peps/pep-0743.rst index 30b293798c9..b7099f0b924 100644 --- a/peps/pep-0743.rst +++ b/peps/pep-0743.rst @@ -183,9 +183,117 @@ Initial set The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to ``0x030e0000`` (3.14) or greater: -- ``PyList_GetItem``, omitted in favour of the existing ``PyList_GetItemRef`` - -- XXX (Victor, do you have a list? :) +- Omit API returning borrowed references: + + ==================================== ============================== + Omitted API Replacement + ==================================== ============================== + ``PyDict_GetItem()`` ``PyDict_GetItemRef()`` + ``PyDict_GetItemString()`` ``PyDict_GetItemStringRef()`` + ``PyImport_AddModule()`` ``PyImport_AddModuleRef()`` + ``PyList_GetItem()`` ``PyList_GetItemRef()`` + ==================================== ============================== + +- Omit deprecated APIs: + + ==================================== ============================== + Omitted Deprecated API Replacement + ==================================== ============================== + ``PY_FORMAT_SIZE_T`` ``"z"`` + ``PY_UNICODE_TYPE`` ``wchar_t`` + ``PyCode_GetFirstFree()`` ``PyUnstable_Code_GetFirstFree()`` + ``PyCode_New()`` ``PyUnstable_Code_New()`` + ``PyCode_NewWithPosOnlyArgs()`` ``PyUnstable_Code_NewWithPosOnlyArgs()`` + ``PyImport_ImportModuleNoBlock()`` ``PyImport_ImportModule()`` + ``PyMem_DEL()`` ``PyMem_Free()`` + ``PyMem_Del()`` ``PyMem_Free()`` + ``PyMem_FREE()`` ``PyMem_Free()`` + ``PyMem_MALLOC()`` ``PyMem_Malloc()`` + ``PyMem_NEW()`` ``PyMem_New()`` + ``PyMem_REALLOC()`` ``PyMem_Realloc()`` + ``PyMem_RESIZE()`` ``PyMem_Resize()`` + ``PyModule_GetFilename()`` ``PyModule_GetFilenameObject()`` + ``PyOS_AfterFork()`` ``PyOS_AfterFork_Child()`` + ``PyObject_DEL()`` ``PyObject_Free()`` + ``PyObject_Del()`` ``PyObject_Free()`` + ``PyObject_FREE()`` ``PyObject_Free()`` + ``PyObject_MALLOC()`` ``PyObject_Malloc()`` + ``PyObject_REALLOC()`` ``PyObject_Realloc()`` + ``PySlice_GetIndicesEx()`` (no replacement) + ``PyThread_ReInitTLS()`` (no longer needed) + ``PyThread_create_key()`` ``PyThread_tss_alloc()`` + ``PyThread_delete_key()`` ``PyThread_tss_free()`` + ``PyThread_delete_key_value()`` ``PyThread_tss_delete()`` + ``PyThread_get_key_value()`` ``PyThread_tss_get()`` + ``PyThread_set_key_value()`` ``PyThread_tss_set()`` + ``PyUnicode_AsDecodedObject()`` ``PyUnicode_Decode()`` + ``PyUnicode_AsDecodedUnicode()`` ``PyUnicode_Decode()`` + ``PyUnicode_AsEncodedObject()`` ``PyUnicode_AsEncodedString()`` + ``PyUnicode_AsEncodedUnicode()`` ``PyUnicode_AsEncodedString()`` + ``PyUnicode_IS_READY()`` (no longer needed) + ``PyUnicode_READY()`` (no longer needed) + ``PyWeakref_GET_OBJECT()`` ``PyWeakref_GetRef()`` + ``PyWeakref_GetObject()`` ``PyWeakref_GetRef()`` + ``Py_UNICODE`` ``wchar_t`` + ``_PyCode_GetExtra()`` ``PyUnstable_Code_GetExtra()`` + ``_PyCode_SetExtra()`` ``PyUnstable_Code_SetExtra()`` + ``_PyDict_GetItemStringWithError()`` ``PyDict_GetItemStringRef()`` + ``_PyEval_RequestCodeExtraIndex()`` ``PyUnstable_Eval_RequestCodeExtraIndex()`` + ``_PyHASH_BITS`` ``PyHASH_BITS`` + ``_PyHASH_IMAG`` ``PyHASH_IMAG`` + ``_PyHASH_INF`` ``PyHASH_INF`` + ``_PyHASH_MODULUS`` ``PyHASH_MODULUS`` + ``_PyHASH_MULTIPLIER`` ``PyHASH_MULTIPLIER`` + ``_PyObject_EXTRA_INIT`` (no longer needed) + ``_PyThreadState_UncheckedGet()`` ``PyThreadState_GetUnchecked()`` + ``_PyUnicode_AsString()`` ``PyUnicode_AsUTF8()`` + ``_Py_HashPointer()`` ``Py_HashPointer()`` + ``_Py_T_OBJECT`` ``Py_T_OBJECT_EX`` + ``_Py_WRITE_RESTRICTED`` (no longer needed) + ==================================== ============================== + +* Omit ```` legacy API: + + ==================================== ============================== + Omitted Deprecated API Replacement + ==================================== ============================== + ``T_SHORT`` ``Py_T_SHORT`` + ``T_INT`` ``Py_T_INT`` + ``T_LONG`` ``Py_T_LONG`` + ``T_FLOAT`` ``Py_T_FLOAT`` + ``T_DOUBLE`` ``Py_T_DOUBLE`` + ``T_STRING`` ``Py_T_STRING`` + ``T_OBJECT`` ``Py_T_OBJECT_EX`` + ``T_CHAR`` ``Py_T_CHAR`` + ``T_BYTE`` ``Py_T_BYTE`` + ``T_UBYTE`` ``Py_T_UBYTE`` + ``T_USHORT`` ``Py_T_USHORT`` + ``T_UINT`` ``Py_T_UINT`` + ``T_ULONG`` ``Py_T_ULONG`` + ``T_STRING_INPLACE`` ``Py_T_STRING_INPLACE`` + ``T_BOOL`` ``Py_T_BOOL`` + ``T_OBJECT_EX`` ``Py_T_OBJECT_EX`` + ``T_LONGLONG`` ``Py_T_LONGLONG`` + ``T_ULONGLONG`` ``Py_T_ULONGLONG`` + ``T_PYSSIZET`` ``Py_T_PYSSIZET`` + ``T_NONE`` ``_Py_T_NONE`` + ``READONLY`` ``Py_READONLY`` + ``PY_AUDIT_READ`` ``Py_AUDIT_READ`` + ``READ_RESTRICTED`` ``Py_AUDIT_READ`` + ``PY_WRITE_RESTRICTED`` (no longer needed) + ``RESTRICTED`` ``(READ_RESTRICTED | PY_WRITE_RESTRICTED)`` + ==================================== ============================== + +- Omit soft deprecated macros: + + ====================== ================ + Omitted Macros Replacement + ====================== ================ + ``Py_IS_NAN()`` ``isnan()`` + ``Py_IS_INFINITY()`` ``isinf(X)`` + ``Py_IS_FINITE()`` ``isfinite(X)`` + ``Py_MEMCPY()`` ``memcpy()`` + ====================== ================ - Typedefs without the ``Py``/``_Py`` prefix (``getter``, ``setter``, ``allocfunc``, …), omitted in favour of *new* ones From cb75c2a8541df5e01b4c46358141447358c01676 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Sun, 7 Jul 2024 18:00:12 +0200 Subject: [PATCH 3/5] Rewordings --- peps/pep-0743.rst | 184 +++++++++++++++++++++++++--------------------- 1 file changed, 101 insertions(+), 83 deletions(-) diff --git a/peps/pep-0743.rst b/peps/pep-0743.rst index b7099f0b924..4e628534812 100644 --- a/peps/pep-0743.rst +++ b/peps/pep-0743.rst @@ -18,8 +18,8 @@ soft-deprecated symbols, allowing users to opt out of using API with known issues that other API solves. The macro is versioned, allowing users to update (or not) on their own pace. -Add convenience macros ``Py_PACK_VERSION`` and ``Py_PACK_VER`` to -make it easier to express versions in C code. +Also, add namespaced alternatives for API without the ``Py_`` prefix, +and soft-deprecate the original names. Motivation @@ -27,34 +27,46 @@ Motivation Some of Python's C API has flaws that are only obvious in hindsight. -For API that prevents adding features, or presents a serious security risk -or maintenance burden, we deprecate and remove as described in :pep:`387`. +If an API prevents adding features or optimizations, or presents a serious +security risk or maintenance burden, we can deprecate and remove it as +described in :pep:`387`. -However, this leaves us with some API that has “sharp edges”, which works fine +However, this leaves us with some API that has “sharp edges” -- it works fine for its current users, but should be avoided in new code. For example: -.. (These are examples, not categories. They overlap.) - -- API that is easy to misuse or subtly dangerous, but doesn't have a scary - name; - API that cannot signal an exception, so failures are either ignored or - exit the process with a fatal error; -- API that is not thread-safe (for example by borrowing references from - mutable objects, or exposing unfinished mutable objects); + exit the process with a fatal error. For example ``PyObject_HasAttr``. +- API that is not thread-safe, for example by borrowing references from + mutable objects, or exposing unfinished mutable objects. For example + ``PyDict_GetItemWithError``. - API with names that don't use the ``Py``/``_Py`` prefix, and so can clash with other code. For example: ``setter``. -XXX: add examples - It is important to note that despite such flaws, it's usually possible to use the API correctly. For example, in a single-threaded environment, thread safety is not an issue. We do not want to break working code, even if it uses API that would be wrong in some -- or even *most* -- other contexts. -But, it's in everyone's best interest to avoid such “undesirable” API in *new* -code, especially if a safer alternative exists. +On the other hand, we want to steer users away from such “undesirable” API +in *new* code, especially if a safer alternative exists. + + +Adding the ``Py`` prefix +------------------------ + +Some names defined in CPython headers is not namespaced: it that lacks the +``Py`` prefix (or a variant: ``_Py``, and alternative capitalizations). +For example, we declare a function type named simply ``setter``. + +While such names are not exported in the ABI (as checked by ``make smelly``), +they can clash with user code and, more importantly, with libraries linked +to third-party extensions. + +While it would be possible to provide namespaced aliases and (soft-)deprecate +these names, the only way to make them not clash with third-party code is to +not define them in Python headers at all. Rationale @@ -66,16 +78,11 @@ choose to do so. It might be be sufficient to leave this to third-party linters. For that we'd need a good way to expose a list of (soft-)deprecated API to such linters. -While adding that, we can -- rather easily -- avoid an extra tool for simple -cases, and do the linter's job directly in CPython headers. - +While adding that, we can -- rather easily -- do the linter's job directly +in CPython headers, avoiding the neel for an extra tool. Unlike Python, C makes it rather easy to limit available API -- for a whole -project or for each individual source file -- by defining a macro. - -Also, for API that lacks the ``Py`` prefix (e.g. ``setter``), hiding the name -entirely means that it cannot clash with user or library code. -A linter cannot do this. -(Of course, we could use another mechanism for just this issue). +project or for each individual source file -- by having users define +an “opt-in” macro. We already do something similar with ``Py_LIMITED_API``, which limits the available API to a subset that compiles to stable ABI. (In hindsight, we should @@ -93,18 +100,25 @@ presents a security risk or maintenance burden. This mechanism, on the other hand, is meant for cases where “we found a slightly better way of doing things” -- perhaps one that's harder to misuse, or just has a less misleading name. -(On a lighter note: the mechanism is meant for the kind of people who -configure a code quality checker to shout at them about the number of blank -lines between functions.) +(On a lighter note: many people configure a code quality checker to shout at +them about the number of blank lines between functions. Let's help them +identify more substantial “code smells”!) The proposed macro does not *change* any API definitions; it only *hides* them. So, if code compiles with the macro, it'll also compile without it, with identical behaviour. This has implications for core devs: to deal with undesirable behaviour, -we'll need to introduce new, better API, and discourage the old one. -We should look at an individual API and fix all its known issues at once, -rather than do codebase-wide sweeps for a single kind of issue, -which would result in multiple renames of the same function. +we'll need to introduce new, better API, and *then* discourage the old one. +In trun, this implies that we should look at an individual API and fix all its +known issues at once, rather than do codebase-wide sweeps for a single kind of +issue, so that we avoid multiple renames of the same function. + + +Adding the ``Py`` prefix +------------------------ + +An opt-in macro allows us to omit definitions that could clash with +third-party libraries. Specification @@ -114,7 +128,7 @@ We introduce a ``Py_COMPAT_API_VERSION`` macro. If this macro is defined before ``#include ``, some API definitions -- as described below -- will be omitted from the Python header files. -The macro only hides top-level definitions exposed from ````. +The macro only omits complete top-level definitions exposed from ````. Other things (the ABI, structure definitions, macro expansions, static inline function bodies, etc.) are not affected. @@ -127,8 +141,8 @@ In rare cases, entries can be removed (i.e. made available for use) at any time. The macro should be defined to a version in the format used by -``PY_VERSION_HEX``, with the “micro”, “release” and “serial” fields do not -matter and should be set to zero. +``PY_VERSION_HEX``, with the “micro”, “release” and “serial” fields +set to zero. For example, to omit API deemed undesirable in 3.14.0b1, users should define ``Py_COMPAT_API_VERSION`` to ``0x030e0000``. @@ -136,8 +150,7 @@ For example, to omit API deemed undesirable in 3.14.0b1, users should define Requirements for omitted API ---------------------------- -An API that is omitted with ``Py_COMPAT_API_VERSION`` must be deprecated -(see :pep:`387`), or: +An API that is omitted with ``Py_COMPAT_API_VERSION`` must: - be soft-deprecated (see :pep:`387`); - for all known use cases of the API, have a documented alternative @@ -146,10 +159,11 @@ An API that is omitted with ``Py_COMPAT_API_VERSION`` must be deprecated ``#define`` or ``typedef``); - be documented (except if it was never mentioned in previous versions of the documentation); and -- be approved by the C API working group. +- be approved by the C API working group. (The WG may give blanket approvals + for groups of related API; see *Initial set* below for examples.) -The mechanism is meant for API that can be trivially replaced by a better -alternative. +Note that ``Py_COMPAT_API_VERSION`` is meant for API that can be trivially +replaced by a better alternative. API without a replacement should generally be deprecated instead. @@ -172,9 +186,13 @@ or ``#define`` for the old name added to ``Include/legacy.h``. Documentation ------------- -Documentation for omitted API should generally appear after the recommended -replacement, reference it (e.g. “Similar to X, but…”), and focus on differences -from the replacement and migration advice. +Documentation for omitted API should generally: + +- appear after the recommended replacement, +- reference the replacement (e.g. “Similar to X, but…”), and +- focus on differences from the replacement and migration advice. + +Exceptions are possible if there is a good reason for them. Initial set @@ -219,7 +237,7 @@ The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to ``PyObject_FREE()`` ``PyObject_Free()`` ``PyObject_MALLOC()`` ``PyObject_Malloc()`` ``PyObject_REALLOC()`` ``PyObject_Realloc()`` - ``PySlice_GetIndicesEx()`` (no replacement) + ``PySlice_GetIndicesEx()`` (two calls; see current docs) ``PyThread_ReInitTLS()`` (no longer needed) ``PyThread_create_key()`` ``PyThread_tss_alloc()`` ``PyThread_delete_key()`` ``PyThread_tss_free()`` @@ -252,7 +270,25 @@ The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to ``_Py_WRITE_RESTRICTED`` (no longer needed) ==================================== ============================== -* Omit ```` legacy API: +- Soft-deprecate and omit APIs: + + ==================================== ============================== + Omitted Deprecated API Replacement + ==================================== ============================== + ``PyDict_GetItemWithError()`` ``PyDict_GetItemRef()`` + ``PyDict_SetDefault()`` ``PyDict_SetDefaultRef()`` + ``PyMapping_HasKey()`` ``PyMapping_HasKeyWithError()`` + ``PyMapping_HasKeyString()`` ``PyMapping_HasKeyStringWithError()`` + ``PyObject_HasAttr()`` ``PyObject_HasAttrWithError()`` + ``PyObject_HasAttrString()`` ``PyObject_HasAttrStringWithError()`` + ==================================== ============================== + +- Omit ```` legacy API: + + The header file ``structmember.h``, which is not included from ```` + and must be included separately, will ``#error`` if + ``Py_COMPAT_API_VERSION`` is defined. + This affects the following API: ==================================== ============================== Omitted Deprecated API Replacement @@ -263,7 +299,7 @@ The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to ``T_FLOAT`` ``Py_T_FLOAT`` ``T_DOUBLE`` ``Py_T_DOUBLE`` ``T_STRING`` ``Py_T_STRING`` - ``T_OBJECT`` ``Py_T_OBJECT_EX`` + ``T_OBJECT`` (``tp_getset``; docs to be written) ``T_CHAR`` ``Py_T_CHAR`` ``T_BYTE`` ``Py_T_BYTE`` ``T_UBYTE`` ``Py_T_UBYTE`` @@ -276,59 +312,41 @@ The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to ``T_LONGLONG`` ``Py_T_LONGLONG`` ``T_ULONGLONG`` ``Py_T_ULONGLONG`` ``T_PYSSIZET`` ``Py_T_PYSSIZET`` - ``T_NONE`` ``_Py_T_NONE`` + ``T_NONE`` (``tp_getset``; docs to be written) ``READONLY`` ``Py_READONLY`` ``PY_AUDIT_READ`` ``Py_AUDIT_READ`` ``READ_RESTRICTED`` ``Py_AUDIT_READ`` ``PY_WRITE_RESTRICTED`` (no longer needed) - ``RESTRICTED`` ``(READ_RESTRICTED | PY_WRITE_RESTRICTED)`` + ``RESTRICTED`` ``Py_AUDIT_READ`` ==================================== ============================== - Omit soft deprecated macros: - ====================== ================ + ====================== ===================================== Omitted Macros Replacement - ====================== ================ - ``Py_IS_NAN()`` ``isnan()`` - ``Py_IS_INFINITY()`` ``isinf(X)`` - ``Py_IS_FINITE()`` ``isfinite(X)`` - ``Py_MEMCPY()`` ``memcpy()`` - ====================== ================ - -- Typedefs without the ``Py``/``_Py`` prefix - (``getter``, ``setter``, ``allocfunc``, …), omitted in favour of *new* ones + ====================== ===================================== + ``Py_IS_NAN()`` ``isnan()`` (C99+ ````) + ``Py_IS_INFINITY()`` ``isinf(X)`` (C99+ ````) + ``Py_IS_FINITE()`` ``isfinite(X)`` (C99+ ````) + ``Py_MEMCPY()`` ``memcpy()`` (C ````) + ====================== ===================================== + +- Soft-deprecate and omit typedefs without the ``Py``/``_Py`` prefix + (``getter``, ``setter``, ``allocfunc``, …), in favour of *new* ones that add the prefix (``Py_getter`` , etc.) -- Macros without the ``Py``/``_Py`` prefix +- Soft-deprecate and omit macros without the ``Py``/``_Py`` prefix (``METH_O``, ``CO_COROUTINE``, ``FUTURE_ANNOTATIONS``, ``WAIT_LOCK``, …), - omitted in favour of *new* ones that add the prefix (``Py_METH_O`` , etc.). - - - Replacements for macros generated by ``configure`` - (``HAVE_*``, ``WITH_*``, ``ALIGNOF_*``, ``SIZEOF_*``, and several without - a common prefix) may not be ready for 3.14; if so they'll be omitted - in later versions. + favour of *new* ones that add the prefix (``Py_METH_O`` , etc.). - Any others approved by the C API workgroup -Helper macros -------------- - -To make it easier to set and test values for ``Py_COMPAT_API_VERSION``, -``Py_LIMITED_API``, and other versions, we will add two helper macros: - -- ``Py_PACK_VERSION(3, y, z, level, serial)`` packs the version - as an integer in the format used by ``PY_VERSION_HEX``. - For example, ``Py_PACK_VERSION(3, 4, 1, 0xA, 2)`` gives ``0x030401a2``. - -- ``Py_PACK_VER(3, y)`` is shorthand for ``Py_PACK_VERSION(3, y, 0, 0, 0)``, - useful because the first two version components often determine ABI - compatibility. -Library functions with the sames and functionality will be exported, for use in -auto-generated wrappers for non-C languages. -(The macro-style naming means that we encourage implementing them as -compile-time constructs, rather than library calls -- but that's harder to -auto-generate.) +If any of these proposed replacements, or associated documentation, +are not added in time for 3.14.0b1, they'll be omitted with later versions +of ``Py_COMPAT_API_VERSION``. +(We expect this for macros generated by ``configure``: ``HAVE_*``, ``WITH_*``, +``ALIGNOF_*``, ``SIZEOF_*``, and several without a common prefix.) Implementation From 690b6775a994f8f4bea6fd7e115de7f5b6147659 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 23 Jul 2024 11:54:40 +0200 Subject: [PATCH 4/5] Typo & formatting --- peps/pep-0743.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0743.rst b/peps/pep-0743.rst index 4e628534812..3e202c88b48 100644 --- a/peps/pep-0743.rst +++ b/peps/pep-0743.rst @@ -109,7 +109,7 @@ So, if code compiles with the macro, it'll also compile without it, with identical behaviour. This has implications for core devs: to deal with undesirable behaviour, we'll need to introduce new, better API, and *then* discourage the old one. -In trun, this implies that we should look at an individual API and fix all its +In turn, this implies that we should look at an individual API and fix all its known issues at once, rather than do codebase-wide sweeps for a single kind of issue, so that we avoid multiple renames of the same function. @@ -380,7 +380,7 @@ Discussions with no known issues `_ (June 2023) -* Finishing the Great Renaming +* `Finishing the Great Renaming `_ (May 2024) From 75ad1d67a63bd96110e68ae2f970d31ebc3f110b Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 23 Jul 2024 12:06:16 +0200 Subject: [PATCH 5/5] Add maself to CODEOWNERS --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 06a7a38b565..cd7679821c2 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -621,7 +621,7 @@ peps/pep-0738.rst @encukou peps/pep-0740.rst @dstufft peps/pep-0741.rst @vstinner peps/pep-0742.rst @JelleZijlstra -peps/pep-0743.rst @vstinner +peps/pep-0743.rst @vstinner @encukou peps/pep-0744.rst @brandtbucher peps/pep-0745.rst @hugovk peps/pep-0746.rst @JelleZijlstra