From 835d0a11cca78deb668ea9e1ced810df2f7e377b Mon Sep 17 00:00:00 2001 From: Joshua Date: Sun, 9 Apr 2023 20:26:08 -0500 Subject: [PATCH 01/21] Add dataclass converter PEP --- .github/CODEOWNERS | 1 + pep-9999.rst | 267 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 268 insertions(+) create mode 100644 pep-9999.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index bdba43cade6..f657d27158a 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -671,3 +671,4 @@ pep-8016.rst @njsmith @dstufft pep-8100.rst @njsmith # pep-8101.rst # pep-8102.rst +pep-9999.rst @thejcannon \ No newline at end of file diff --git a/pep-9999.rst b/pep-9999.rst new file mode 100644 index 00000000000..a38ac95d0dd --- /dev/null +++ b/pep-9999.rst @@ -0,0 +1,267 @@ +PEP: 9999 +Title: Adding "converter" dataclasses field specifier parameter +Author: Joshua Cannon +Sponsor: Eric V. Smith +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 01-Jan-2023 + + +Abstract +======== + +:pep:`557` added dataclasses to the Python stdlib. :pep:`681` added +``dataclass_transform`` to help type checkers understand several common +dataclass-like libraries, such as ``attrs``, ``pydantic``, and object +relational mapper (ORM) packages such as SQLAlchemy and Django. + +A common feature these libraries provide over the standard library +implementation is the ability for the library to convert arguments given at +initialization time into the types expected for each field using a +user-provided conversion function. + +Motivation +========== + +There is no existing, standard way for ``dataclass`` or third-party +dataclass-like libraries to support argument conversion in a type-checkable +way. To workaround this limitation, library authors/users are forced to choose +to: + +* Opt-in to a custom Mypy plugin. These plugins help Mypy understand the +conversion semantics, but not other tools. +* Shuck conversion responsibility onto the caller of the ``dataclass`` +constructor. This can make constructing certain ``dataclasses`` unnecessarily +verbose and repetitive. +* Provide a custom ``__init__`` and which declares "wider" parameter types and +converts them when setting the appropriate attribute. This not only duplicates +the typing annotations between the converter and ``__init__``, but also opts +the user out of many of the features ``dataclass`` provides. +* Not rely on, or ignore type-checking. + +None of these choices are ideal. + +Rationale +========= + +Adding argument conversion semantics is useful and beneficial enough that most +dataclass-like libraries provide support. Adding this feature to the standard +library means more users are able to opt-in to these benefits without requiring +third-party libraries. Additionally third-party libraries are able to clue +type-checkers into their own conversion semantics through added support in +``dataclass_transform``, meaning users of those libraries benefit as well. + +Specification +============= + +New ``converter`` parameter +--------------------------- + +This specification introduces a new parameter named ``converter`` to +``dataclasses.field`` function. When an ``__init__`` method is synthesized by +``dataclass``-like semantics, if an argument is provided for the field, the +``dataclass`` object's attribute will be assigned the result of calling the +converter with a single argument: the provided argument. If no argument is +given, the normal ``dataclass`` semantics for defaulting the attribute value +is used and conversion is not applied to the default value. + +Adding this parameter also implies the following changes: + +* A ``converter`` attribute will be added to ``dataclasses.Field``. +* Adds ``converter`` to the field specifier parameters of arguments provided to +``typing.dataclass_transform``'s ``field`` parameter. + +Example +''''''' + +.. code-block:: python + + @dataclasses.dataclass + class InventoryItem: + # `converter` as a type + id: int = dataclasses.field(converter=int) + skus: tuple[int] = dataclasses.field(converter=tuple[int]) + # `converter` as a callable + names: tuple[str] = dataclasses.field( + converter=lambda names: tuple(map(str.lower, names)) + ) + + # Since the value is not converted, type checkers should flag the default + # as having the wrong type. + # There is no error at runtime however, and `quantity_on_hand` will be + # `"0"` if no value is provided. + quantity_on_hand: int = dataclasses.field(converter=int, default="0") + + item1 = InventoryItem("1", [234, 765], ["PYTHON PLUSHIE", "FLUFFY SNAKE"]) + # `item1` would have the following values: + # id=1 + # skus=(234, 765) + # names=('python plushie', 'fluffy snake') + # quantity_on_hand='0' + +Impact on typing +---------------- + +``converter`` arguments are expected to be callable objects which accept a +unary argument and return a type compatible with the field's annotated type. +The callable's unary argument's type is used as the type of the parameter in +the synthesized ``__init__`` method. + +Type-narrowing the argument type +'''''''''''''''''''''''''''''''' + +For the purpose of deducing the type of the argument in the synthesized +``__init__`` method, the ``converter`` argument's type can be "narrowed" using +the following rules: + +* If the ``converter`` is of type ``Any``, it is assumed to be callable with a +unary ``Any`` typed-argument. +* All keyword-only parameters can be ignored. +* ``**kwargs`` can be ignored. +* ``*args`` can be ignored if any parameters precede it. Otherwise if ``*args`` +is the only non-ignored parameter, the type it accepts for each positional +argument is the type of the unary argument. E.g. given params +``(x: str, *args: str)``, ``*args`` can be ignored. However, given params +``(*args: str)``, the callable type can be narrowed to ``(__x: str, /)``. +* Parameters with default values that aren't the first parameter can be +ignored. E.g. given params ``(x: str = "0", y: int = 1)``, parameter ``y`` can +be ignored and the type can be assumed to be ``(x: str)``. + +Type-checking the return type +''''''''''''''''''''''''''''' + +The return type of the callable must be a type that's compatible with the +field's declared type. This includes the field's type exactly, but can also be +a type that's more specialized (such as a converter returning a ``list[int]`` +for a field annotated as ``list``, or a converter returning an ``int`` for a +field annotated as ``int | str``). + +Overloads +''''''''' + +The above rules should be applied to each ``@overload`` for overloaded +functions. If after these rules are applied an overload is invalid (either +because there is no overload that would accept a unary argument, or because +there is no overload that returns an acceptable type) it should be ignored. +If multiple overloads are valid after these rules are applied, the +type-checker can assume the converter's unary argument type is the union of +each overload's unary argument type. If no overloads are valid, it is a type +error. + +Example +''''''' + +.. code-block:: python + + # The following are valid converter types, with a comment containing the + # synthesized __init__ argument's type. + converter: Any # Any + def converter(x: int): ... # int + def converter(x: int | str): ... # int | str + def converter(x: int, y: str = "a"): ... # int + def converter(x: int, *args: str): ... # int + def converter(*args: str): ... # str + def converter(*args: str, x: int = 0): ... # str + + @overload + def converter(x: int): ... # <- valid + @overload + def converter(x: int, y: str): ... # <- ignored + @overload + def converter(x: list): ... # <- valid + def converter(x, y = ...): ... # int | list + + # The following are valid converter types for a field annotated as type `list`. + def converter(x) -> list: ... + def converter(x) -> Any: ... + def converter(x) -> list[int]: ... + + @overload + def converter(x: int) -> tuple: ... # <- ignored + @overload + def converter(x: str) -> list: ... # <- valid + @overload + def converter(x: bytes) -> list: ... # <- valid + def converter(x): ... # __init__ would use argument type `str | bytes`. + + # The following are invalid converter types. + def converter(): ... + def converter(**kwargs): ... + def converter(x, y): ... + def converter(*, x): ... + def converter(*args, x): ... + + @overload + def converter(): ... + @overload + def converter(x: int, y: str): ... + def converter(x=..., y = ...): ... + + # The following are invalid converter types for a field annotated as type `list`. + def converter(x) -> tuple: ... + def converter(x) -> Sequence: ... + + @overload + def converter(x) -> tuple: ... + @overload + def converter(x: int, y: str) -> list: ... + def converter(x=..., y = ...): ... + + +Reference Implementation +======================== + +The `attrs <#attrs-converters>`_ library already includes a ``converter`` +parameter matching these +semantics. + +The reference implementation + +Rejected Ideas +============== + +Just adding "converter" to ``dataclass_transform``'s ``field_specifiers`` +------------------------------------------------------------------------- + +The idea of isolating this addition to ``dataclass_transform`` was briefly +discussed in `Typing-sig <#only-dataclass-transform>`_ where it was suggested +to open this to ``dataclasses``. + +Additionally, adding this to ``dataclasses`` ensures anyone can reap the +benefits without requiring additional libraries. + +Automatic conversion using the field's type +------------------------------------------- + +One idea could be to allow the type of the field specified (e.g. ``str`` or +``int``) to be used as a converter for each argument provided. +`Pydantic's data conversion <#pydantic-data-conversion>`_ has semantics which +appear to be similar to this approach. + +This works well for fairly simple types, but leads to ambiguity in expected +behavior for complex types such as generics. E.g. For ``tuple[int]`` it is +ambiguous if the converter is supposed to simply convert an iterable to a tuple, +or if it is additionally supposed to convert each element type to ``int``. + +Converting the default values +----------------------------- + +Having the synthesized ``__init__`` also convert the default values (such as +``default`` or the return type of ``default_factory``) when the would make the +expected type of these parameters complex for type-checkers, and does not add +significant value. + +References +========== +.. _#typeshed: https://github.com/python/typeshed +.. _#attrs-converters: https://www.attrs.org/en/21.2.0/examples.html#conversion +.. _#only-dataclass-transform: https://mail.python.org/archives/list/typing-sig@python.org/thread/NWZQIINJQZDOCZGO6TGCUP2PNW4PEKNY/ +.. _#pydantic-data-conversion: https://docs.pydantic.dev/usage/models/#data-conversion + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From d84dd26b4fbbeb6c04a1d5c29a82bcfa79fc4275 Mon Sep 17 00:00:00 2001 From: Joshua Date: Sun, 9 Apr 2023 20:28:54 -0500 Subject: [PATCH 02/21] Add Eric as CODEOWNER --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index f657d27158a..83d92264a53 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -671,4 +671,4 @@ pep-8016.rst @njsmith @dstufft pep-8100.rst @njsmith # pep-8101.rst # pep-8102.rst -pep-9999.rst @thejcannon \ No newline at end of file +pep-9999.rst @thejcannon @ericvsmith \ No newline at end of file From 1431bbb8d59506abff59dadd2e721f25b59b90e9 Mon Sep 17 00:00:00 2001 From: Joshua Date: Sun, 9 Apr 2023 20:38:53 -0500 Subject: [PATCH 03/21] fix lint errors --- pep-9999.rst | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/pep-9999.rst b/pep-9999.rst index a38ac95d0dd..1a63211c3a6 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -30,14 +30,14 @@ way. To workaround this limitation, library authors/users are forced to choose to: * Opt-in to a custom Mypy plugin. These plugins help Mypy understand the -conversion semantics, but not other tools. + conversion semantics, but not other tools. * Shuck conversion responsibility onto the caller of the ``dataclass`` -constructor. This can make constructing certain ``dataclasses`` unnecessarily -verbose and repetitive. + constructor. This can make constructing certain ``dataclasses`` unnecessarily + verbose and repetitive. * Provide a custom ``__init__`` and which declares "wider" parameter types and -converts them when setting the appropriate attribute. This not only duplicates -the typing annotations between the converter and ``__init__``, but also opts -the user out of many of the features ``dataclass`` provides. + converts them when setting the appropriate attribute. This not only duplicates + the typing annotations between the converter and ``__init__``, but also opts + the user out of many of the features ``dataclass`` provides. * Not rely on, or ignore type-checking. None of these choices are ideal. @@ -70,7 +70,7 @@ Adding this parameter also implies the following changes: * A ``converter`` attribute will be added to ``dataclasses.Field``. * Adds ``converter`` to the field specifier parameters of arguments provided to -``typing.dataclass_transform``'s ``field`` parameter. + ``typing.dataclass_transform``'s ``field`` parameter. Example ''''''' @@ -94,7 +94,7 @@ Example quantity_on_hand: int = dataclasses.field(converter=int, default="0") item1 = InventoryItem("1", [234, 765], ["PYTHON PLUSHIE", "FLUFFY SNAKE"]) - # `item1` would have the following values: + # item1 would have the following values: # id=1 # skus=(234, 765) # names=('python plushie', 'fluffy snake') @@ -116,17 +116,17 @@ For the purpose of deducing the type of the argument in the synthesized the following rules: * If the ``converter`` is of type ``Any``, it is assumed to be callable with a -unary ``Any`` typed-argument. + unary ``Any`` typed-argument. * All keyword-only parameters can be ignored. * ``**kwargs`` can be ignored. * ``*args`` can be ignored if any parameters precede it. Otherwise if ``*args`` -is the only non-ignored parameter, the type it accepts for each positional -argument is the type of the unary argument. E.g. given params -``(x: str, *args: str)``, ``*args`` can be ignored. However, given params -``(*args: str)``, the callable type can be narrowed to ``(__x: str, /)``. + is the only non-ignored parameter, the type it accepts for each positional + argument is the type of the unary argument. E.g. given params + ``(x: str, *args: str)``, ``*args`` can be ignored. However, given params + ``(*args: str)``, the callable type can be narrowed to ``(__x: str, /)``. * Parameters with default values that aren't the first parameter can be -ignored. E.g. given params ``(x: str = "0", y: int = 1)``, parameter ``y`` can -be ignored and the type can be assumed to be ``(x: str)``. + ignored. E.g. given params ``(x: str = "0", y: int = 1)``, parameter ``y`` + can be ignored and the type can be assumed to be ``(x: str)``. Type-checking the return type ''''''''''''''''''''''''''''' @@ -172,7 +172,7 @@ Example def converter(x: list): ... # <- valid def converter(x, y = ...): ... # int | list - # The following are valid converter types for a field annotated as type `list`. + # The following are valid converter types for a field annotated as type 'list'. def converter(x) -> list: ... def converter(x) -> Any: ... def converter(x) -> list[int]: ... @@ -183,7 +183,7 @@ Example def converter(x: str) -> list: ... # <- valid @overload def converter(x: bytes) -> list: ... # <- valid - def converter(x): ... # __init__ would use argument type `str | bytes`. + def converter(x): ... # __init__ would use argument type 'str | bytes'. # The following are invalid converter types. def converter(): ... @@ -198,7 +198,7 @@ Example def converter(x: int, y: str): ... def converter(x=..., y = ...): ... - # The following are invalid converter types for a field annotated as type `list`. + # The following are invalid converter types for a field annotated as type 'list'. def converter(x) -> tuple: ... def converter(x) -> Sequence: ... From 4797b36c04ec1e6aba7162df1ace4fe29bc1ad10 Mon Sep 17 00:00:00 2001 From: Joshua Date: Sun, 9 Apr 2023 20:45:05 -0500 Subject: [PATCH 04/21] Fixup Reference Implementation --- pep-9999.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-9999.rst b/pep-9999.rst index 1a63211c3a6..04fcbf73891 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -213,10 +213,9 @@ Reference Implementation ======================== The `attrs <#attrs-converters>`_ library already includes a ``converter`` -parameter matching these -semantics. +parameter containing converter semantics. -The reference implementation +CPython support can be seen on a branch: `GitHub <#attrs-converters>`. Rejected Ideas ============== @@ -256,6 +255,7 @@ References ========== .. _#typeshed: https://github.com/python/typeshed .. _#attrs-converters: https://www.attrs.org/en/21.2.0/examples.html#conversion +.. _#cpython-branch: https://github.com/thejcannon/cpython/tree/converter .. _#only-dataclass-transform: https://mail.python.org/archives/list/typing-sig@python.org/thread/NWZQIINJQZDOCZGO6TGCUP2PNW4PEKNY/ .. _#pydantic-data-conversion: https://docs.pydantic.dev/usage/models/#data-conversion From eff261c5e85ef7d8862c8f1150a2708481dd0085 Mon Sep 17 00:00:00 2001 From: Joshua Date: Mon, 10 Apr 2023 08:37:34 -0500 Subject: [PATCH 05/21] Rename to PEP 712 --- .github/CODEOWNERS | 4 ++-- pep-9999.rst => pep-0712.rst | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) rename pep-9999.rst => pep-0712.rst (99%) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 83d92264a53..2adb0b77064 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -584,6 +584,7 @@ pep-0700.rst @pfmoore pep-0701.rst @pablogsal @isidentical @lysnikolaou pep-0702.rst @jellezijlstra pep-0703.rst @ambv +pep-0712.rst @ericvsmith # ... # pep-0754.txt # ... @@ -670,5 +671,4 @@ pep-8016.rst @njsmith @dstufft # ... pep-8100.rst @njsmith # pep-8101.rst -# pep-8102.rst -pep-9999.rst @thejcannon @ericvsmith \ No newline at end of file +# pep-8102.rst \ No newline at end of file diff --git a/pep-9999.rst b/pep-0712.rst similarity index 99% rename from pep-9999.rst rename to pep-0712.rst index 04fcbf73891..3ec3c6c1127 100644 --- a/pep-9999.rst +++ b/pep-0712.rst @@ -1,4 +1,4 @@ -PEP: 9999 +PEP: 712 Title: Adding "converter" dataclasses field specifier parameter Author: Joshua Cannon Sponsor: Eric V. Smith From 5f6ff86ee236234348ada91f4725f6b1e21b9f53 Mon Sep 17 00:00:00 2001 From: Joshua Date: Mon, 10 Apr 2023 08:38:04 -0500 Subject: [PATCH 06/21] add missing underscore --- pep-0712.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0712.rst b/pep-0712.rst index 3ec3c6c1127..963f066f251 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -215,7 +215,7 @@ Reference Implementation The `attrs <#attrs-converters>`_ library already includes a ``converter`` parameter containing converter semantics. -CPython support can be seen on a branch: `GitHub <#attrs-converters>`. +CPython support can be seen on a branch: `GitHub <#attrs-converters>`_. Rejected Ideas ============== From a73569b9a12a3ab6fa71ca1262d518e0ad3cc07e Mon Sep 17 00:00:00 2001 From: Joshua Date: Tue, 11 Apr 2023 09:32:56 -0500 Subject: [PATCH 07/21] review feedback --- .github/CODEOWNERS | 2 +- pep-0712.rst | 102 ++++++++++++++++++++++++--------------------- 2 files changed, 56 insertions(+), 48 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 2adb0b77064..5a99ea2361d 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -671,4 +671,4 @@ pep-8016.rst @njsmith @dstufft # ... pep-8100.rst @njsmith # pep-8101.rst -# pep-8102.rst \ No newline at end of file +# pep-8102.rst diff --git a/pep-0712.rst b/pep-0712.rst index 963f066f251..39b96e34e4d 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -1,19 +1,19 @@ PEP: 712 -Title: Adding "converter" dataclasses field specifier parameter +Title: Adding a "converter" parameter to dataclasses.field Author: Joshua Cannon Sponsor: Eric V. Smith Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 01-Jan-2023 - +Post-History: `27-Dec-2022 `__, `19-Jan-2023 `__ Abstract ======== -:pep:`557` added dataclasses to the Python stdlib. :pep:`681` added +:pep:`557` added :mod:`dataclasses` to the Python stdlib. :pep:`681` added ``dataclass_transform`` to help type checkers understand several common -dataclass-like libraries, such as ``attrs``, ``pydantic``, and object +dataclass-like libraries, such as attrs, Pydantic, and object relational mapper (ORM) packages such as SQLAlchemy and Django. A common feature these libraries provide over the standard library @@ -21,23 +21,29 @@ implementation is the ability for the library to convert arguments given at initialization time into the types expected for each field using a user-provided conversion function. +Therefore, this PEP adds a ``converter`` parameter to :func:`dataclasses.field` +(along with the requisite changes to :class:`dataclasses.Field` and +:func:`py3.11:~typing.dataclass_transform` to specify the function to use to +convert the input value for each field to the representation to be stored in +the dataclass. + Motivation ========== -There is no existing, standard way for ``dataclass`` or third-party +There is no existing, standard way for :mod:`dataclasses` or third-party dataclass-like libraries to support argument conversion in a type-checkable -way. To workaround this limitation, library authors/users are forced to choose +way. To work around this limitation, library authors/users are forced to choose to: * Opt-in to a custom Mypy plugin. These plugins help Mypy understand the conversion semantics, but not other tools. -* Shuck conversion responsibility onto the caller of the ``dataclass`` - constructor. This can make constructing certain ``dataclasses`` unnecessarily +* Shift conversion responsibility onto the caller of the dataclass + constructor. This can make constructing certain dataclasses unnecessarily verbose and repetitive. * Provide a custom ``__init__`` and which declares "wider" parameter types and converts them when setting the appropriate attribute. This not only duplicates the typing annotations between the converter and ``__init__``, but also opts - the user out of many of the features ``dataclass`` provides. + the user out of many of the features :mod:`dataclasses` provides. * Not rely on, or ignore type-checking. None of these choices are ideal. @@ -50,7 +56,8 @@ dataclass-like libraries provide support. Adding this feature to the standard library means more users are able to opt-in to these benefits without requiring third-party libraries. Additionally third-party libraries are able to clue type-checkers into their own conversion semantics through added support in -``dataclass_transform``, meaning users of those libraries benefit as well. +:func:`py3.11:~typing.dataclass_transform`, meaning users of those libraries +benefit as well. Specification ============= @@ -58,19 +65,19 @@ Specification New ``converter`` parameter --------------------------- -This specification introduces a new parameter named ``converter`` to -``dataclasses.field`` function. When an ``__init__`` method is synthesized by +This specification introduces a new parameter named ``converter`` to the +:func:`dataclasses.field` function. When an ``__init__`` method is synthesized by ``dataclass``-like semantics, if an argument is provided for the field, the ``dataclass`` object's attribute will be assigned the result of calling the -converter with a single argument: the provided argument. If no argument is -given, the normal ``dataclass`` semantics for defaulting the attribute value -is used and conversion is not applied to the default value. +converter on the provided argument. If no argument is given and the field was +constructed with a default value, the ``dataclass`` object's attribute will be +assigned the result of calling the converter on the provided default. Adding this parameter also implies the following changes: -* A ``converter`` attribute will be added to ``dataclasses.Field``. -* Adds ``converter`` to the field specifier parameters of arguments provided to - ``typing.dataclass_transform``'s ``field`` parameter. +* A ``converter`` attribute will be added to :class:`dataclasses.Field`. +* ``converter`` will be added to the field specifier parameters of arguments + provided to :func:`py3.11:~typing.dataclass_transform`'s ``field`` parameter. Example ''''''' @@ -81,24 +88,25 @@ Example class InventoryItem: # `converter` as a type id: int = dataclasses.field(converter=int) - skus: tuple[int] = dataclasses.field(converter=tuple[int]) + skus: tuple[int, ...] = dataclasses.field(converter=tuple[int]) # `converter` as a callable - names: tuple[str] = dataclasses.field( + names: tuple[str, ...] = dataclasses.field( converter=lambda names: tuple(map(str.lower, names)) ) - # Since the value is not converted, type checkers should flag the default - # as having the wrong type. - # There is no error at runtime however, and `quantity_on_hand` will be - # `"0"` if no value is provided. - quantity_on_hand: int = dataclasses.field(converter=int, default="0") + # The default value is also converted, therefore the following is not a + # type error. + stock_image_path: pathlib.PurePosixPath = dataclasses.field( + converter=pathlib.PurePosixPath, default="assets/unknown.png" + ) item1 = InventoryItem("1", [234, 765], ["PYTHON PLUSHIE", "FLUFFY SNAKE"]) # item1 would have the following values: # id=1 # skus=(234, 765) # names=('python plushie', 'fluffy snake') - # quantity_on_hand='0' + # stock_image_path=pathlib.PurePosixPath("assets/unknown.png") + Impact on typing ---------------- @@ -128,6 +136,14 @@ the following rules: ignored. E.g. given params ``(x: str = "0", y: int = 1)``, parameter ``y`` can be ignored and the type can be assumed to be ``(x: str)``. +Type-checking the default value +''''''''''''''''''''''''''''''' + +Because the ``default`` value is unconditionally converted using ``converter``, +if arguments for both ``converter`` and ``default`` are provided to +:func:`dataclasses.field`, the ``default`` argument's type should be checked +using the ``converter``'s unary argument's type. + Type-checking the return type ''''''''''''''''''''''''''''' @@ -212,22 +228,23 @@ Example Reference Implementation ======================== -The `attrs <#attrs-converters>`_ library already includes a ``converter`` +The attrs library `already includes `__ a ``converter`` parameter containing converter semantics. -CPython support can be seen on a branch: `GitHub <#attrs-converters>`_. +CPython support is implemented on `a branch in the author's fork `__. Rejected Ideas ============== -Just adding "converter" to ``dataclass_transform``'s ``field_specifiers`` +Just adding "converter" to :func:`py3.11:~typing.dataclass_transform`'s ``field_specifiers`` ------------------------------------------------------------------------- -The idea of isolating this addition to ``dataclass_transform`` was briefly -discussed in `Typing-sig <#only-dataclass-transform>`_ where it was suggested -to open this to ``dataclasses``. +The idea of isolating this addition to +:func:`py3.11:~typing.dataclass_transform` was briefly +`discussed on Typing-SIG `__ where it was suggested +to broaden this to :mod:`dataclasses` more generally. -Additionally, adding this to ``dataclasses`` ensures anyone can reap the +Additionally, adding this to :mod:`dataclasses` ensures anyone can reap the benefits without requiring additional libraries. Automatic conversion using the field's type @@ -235,7 +252,7 @@ Automatic conversion using the field's type One idea could be to allow the type of the field specified (e.g. ``str`` or ``int``) to be used as a converter for each argument provided. -`Pydantic's data conversion <#pydantic-data-conversion>`_ has semantics which +`Pydantic's data conversion `__ has semantics which appear to be similar to this approach. This works well for fairly simple types, but leads to ambiguity in expected @@ -243,21 +260,12 @@ behavior for complex types such as generics. E.g. For ``tuple[int]`` it is ambiguous if the converter is supposed to simply convert an iterable to a tuple, or if it is additionally supposed to convert each element type to ``int``. -Converting the default values ------------------------------ - -Having the synthesized ``__init__`` also convert the default values (such as -``default`` or the return type of ``default_factory``) when the would make the -expected type of these parameters complex for type-checkers, and does not add -significant value. - References ========== -.. _#typeshed: https://github.com/python/typeshed -.. _#attrs-converters: https://www.attrs.org/en/21.2.0/examples.html#conversion -.. _#cpython-branch: https://github.com/thejcannon/cpython/tree/converter -.. _#only-dataclass-transform: https://mail.python.org/archives/list/typing-sig@python.org/thread/NWZQIINJQZDOCZGO6TGCUP2PNW4PEKNY/ -.. _#pydantic-data-conversion: https://docs.pydantic.dev/usage/models/#data-conversion +.. _attrs-converters: https://www.attrs.org/en/21.2.0/examples.html#conversion +.. _cpython-branch: https://github.com/thejcannon/cpython/tree/converter +.. _only-dataclass-transform: https://mail.python.org/archives/list/typing-sig@python.org/thread/NWZQIINJQZDOCZGO6TGCUP2PNW4PEKNY/ +.. _pydantic-data-conversion: https://docs.pydantic.dev/usage/models/#data-conversion Copyright From 18ae4d38daa3780dd36ed3209af466376fc35c6f Mon Sep 17 00:00:00 2001 From: Joshua Date: Tue, 11 Apr 2023 09:38:33 -0500 Subject: [PATCH 08/21] Add Python version --- pep-0712.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/pep-0712.rst b/pep-0712.rst index 39b96e34e4d..56776f509dd 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -6,6 +6,7 @@ Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 01-Jan-2023 +Python-Version: 3.13 Post-History: `27-Dec-2022 `__, `19-Jan-2023 `__ Abstract From 1ae265ca15263f3f8506e5b755b513747de97e43 Mon Sep 17 00:00:00 2001 From: Joshua Date: Tue, 11 Apr 2023 09:38:55 -0500 Subject: [PATCH 09/21] fix line length --- pep-0712.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index 56776f509dd..08c94bce19b 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -67,8 +67,8 @@ New ``converter`` parameter --------------------------- This specification introduces a new parameter named ``converter`` to the -:func:`dataclasses.field` function. When an ``__init__`` method is synthesized by -``dataclass``-like semantics, if an argument is provided for the field, the +:func:`dataclasses.field` function. When an ``__init__`` method is synthesized +by ``dataclass``-like semantics, if an argument is provided for the field, the ``dataclass`` object's attribute will be assigned the result of calling the converter on the provided argument. If no argument is given and the field was constructed with a default value, the ``dataclass`` object's attribute will be From a1615b92dc241193d7887b31180b0a91a44f8d7c Mon Sep 17 00:00:00 2001 From: Joshua Date: Tue, 11 Apr 2023 09:44:52 -0500 Subject: [PATCH 10/21] Fix title underline --- pep-0712.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0712.rst b/pep-0712.rst index 08c94bce19b..209665440e4 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -238,7 +238,7 @@ Rejected Ideas ============== Just adding "converter" to :func:`py3.11:~typing.dataclass_transform`'s ``field_specifiers`` -------------------------------------------------------------------------- +-------------------------------------------------------------------------------------------- The idea of isolating this addition to :func:`py3.11:~typing.dataclass_transform` was briefly From 60b26c4ee8eed711af2ebabdd5f3bdd452f1324c Mon Sep 17 00:00:00 2001 From: Joshua Date: Tue, 11 Apr 2023 09:48:50 -0500 Subject: [PATCH 11/21] Add another cross-link --- pep-0712.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index 209665440e4..e3550b6a950 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -13,8 +13,8 @@ Abstract ======== :pep:`557` added :mod:`dataclasses` to the Python stdlib. :pep:`681` added -``dataclass_transform`` to help type checkers understand several common -dataclass-like libraries, such as attrs, Pydantic, and object +:func:`py3.11:~typing.dataclass_transform` to help type checkers understand +several common dataclass-like libraries, such as attrs, Pydantic, and object relational mapper (ORM) packages such as SQLAlchemy and Django. A common feature these libraries provide over the standard library From 7a7c47d1cfe48361f1bdf732e25537746d340185 Mon Sep 17 00:00:00 2001 From: Joshua Date: Tue, 11 Apr 2023 09:50:21 -0500 Subject: [PATCH 12/21] Fix broken cross-links --- pep-0712.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index e3550b6a950..4ae65b61d6d 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -13,7 +13,7 @@ Abstract ======== :pep:`557` added :mod:`dataclasses` to the Python stdlib. :pep:`681` added -:func:`py3.11:~typing.dataclass_transform` to help type checkers understand +:func:`~py3.11:~typing.dataclass_transform` to help type checkers understand several common dataclass-like libraries, such as attrs, Pydantic, and object relational mapper (ORM) packages such as SQLAlchemy and Django. @@ -24,7 +24,7 @@ user-provided conversion function. Therefore, this PEP adds a ``converter`` parameter to :func:`dataclasses.field` (along with the requisite changes to :class:`dataclasses.Field` and -:func:`py3.11:~typing.dataclass_transform` to specify the function to use to +:func:`~py3.11:~typing.dataclass_transform` to specify the function to use to convert the input value for each field to the representation to be stored in the dataclass. @@ -57,7 +57,7 @@ dataclass-like libraries provide support. Adding this feature to the standard library means more users are able to opt-in to these benefits without requiring third-party libraries. Additionally third-party libraries are able to clue type-checkers into their own conversion semantics through added support in -:func:`py3.11:~typing.dataclass_transform`, meaning users of those libraries +:func:`~py3.11:~typing.dataclass_transform`, meaning users of those libraries benefit as well. Specification @@ -78,7 +78,7 @@ Adding this parameter also implies the following changes: * A ``converter`` attribute will be added to :class:`dataclasses.Field`. * ``converter`` will be added to the field specifier parameters of arguments - provided to :func:`py3.11:~typing.dataclass_transform`'s ``field`` parameter. + provided to :func:`~py3.11:~typing.dataclass_transform`'s ``field`` parameter. Example ''''''' @@ -237,11 +237,11 @@ CPython support is implemented on `a branch in the author's fork `__ where it was suggested to broaden this to :mod:`dataclasses` more generally. From 8ea8463deb1b0645b18075d1063a09df1fa99c76 Mon Sep 17 00:00:00 2001 From: Joshua Date: Tue, 11 Apr 2023 11:42:23 -0500 Subject: [PATCH 13/21] Add missing sections --- pep-0712.rst | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/pep-0712.rst b/pep-0712.rst index 4ae65b61d6d..4ae0dd19074 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -225,6 +225,24 @@ Example def converter(x: int, y: str) -> list: ... def converter(x=..., y = ...): ... +Backward Compatibility +====================== + +These changes don't introduce any compatibility problems since they +only introduce new features. + +Security Implications +====================== + +There are no direct security concerns with these changes. + +How to Teach This +================= + +Documentation and examples explaining the new parameter and behavior will be +added to the relevant sections of the docs site (primarily on +:mod:`dataclasses`) and linked from the *What's New* document. + Reference Implementation ======================== From c6ec5999f049609e3f9a5814253d025c5db28a6c Mon Sep 17 00:00:00 2001 From: Joshua Date: Wed, 12 Apr 2023 09:47:15 -0500 Subject: [PATCH 14/21] feedback --- pep-0712.rst | 52 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index 4ae0dd19074..47d9fa937a5 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -7,13 +7,14 @@ Type: Standards Track Content-Type: text/x-rst Created: 01-Jan-2023 Python-Version: 3.13 -Post-History: `27-Dec-2022 `__, `19-Jan-2023 `__ +Post-History: `27-Dec-2022 `__, + `19-Jan-2023 `__, Abstract ======== :pep:`557` added :mod:`dataclasses` to the Python stdlib. :pep:`681` added -:func:`~py3.11:~typing.dataclass_transform` to help type checkers understand +:func:`~py3.11:typing.dataclass_transform` to help type checkers understand several common dataclass-like libraries, such as attrs, Pydantic, and object relational mapper (ORM) packages such as SQLAlchemy and Django. @@ -24,7 +25,7 @@ user-provided conversion function. Therefore, this PEP adds a ``converter`` parameter to :func:`dataclasses.field` (along with the requisite changes to :class:`dataclasses.Field` and -:func:`~py3.11:~typing.dataclass_transform` to specify the function to use to +:func:`~py3.11:typing.dataclass_transform`) to specify the function to use to convert the input value for each field to the representation to be stored in the dataclass. @@ -57,7 +58,7 @@ dataclass-like libraries provide support. Adding this feature to the standard library means more users are able to opt-in to these benefits without requiring third-party libraries. Additionally third-party libraries are able to clue type-checkers into their own conversion semantics through added support in -:func:`~py3.11:~typing.dataclass_transform`, meaning users of those libraries +:func:`~py3.11:typing.dataclass_transform`, meaning users of those libraries benefit as well. Specification @@ -78,7 +79,7 @@ Adding this parameter also implies the following changes: * A ``converter`` attribute will be added to :class:`dataclasses.Field`. * ``converter`` will be added to the field specifier parameters of arguments - provided to :func:`~py3.11:~typing.dataclass_transform`'s ``field`` parameter. + provided to :func:`~py3.11:typing.dataclass_transform`'s ``field`` parameter. Example ''''''' @@ -95,7 +96,7 @@ Example converter=lambda names: tuple(map(str.lower, names)) ) - # The default value is also converted, therefore the following is not a + # The default value is also converted; therefore the following is not a # type error. stock_image_path: pathlib.PurePosixPath = dataclasses.field( converter=pathlib.PurePosixPath, default="assets/unknown.png" @@ -143,7 +144,7 @@ Type-checking the default value Because the ``default`` value is unconditionally converted using ``converter``, if arguments for both ``converter`` and ``default`` are provided to :func:`dataclasses.field`, the ``default`` argument's type should be checked -using the ``converter``'s unary argument's type. +using the type of the unary argument to the ``converter`` callable. Type-checking the return type ''''''''''''''''''''''''''''' @@ -225,11 +226,23 @@ Example def converter(x: int, y: str) -> list: ... def converter(x=..., y = ...): ... + # Type checkers should not error on the following, since the default value + # is type-checked against the converter's unary argument type if a converter + # is provided. + @dataclasses.dataclass + class Example: + # Although the default value is of type `str` and the field is declared to + # be of type `pathlib.Path`, this is not a type error because the default + # value will be converted. + tmpdir: pathlib.Path = dataclasses.field(default="/tmp", converter=pathlib.Path) + + + Backward Compatibility ====================== These changes don't introduce any compatibility problems since they -only introduce new features. +only introduce opt-in new features. Security Implications ====================== @@ -243,6 +256,10 @@ Documentation and examples explaining the new parameter and behavior will be added to the relevant sections of the docs site (primarily on :mod:`dataclasses`) and linked from the *What's New* document. +The added documentation/examples will also cover the "common pitfalls" that +users of converters are likely to encounter. Such pitfalls include: +* Needing to handle ``None``/sentinel values. +* Needing to handle values that are already of the correct type. Reference Implementation ======================== @@ -255,8 +272,8 @@ CPython support is implemented on `a branch in the author's fork Date: Wed, 12 Apr 2023 09:51:56 -0500 Subject: [PATCH 15/21] Fix indentation --- pep-0712.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index 47d9fa937a5..8a217eab475 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -291,10 +291,10 @@ Leaving default values as-is allows type-checkers and dataclass authors to expect that the type of the default matches the type of the field. However, converting default values has two large advantages: 1. Compatibility with ``attrs``. Attrs unconditionally uses the converter to - convert the default value. + convert the default value. 2. Simpler defaults. Allowing the default value to have the same type as - user-provided values means dataclass authors get the same conveniences as - their callers. + user-provided values means dataclass authors get the same conveniences as + their callers. Automatic conversion using the field's type ------------------------------------------- From df21ecca38d49e8619881e6697504b6e6d959f6d Mon Sep 17 00:00:00 2001 From: Joshua Date: Wed, 12 Apr 2023 09:55:23 -0500 Subject: [PATCH 16/21] Try with newlines --- pep-0712.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pep-0712.rst b/pep-0712.rst index 8a217eab475..78d40d61a9a 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -290,8 +290,10 @@ There are pros and cons with both converting and not converting default values. Leaving default values as-is allows type-checkers and dataclass authors to expect that the type of the default matches the type of the field. However, converting default values has two large advantages: + 1. Compatibility with ``attrs``. Attrs unconditionally uses the converter to convert the default value. + 2. Simpler defaults. Allowing the default value to have the same type as user-provided values means dataclass authors get the same conveniences as their callers. From 74b8dce0e521e5b7d4d9475c3a6eae7088d8fbd2 Mon Sep 17 00:00:00 2001 From: Joshua Date: Thu, 13 Apr 2023 10:28:39 -0500 Subject: [PATCH 17/21] review comments --- pep-0712.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index 78d40d61a9a..ab2078a716d 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -231,10 +231,10 @@ Example # is provided. @dataclasses.dataclass class Example: - # Although the default value is of type `str` and the field is declared to - # be of type `pathlib.Path`, this is not a type error because the default - # value will be converted. - tmpdir: pathlib.Path = dataclasses.field(default="/tmp", converter=pathlib.Path) + # Although the default value is of type `str` and the field is declared to + # be of type `pathlib.Path`, this is not a type error because the default + # value will be converted. + tmpdir: pathlib.Path = dataclasses.field(default="/tmp", converter=pathlib.Path) @@ -276,7 +276,7 @@ Just adding "converter" to ``typing.dataclass_transform``'s ``field_specifiers`` -------------------------------------------------------------------------------- The idea of isolating this addition to -:func:`~py3.11:~typing.dataclass_transform` was briefly +:func:`~py3.11:typing.dataclass_transform` was briefly `discussed on Typing-SIG `__ where it was suggested to broaden this to :mod:`dataclasses` more generally. @@ -291,7 +291,7 @@ Leaving default values as-is allows type-checkers and dataclass authors to expect that the type of the default matches the type of the field. However, converting default values has two large advantages: -1. Compatibility with ``attrs``. Attrs unconditionally uses the converter to +1. Compatibility with attrs. Attrs unconditionally uses the converter to convert the default value. 2. Simpler defaults. Allowing the default value to have the same type as From a0c1d7790db6a1758f976f1827fe4c2aa980a901 Mon Sep 17 00:00:00 2001 From: Joshua Date: Tue, 18 Apr 2023 13:12:32 -0500 Subject: [PATCH 18/21] Jelle's comments --- pep-0712.rst | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index ab2078a716d..be3038ee557 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -42,7 +42,7 @@ to: * Shift conversion responsibility onto the caller of the dataclass constructor. This can make constructing certain dataclasses unnecessarily verbose and repetitive. -* Provide a custom ``__init__`` and which declares "wider" parameter types and +* Provide a custom ``__init__`` which declares "wider" parameter types and converts them when setting the appropriate attribute. This not only duplicates the typing annotations between the converter and ``__init__``, but also opts the user out of many of the features :mod:`dataclasses` provides. @@ -90,7 +90,7 @@ Example class InventoryItem: # `converter` as a type id: int = dataclasses.field(converter=int) - skus: tuple[int, ...] = dataclasses.field(converter=tuple[int]) + skus: tuple[int, ...] = dataclasses.field(converter=tuple[int, ...]) # `converter` as a callable names: tuple[str, ...] = dataclasses.field( converter=lambda names: tuple(map(str.lower, names)) @@ -114,9 +114,9 @@ Impact on typing ---------------- ``converter`` arguments are expected to be callable objects which accept a -unary argument and return a type compatible with the field's annotated type. -The callable's unary argument's type is used as the type of the parameter in -the synthesized ``__init__`` method. +single argument and return a type compatible with the field's annotated type. +The type of the callable's argument is used as the type of the corresponding +parameter in the synthesized ``__init__`` method. Type-narrowing the argument type '''''''''''''''''''''''''''''''' @@ -126,12 +126,12 @@ For the purpose of deducing the type of the argument in the synthesized the following rules: * If the ``converter`` is of type ``Any``, it is assumed to be callable with a - unary ``Any`` typed-argument. + single ``Any`` typed-argument. * All keyword-only parameters can be ignored. -* ``**kwargs`` can be ignored. +* ``**kwargs`` with defaults can be ignored. * ``*args`` can be ignored if any parameters precede it. Otherwise if ``*args`` is the only non-ignored parameter, the type it accepts for each positional - argument is the type of the unary argument. E.g. given params + argument is the type of the single argument. E.g. given params ``(x: str, *args: str)``, ``*args`` can be ignored. However, given params ``(*args: str)``, the callable type can be narrowed to ``(__x: str, /)``. * Parameters with default values that aren't the first parameter can be @@ -144,7 +144,7 @@ Type-checking the default value Because the ``default`` value is unconditionally converted using ``converter``, if arguments for both ``converter`` and ``default`` are provided to :func:`dataclasses.field`, the ``default`` argument's type should be checked -using the type of the unary argument to the ``converter`` callable. +using the type of the single argument to the ``converter`` callable. Type-checking the return type ''''''''''''''''''''''''''''' @@ -160,11 +160,11 @@ Overloads The above rules should be applied to each ``@overload`` for overloaded functions. If after these rules are applied an overload is invalid (either -because there is no overload that would accept a unary argument, or because +because there is no overload that would accept a single argument, or because there is no overload that returns an acceptable type) it should be ignored. If multiple overloads are valid after these rules are applied, the -type-checker can assume the converter's unary argument type is the union of -each overload's unary argument type. If no overloads are valid, it is a type +type-checker can assume the converter's single argument type is the union of +each overload's single argument type. If no overloads are valid, it is a type error. Example @@ -227,7 +227,7 @@ Example def converter(x=..., y = ...): ... # Type checkers should not error on the following, since the default value - # is type-checked against the converter's unary argument type if a converter + # is type-checked against the converter's single argument type if a converter # is provided. @dataclasses.dataclass class Example: From 3677a9b94448dd2d788a280fef9214e7354d1408 Mon Sep 17 00:00:00 2001 From: Joshua Cannon Date: Sat, 22 Apr 2023 13:30:31 -0500 Subject: [PATCH 19/21] Apply suggestions from code review Co-authored-by: Erik De Bonte Co-authored-by: C.A.M. Gerlach --- pep-0712.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index be3038ee557..4fe6c19cc8c 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -160,8 +160,8 @@ Overloads The above rules should be applied to each ``@overload`` for overloaded functions. If after these rules are applied an overload is invalid (either -because there is no overload that would accept a single argument, or because -there is no overload that returns an acceptable type) it should be ignored. +because it does not accept a single argument, or because +it does not return an acceptable type) it should be ignored. If multiple overloads are valid after these rules are applied, the type-checker can assume the converter's single argument type is the union of each overload's single argument type. If no overloads are valid, it is a type @@ -313,6 +313,7 @@ or if it is additionally supposed to convert each element type to ``int``. References ========== + .. _attrs-converters: https://www.attrs.org/en/21.2.0/examples.html#conversion .. _cpython-branch: https://github.com/thejcannon/cpython/tree/converter .. _only-dataclass-transform: https://mail.python.org/archives/list/typing-sig@python.org/thread/NWZQIINJQZDOCZGO6TGCUP2PNW4PEKNY/ From 373dd07e06cd4a1c895653ed280675217c6f72c1 Mon Sep 17 00:00:00 2001 From: Joshua Cannon Date: Sat, 22 Apr 2023 13:57:38 -0500 Subject: [PATCH 20/21] Encorproate feedback from my personal celebrities --- pep-0712.rst | 118 +++++++++------------------------------------------ 1 file changed, 20 insertions(+), 98 deletions(-) diff --git a/pep-0712.rst b/pep-0712.rst index 4fe6c19cc8c..a3aee1dd636 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -46,7 +46,8 @@ to: converts them when setting the appropriate attribute. This not only duplicates the typing annotations between the converter and ``__init__``, but also opts the user out of many of the features :mod:`dataclasses` provides. -* Not rely on, or ignore type-checking. +* Provide a custom ``__init__`` but without meaningful type annotations + for the parameter types requiring conversion. None of these choices are ideal. @@ -78,8 +79,8 @@ assigned the result of calling the converter on the provided default. Adding this parameter also implies the following changes: * A ``converter`` attribute will be added to :class:`dataclasses.Field`. -* ``converter`` will be added to the field specifier parameters of arguments - provided to :func:`~py3.11:typing.dataclass_transform`'s ``field`` parameter. +* ``converter`` will be added to :func:`~py3.11:typing.dataclass_transform`'s + list of supported field specifier parameters. Example ''''''' @@ -113,30 +114,13 @@ Example Impact on typing ---------------- -``converter`` arguments are expected to be callable objects which accept a -single argument and return a type compatible with the field's annotated type. -The type of the callable's argument is used as the type of the corresponding -parameter in the synthesized ``__init__`` method. - -Type-narrowing the argument type -'''''''''''''''''''''''''''''''' - -For the purpose of deducing the type of the argument in the synthesized -``__init__`` method, the ``converter`` argument's type can be "narrowed" using -the following rules: - -* If the ``converter`` is of type ``Any``, it is assumed to be callable with a - single ``Any`` typed-argument. -* All keyword-only parameters can be ignored. -* ``**kwargs`` with defaults can be ignored. -* ``*args`` can be ignored if any parameters precede it. Otherwise if ``*args`` - is the only non-ignored parameter, the type it accepts for each positional - argument is the type of the single argument. E.g. given params - ``(x: str, *args: str)``, ``*args`` can be ignored. However, given params - ``(*args: str)``, the callable type can be narrowed to ``(__x: str, /)``. -* Parameters with default values that aren't the first parameter can be - ignored. E.g. given params ``(x: str = "0", y: int = 1)``, parameter ``y`` - can be ignored and the type can be assumed to be ``(x: str)``. +A ``converter`` must be a callable that accepts a single positional argument, and +the parameter type corresponding to this positional argument provides the type +of the the synthesized ``__init__`` parameter associated with the field. + +In other words, the argument provided for the converter parameter must be +compatible with ``Callable[[T], X]`` where ``T`` is the input type for +the converter and ``X`` is the output type of the converter. Type-checking the default value ''''''''''''''''''''''''''''''' @@ -146,8 +130,8 @@ if arguments for both ``converter`` and ``default`` are provided to :func:`dataclasses.field`, the ``default`` argument's type should be checked using the type of the single argument to the ``converter`` callable. -Type-checking the return type -''''''''''''''''''''''''''''' +Converter return type +''''''''''''''''''''' The return type of the callable must be a type that's compatible with the field's declared type. This includes the field's type exactly, but can also be @@ -155,82 +139,17 @@ a type that's more specialized (such as a converter returning a ``list[int]`` for a field annotated as ``list``, or a converter returning an ``int`` for a field annotated as ``int | str``). -Overloads -''''''''' - -The above rules should be applied to each ``@overload`` for overloaded -functions. If after these rules are applied an overload is invalid (either -because it does not accept a single argument, or because -it does not return an acceptable type) it should be ignored. -If multiple overloads are valid after these rules are applied, the -type-checker can assume the converter's single argument type is the union of -each overload's single argument type. If no overloads are valid, it is a type -error. - Example ''''''' .. code-block:: python - # The following are valid converter types, with a comment containing the - # synthesized __init__ argument's type. - converter: Any # Any - def converter(x: int): ... # int - def converter(x: int | str): ... # int | str - def converter(x: int, y: str = "a"): ... # int - def converter(x: int, *args: str): ... # int - def converter(*args: str): ... # str - def converter(*args: str, x: int = 0): ... # str - - @overload - def converter(x: int): ... # <- valid - @overload - def converter(x: int, y: str): ... # <- ignored - @overload - def converter(x: list): ... # <- valid - def converter(x, y = ...): ... # int | list - - # The following are valid converter types for a field annotated as type 'list'. - def converter(x) -> list: ... - def converter(x) -> Any: ... - def converter(x) -> list[int]: ... - - @overload - def converter(x: int) -> tuple: ... # <- ignored - @overload - def converter(x: str) -> list: ... # <- valid - @overload - def converter(x: bytes) -> list: ... # <- valid - def converter(x): ... # __init__ would use argument type 'str | bytes'. - - # The following are invalid converter types. - def converter(): ... - def converter(**kwargs): ... - def converter(x, y): ... - def converter(*, x): ... - def converter(*args, x): ... - - @overload - def converter(): ... - @overload - def converter(x: int, y: str): ... - def converter(x=..., y = ...): ... - - # The following are invalid converter types for a field annotated as type 'list'. - def converter(x) -> tuple: ... - def converter(x) -> Sequence: ... - - @overload - def converter(x) -> tuple: ... - @overload - def converter(x: int, y: str) -> list: ... - def converter(x=..., y = ...): ... - - # Type checkers should not error on the following, since the default value - # is type-checked against the converter's single argument type if a converter - # is provided. @dataclasses.dataclass class Example: + my_int: int = dataclasses.field(converter=int) + my_tuple: tuple[int, ...] = dataclasses.field(converter=tuple[int, ...]) + my_cheese: Cheese = dataclasses.field(converter=make_cheese) + # Although the default value is of type `str` and the field is declared to # be of type `pathlib.Path`, this is not a type error because the default # value will be converted. @@ -258,8 +177,11 @@ added to the relevant sections of the docs site (primarily on The added documentation/examples will also cover the "common pitfalls" that users of converters are likely to encounter. Such pitfalls include: + * Needing to handle ``None``/sentinel values. * Needing to handle values that are already of the correct type. +* Avoiding ``lambda``s for converters, as the synthesized ``__init__`` + parameter's type will become ``Any``. Reference Implementation ======================== From e92ff5500491476bc18cd71b6e2521f7115f4078 Mon Sep 17 00:00:00 2001 From: Joshua Cannon Date: Sat, 22 Apr 2023 18:20:51 -0500 Subject: [PATCH 21/21] fix lint --- pep-0712.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0712.rst b/pep-0712.rst index a3aee1dd636..c5dd8b99b74 100644 --- a/pep-0712.rst +++ b/pep-0712.rst @@ -180,7 +180,7 @@ users of converters are likely to encounter. Such pitfalls include: * Needing to handle ``None``/sentinel values. * Needing to handle values that are already of the correct type. -* Avoiding ``lambda``s for converters, as the synthesized ``__init__`` +* Avoiding lambdas for converters, as the synthesized ``__init__`` parameter's type will become ``Any``. Reference Implementation