Self-referential type specification issues

[altendky]

• cattrs version: 02d9da0
• Python version: 3.6.2
• Operating System: Kubuntu 16.04

Description

Proper definition of self-referential types seems unclear. This would relate to the forward declarations problem discussed in PEP484. I started digging on this because of the typing.List[typing.Union['WithUnion', WithChild]] case but provide a few attempts building up to that for reference.

My big picture is to easily [de]serialize attrs classes which I am using for a PyQt5 tree model. They can allow multiple types of children within lists (a group node would be able to hold both other groups and parameters, for example).

In some of the cases self-reference seems achievable by registering typing._ForwardRef('MyClass') but it doesn't seem clean to have to register a private member of typing so I thought it would be worth considering a cleaner approach. I haven't worked with typing before so I'm not sure exactly how it goes about dealing with class name strings and disambiguation of matching class names in different scopes or modules.

As a side note, I first rolled my own [de]serialization scheme around attrs. Then started doing an attrs/marshmallow integration in graham. Now I am trying out cattrs for the same roll.

What I Did

import traceback
import typing

import attr
import cattr


@attr.s
class WithChild:
    child = cattr.typed(
        default=None,
        type='WithChild',
    )


@attr.s
class WithChildren:
    children = cattr.typed(
        default=attr.Factory(list),
        type=typing.List['WithChildren'],
    )


@attr.s
class WithUnion:
    children = cattr.typed(
        default=attr.Factory(list),
        type=typing.List[typing.Union['WithUnion']]
    )


@attr.s
class WithUnionTwo:
    children = cattr.typed(
        default=attr.Factory(list),
        type=typing.List[typing.Union['WithUnion', WithChild]]
    )


def tryit():
    try:
        cattr.structure({}, WithChild)
    except:
        print(' - - - - - - WithChild failed')
        print(traceback.format_exc())
    else:
        print(' - - - - - - WithChild succeeded')

    try:
        cattr.structure({'children': [{}]}, WithChildren)
    except:
        print(' - - - - - - WithChildren failed')
        print(traceback.format_exc())
    else:
        print(' - - - - - - WithChildren succeeded')

    try:
        cattr.structure({'children': [{}]}, WithUnion)
    except:
        print(' - - - - - - WithUnion failed')
        print(traceback.format_exc())
    else:
        print(' - - - - - - WithUnion succeeded')

    try:
        cattr.structure({'children': [{}]}, WithUnionTwo)
    except:
        print(' - - - - - - WithUnionTwo failed')
        print(traceback.format_exc())
    else:
        print(' - - - - - - WithUnionTwo succeeded')


print(' == == == == == == == == == == == == No registration\n')
tryit()

cattr.register_structure_hook('WithChild', lambda d, t: WithChild(**d))
cattr.register_structure_hook('WithChildren', lambda d, t: WithChildren(**d))
cattr.register_structure_hook('WithUnion', lambda d, t: WithUnion(**d))
cattr.register_structure_hook('WithUnionTwo', lambda d, t: WithUnionTwo(**d))

print(' == == == == == == == == == == == == Registered strings\n')
tryit()

cattr.register_structure_hook(
    typing._ForwardRef('WithChild'),
    lambda d, t: WithChild(**d),
)
cattr.register_structure_hook(
    typing._ForwardRef('WithChildren'),
    lambda d, t: WithChildren(**d),
)
cattr.register_structure_hook(
    typing._ForwardRef('WithUnion'),
    lambda d, t: WithUnion(**d),
)
cattr.register_structure_hook(
    typing._ForwardRef('WithUnionTwo'),
    lambda d, t: WithUnionTwo(**d),
)

print(' == == == == == == == == == == == == Registered typing._ForwardRef()\n')
tryit()

 == == == == == == == == == == == == No registration

 - - - - - - WithChild failed
Traceback (most recent call last):
  File "/epc/t/472/p/venv/src/graham/src/graham/tests/test_cattrs_a.py", line 42, in tryit
    cattr.structure({}, WithChild)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 171, in structure
    return self.structure_func.dispatch(cl)(obj, cl)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 86, in <lambda>
    lambda *args, **kwargs: self.structure_attrs(*args, **kwargs))
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 276, in structure_attrs_fromdict
    converted = self._structure_attr_from_dict(a, name, obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 293, in _structure_attr_from_dict
    return self._structure.dispatch(type_)(mapping.get(a.name), type_)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 229, in _structure_default
    raise ValueError(msg)
ValueError: Unsupported type: WithChild. Register a structure hook for it.

 - - - - - - WithChildren failed
Traceback (most recent call last):
  File "/epc/t/472/p/venv/src/graham/src/graham/tests/test_cattrs_a.py", line 50, in tryit
    cattr.structure({'children': [{}]}, WithChildren)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 171, in structure
    return self.structure_func.dispatch(cl)(obj, cl)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 86, in <lambda>
    lambda *args, **kwargs: self.structure_attrs(*args, **kwargs))
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 276, in structure_attrs_fromdict
    converted = self._structure_attr_from_dict(a, name, obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 293, in _structure_attr_from_dict
    return self._structure.dispatch(type_)(mapping.get(a.name), type_)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in _structure_list
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in <listcomp>
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 229, in _structure_default
    raise ValueError(msg)
ValueError: Unsupported type: _ForwardRef('WithChildren'). Register a structure hook for it.

 - - - - - - WithUnion failed
Traceback (most recent call last):
  File "/epc/t/472/p/venv/src/graham/src/graham/tests/test_cattrs_a.py", line 58, in tryit
    cattr.structure({'children': [{}]}, WithUnion)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 171, in structure
    return self.structure_func.dispatch(cl)(obj, cl)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 86, in <lambda>
    lambda *args, **kwargs: self.structure_attrs(*args, **kwargs))
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 276, in structure_attrs_fromdict
    converted = self._structure_attr_from_dict(a, name, obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 293, in _structure_attr_from_dict
    return self._structure.dispatch(type_)(mapping.get(a.name), type_)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in _structure_list
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in <listcomp>
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 229, in _structure_default
    raise ValueError(msg)
ValueError: Unsupported type: _ForwardRef('WithUnion'). Register a structure hook for it.

 - - - - - - WithUnionTwo failed
Traceback (most recent call last):
  File "/epc/t/472/p/venv/src/graham/src/graham/tests/test_cattrs_a.py", line 66, in tryit
    cattr.structure({'children': [{}]}, WithUnionTwo)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 171, in structure
    return self.structure_func.dispatch(cl)(obj, cl)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 86, in <lambda>
    lambda *args, **kwargs: self.structure_attrs(*args, **kwargs))
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 276, in structure_attrs_fromdict
    converted = self._structure_attr_from_dict(a, name, obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 293, in _structure_attr_from_dict
    return self._structure.dispatch(type_)(mapping.get(a.name), type_)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in _structure_list
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in <listcomp>
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 372, in _structure_union
    cl = self._dis_func_cache(union)(obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 398, in _get_dis_func
    raise ValueError('Only unions of attr classes supported '
ValueError: Only unions of attr classes supported currently. Register a loads hook manually.

 == == == == == == == == == == == == Registered strings

 - - - - - - WithChild failed
Traceback (most recent call last):
  File "/epc/t/472/p/venv/src/graham/src/graham/tests/test_cattrs_a.py", line 42, in tryit
    cattr.structure({}, WithChild)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 171, in structure
    return self.structure_func.dispatch(cl)(obj, cl)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 86, in <lambda>
    lambda *args, **kwargs: self.structure_attrs(*args, **kwargs))
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 276, in structure_attrs_fromdict
    converted = self._structure_attr_from_dict(a, name, obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 293, in _structure_attr_from_dict
    return self._structure.dispatch(type_)(mapping.get(a.name), type_)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 229, in _structure_default
    raise ValueError(msg)
ValueError: Unsupported type: WithChild. Register a structure hook for it.

 - - - - - - WithChildren failed
Traceback (most recent call last):
  File "/epc/t/472/p/venv/src/graham/src/graham/tests/test_cattrs_a.py", line 50, in tryit
    cattr.structure({'children': [{}]}, WithChildren)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 171, in structure
    return self.structure_func.dispatch(cl)(obj, cl)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 86, in <lambda>
    lambda *args, **kwargs: self.structure_attrs(*args, **kwargs))
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 276, in structure_attrs_fromdict
    converted = self._structure_attr_from_dict(a, name, obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 293, in _structure_attr_from_dict
    return self._structure.dispatch(type_)(mapping.get(a.name), type_)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in _structure_list
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in <listcomp>
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 229, in _structure_default
    raise ValueError(msg)
ValueError: Unsupported type: _ForwardRef('WithChildren'). Register a structure hook for it.

 - - - - - - WithUnion failed
Traceback (most recent call last):
  File "/epc/t/472/p/venv/src/graham/src/graham/tests/test_cattrs_a.py", line 58, in tryit
    cattr.structure({'children': [{}]}, WithUnion)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 171, in structure
    return self.structure_func.dispatch(cl)(obj, cl)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 86, in <lambda>
    lambda *args, **kwargs: self.structure_attrs(*args, **kwargs))
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 276, in structure_attrs_fromdict
    converted = self._structure_attr_from_dict(a, name, obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 293, in _structure_attr_from_dict
    return self._structure.dispatch(type_)(mapping.get(a.name), type_)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in _structure_list
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in <listcomp>
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 229, in _structure_default
    raise ValueError(msg)
ValueError: Unsupported type: _ForwardRef('WithUnion'). Register a structure hook for it.

 - - - - - - WithUnionTwo failed
Traceback (most recent call last):
  File "/epc/t/472/p/venv/src/graham/src/graham/tests/test_cattrs_a.py", line 66, in tryit
    cattr.structure({'children': [{}]}, WithUnionTwo)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 171, in structure
    return self.structure_func.dispatch(cl)(obj, cl)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 86, in <lambda>
    lambda *args, **kwargs: self.structure_attrs(*args, **kwargs))
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 276, in structure_attrs_fromdict
    converted = self._structure_attr_from_dict(a, name, obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 293, in _structure_attr_from_dict
    return self._structure.dispatch(type_)(mapping.get(a.name), type_)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in _structure_list
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in <listcomp>
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 372, in _structure_union
    cl = self._dis_func_cache(union)(obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 398, in _get_dis_func
    raise ValueError('Only unions of attr classes supported '
ValueError: Only unions of attr classes supported currently. Register a loads hook manually.

 == == == == == == == == == == == == Registered typing._ForwardRef()

 - - - - - - WithChild failed
Traceback (most recent call last):
  File "/epc/t/472/p/venv/src/graham/src/graham/tests/test_cattrs_a.py", line 42, in tryit
    cattr.structure({}, WithChild)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 171, in structure
    return self.structure_func.dispatch(cl)(obj, cl)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 86, in <lambda>
    lambda *args, **kwargs: self.structure_attrs(*args, **kwargs))
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 276, in structure_attrs_fromdict
    converted = self._structure_attr_from_dict(a, name, obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 293, in _structure_attr_from_dict
    return self._structure.dispatch(type_)(mapping.get(a.name), type_)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 229, in _structure_default
    raise ValueError(msg)
ValueError: Unsupported type: WithChild. Register a structure hook for it.

 - - - - - - WithChildren succeeded
 - - - - - - WithUnion succeeded
 - - - - - - WithUnionTwo failed
Traceback (most recent call last):
  File "/epc/t/472/p/venv/src/graham/src/graham/tests/test_cattrs_a.py", line 66, in tryit
    cattr.structure({'children': [{}]}, WithUnionTwo)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 171, in structure
    return self.structure_func.dispatch(cl)(obj, cl)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 86, in <lambda>
    lambda *args, **kwargs: self.structure_attrs(*args, **kwargs))
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 276, in structure_attrs_fromdict
    converted = self._structure_attr_from_dict(a, name, obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 293, in _structure_attr_from_dict
    return self._structure.dispatch(type_)(mapping.get(a.name), type_)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in _structure_list
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 303, in <listcomp>
    for e in obj]
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 372, in _structure_union
    cl = self._dis_func_cache(union)(obj)
  File "/epc/t/472/p/venv/src/cattrs/cattr/converters.py", line 398, in _get_dis_func
    raise ValueError('Only unions of attr classes supported '
ValueError: Only unions of attr classes supported currently. Register a loads hook manually.

[Tinche]

Hi! Thanks for the detailed write up. I'll brood on this and get back to you.

[johnwlockwood]

This script demos the issue:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from typing import List, Optional
from cattr import structure, unstructure
import attr


@attr.s
class Node(object):
    value: int = attr.ib()
    children: Optional[List['Node']] = attr.ib(default=None)


def main():
    g = {
        'value': 1,
        'children': [
            {'value': 3},
            {'value': 4, 'children': [{'value':44}]},
            {'value': 5, 'children': [{'value':55}]}
        ]
    }
    n = structure(g, Node)
    print(n)


if __name__ == "__main__":
    main()

[dcczi]

For those of you who found your way to this issue when attempting to structure self-referential types, here is a demonstration of how to make it work:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from typing import List, Optional, ForwardRef

import attr
from cattr import structure, unstructure, register_structure_hook


@attr.s
class Subattr(object):
    a: int = attr.ib()


@attr.s
class Node(object):
    value: int = attr.ib()
    subattr: Optional[Subattr] = attr.ib(default=None)
    children: List['Node'] = attr.ib(default=[])


def main():
    child_with_attr_field = Node(value=1, children=[Node(value=2, subattr=Subattr(a=2))])
    print(child_with_attr_field)
    print(unstructure(child_with_attr_field))
    print(structure(unstructure(child_with_attr_field), Node))


register_structure_hook(
    ForwardRef("Node"), lambda d, t: structure(d, Node),
)

Notably, if you use the structure hook from the sample code in the issue description:

register_structure_hook(
    ForwardRef("Node"), lambda d, t: Node(**d),  # wrong
)

... then attributes that are themselves attr types will not structure as their attr classes.

Edited: The previous incorrect comment has been replaced with a useful one.

[Tinche]

Apart from this issue being almost 6 years old, this actually works out of the box nowadays!

from __future__ import annotations

from attrs import Factory, define

from cattrs import structure, unstructure


@define
class Subattr:
    a: int


@define
class Node:
    value: int
    subattr: Subattr | None = None
    children: list[Node] = Factory(list)


child_with_attr_field = Node(value=1, children=[Node(value=2, subattr=Subattr(a=2))])
print(child_with_attr_field)
print(unstructure(child_with_attr_field))
print(structure(unstructure(child_with_attr_field), Node))

Also super cool to see how attrs, cattrs and Python typing in general have come a long way!

[c30ra]

Sorry to resurrect this, but using from __future__ import annotations break other registered structure hooks (there should be an open issue on this), if I don't use it, the example before doesn't work. The proper way should be to use Self, but it seems not supported.

[Tinche]

If you remove the from __future__ import annotations, but change Node to be:

@define
class Node:
    value: int
    subattr: Subattr | None = None
    children: "list[Node]" = Factory(list)

it still works.

That said, Self support is on my radar.

[Tinche]

I'm about to land #627 . On that branch, this also works:

from typing import Self

from attrs import Factory, define

from cattrs import structure, unstructure


@define
class Subattr:
    a: int


@define
class Node:
    value: int
    subattr: Subattr | None = None
    children: list[Self] = Factory(list)

[c30ra]

nice thank you for the help!!