Add typing_extensions module for new / optional typing features to PyPI

[vlasovskikh]

Let's add typing_extensions module to PyPI in addition to Mypy-specific mypy_extensions.

Currently new stuff in typing appears in mypy_extensions even though most of the new typing features are not Mypy-specific.

In order to make new typing features available to 1) previous Python versions, 2) all type checkers I would suggest to move these features to typing_extensions.

The good candidates for including in typing_extensions could be:

• NoReturn (apparently already in PEP 484)
• Protocol (draft PEP 544)
• Argspecs?
• TypedDict?

[gvanrossum]

Agreed. I do think there should be a requirement for things to appear in a PEP (either 484 or a new one) before they go into typing_extensions. This means we should probably write up TypedDict and maybe wait for argspecs (for those there's still a lot of possible variation in the eventual design).

[Michael0x2a]

Type and maybe NewType would be other good candidates. They're present in the later versions of Python 3.5, but not the earlier ones, iirc.

[gvanrossum]

Oooh, might as well!

[Michael0x2a]

Other ones perhaps worth adding in:

• typing.Collection -- added in 3.6
• typing.Deque -- added in 3.6.1
• typing.ContextManager -- added in 3.6
• typing.Counter -- added in 3.6.1
• typing.ChainMap -- added in 3.6.1
• typing.AsyncGenerator -- added in 3.5.4
• typing.ClassVar -- added in 3.5.3

(I got this list by looking for the "New in" phrase in the typing docs, though it might not be comprehensive. For example, neither Type nor NewType have the "New in ..." phrases next to them in the index at the end (which might in part be my fault, tbh))

And for the sake of completeness, we could also add in the TYPE_CHECKING constant (idk when exactly it was added, but it's listed as being new in the Python 3.6 release notes). It's a bit unnecessary since we can just do if False or several other alternatives instead, but hey, as long as we're here...

[JelleZijlstra]

Sounds like it might be worthwhile to just shadow/duplicate all of regular typing, so that people aren't tied to the stdlib's version of typing. The disadvantage is that type checkers would have to add special-casing for things imported from typing_extensions.

I've been thinking of a related idea to create a library of commonly useful type aliases. Some things that could be included are:

• An alias for Union[bytes, Text] (which is needed very often in typeshed)
• A generic alias for Union[AnyStr, os.PathLike[AnyStr]] that falls back to AnyStr before 3.6 (another common typeshed need)
• The types from https://github.com/python/typeshed/blob/master/stdlib/3/wsgiref/types.pyi, which don't actually exist at runtime.

[Michael0x2a]

I was curious and decided to just install Python 3.5.0 and 3.6.1 and do a diff, and found a few more (non-private) names that I missed: AsyncIterable, AsyncIterator, Awaitable, MethodDescriptorType, MethodWrapperType, NamedTupleMeta, SlotWrapperType, Text. Full list here.

(Though, I'm not sure how many of these are meant to be publicly used or not? The last four items on the list certainly don't appear to be documented anywhere.)

In any case, I agree with @JelleZijlstra that just shadowing typing might be the simplest option at this point given the number of differences (16 to 20, depending on how you count the last four).

[JukkaL]

I'm not sure if I like shadowing everything, as it would potentially lead into a subset of users just importing everything from typing_extensions. This would lead into inconsistency -- you might start seeing odd-looking code like this (and this could confuse some users and they could start start cargo culting the idiom):

# Just to make sure everything works in 3.5.0
from typing_extensions import Optional, Tuple

Also, if implementations of things like List, Tuple and such in typing_extensions would track the latest typing version, they might not always support mixing with types inside the system typing (for example, if we change the representation of List[int] as has been discussed). This could mean that something like this could fail at runtime:

# a.py
from typing_extensions import List
A = List

# b.py
from a import A
from typing import List
List[A[int]]  # might fail

The memory usage and startup time cost of typing_extensions would also be bigger if it would reimplement everything in typing.

If we'd just import the basic primitives like List from the system typing and export them through typing_extensions, we wouldn't have this problem, but people might be surprised by this as well, if they are expecting that all typing changes get backported to typing_extensions. Also, this opens to doors to the cargo culting example I gave earlier.

My preference would be just to include new things added to the public API of typing (or significantly changed things) since 3.5.0 to typing_extensions.

[ilevkivskyi]

@JukkaL

This could mean that something like this could fail at runtime:

I would say it will most certainly fail. I remember having big troubles because of accidentally having two copies of typing in different places.

My preference would be just to include new things added to the public API of typing (or significantly changed things) since 3.5.0 to typing_extensions.

I agree we should not shadow everything. Also I could propose adding stuff from typing_inspect to typing_extensions (if anyone still cares about runtime introspection).

[Michael0x2a]

Is anybody working on this at the moment? If not, I'd be happy to volunteer :)

I don't think I should be the one to publish the package on pypi or anything, but I could definitely do the preliminary work of pulling together the module for somebody to review, and work on making whatever changes are necessary to mypy itself.

[ilevkivskyi]

@Michael0x2a
It looks like nobody is working on this now, so that you can go ahead. I would propose to just combine three things in typing_extensions:

• New types added after 3.5.0
• Things from mypy_extensions (note there is a PR Support TypedDicts with missing keys (total=False) mypy#3558 that will change TypedDict)
• Introspection functions from typing_inspect

[JukkaL]

I think that we should focus on things in mypy_extensions that we think are likely to be promoted to typing without API changes. Some of the features are likely to change or we don't have much experience with them yet, and we may want to only have them in mypy_extensions for now. Examples include TypedDict, as mentioned by @ilevkivskyi, and also the flexible callable syntax (at least naming might still change).

Similarly, maybe typing_inspect is still too experimental for typing_extensions. Also, it's already available on PyPI, so the benefit of including it in typing_extensions seems limited.

[ilevkivskyi]

@JukkaL

... without API changes ...

This is a reasonable approach. Then my list above would be even simpler: NoReturn is already in typing, and TypedDict and flexible Callable still might change, therefore we can start from the first bullet -- new types added after 3.5.0.

[Michael0x2a]

I've started work on this, and ran into some complications I'd like to get some feedback on.

---

First, typing.overload originally threw an exception if you tried to use it at runtime, outside of stub files-- this was changed in Python 3.5.2.

I know we discussed up above that we wanted to minimize duplication, but I think it makes sense here to include the modified version of overload in typing_extensions? I can't really see this causing subtle runtime issues, and defining overloads at runtime is handy/would make it easier to write code compatible across multiple Python versions.

---

Second, the typing module seems to have been refactored between Python 3.5.2 and 3.5.3. In the older variant, you create new types by doing something like this:

class UnionMeta(TypingMeta):
    # Complicated code here

class Union(Final, metaclass=UnionMeta, _root=True):
    # Simple code here

In the newer variant, you instead do this:

class _Union(_FinalTypingBase, _root=True):
    # Complex code here

Union = _Union(_root=True)

This causes some complications with the NoReturn and ClassVar types, since both are currently defined using the newer variant.

One solution is to copy over _FinalTypingBase and all associated helper functions/parent classes to typing_extensions. This means we don't need to change NoReturn and ClassVar at all -- my main concern is that it could lead to subtle compatibility issues when mixing types (though to be fair, you can't really do that with NoReturn and ClassVar...).

The other solution would be to re-implement NoReturn and ClassVar to use the older variant and export that when the user is using Python 3.5.0 - 3.5.2. This likely minimize the chances of weirdness, but would likely make it harder to backport more complex types (such as TypedDict?) in the future.

I'm tentatively leaning towards the second solution -- thoughts?

---

I don't think this last thing is that big of a deal, but the exact definition of some of the collections types were changed over time. For example, typing.Sequence was changed to subclass typing.Reversible in 3.5.2, and was changed again to subclass typing.Collection in 3.5.3.

The question is whether we should include these modified types in typing_extensions -- no matter what you do, you're always going to run into inconsistencies related to issubclass checks at runtime (e.g. doing things like issubclass(Sequence[...], Reversible)),.

However, I don't think this is a feature many people rely on (maybe except for people writing runtime type checkers?), so I'm planning on not including these modified types to minimize duplication.

Does that sound correct, or should we backport the modified collection types anyways for the sake of consistency? A third option might be to rename any modified types in typing_extensions (e.g. provide a ReversibleSequence type), but that also seems confusing.

[gvanrossum]

I hope Ivan can provide more guidance here, but I also hope that typing_extensions.py doesn't end up containing a whole bunch of reimplementations of stuff that's also in typing.py. I do accept that in order to support the full range of released versions of typing.py, you may need to implement some things in different ways -- hopefully checking sys.version_info should suffice there (since typing.py release Python versions anyway; I don't really want to support configurations where someone installed e.g. typing.py 3.5.3 on Python 3.5.1).