Backport Python 3.10 NewTypes

[effigies]

As of Python 3.10, the following types are picklable:

from pathlib import Path
from typing import NewType

File = NewType("File", Path)
Directory = NewType("Directory", Path)

Prior to that, this fails. Would it be possible to backport the NewType class to Python 3.7+?

[AlexWaygood]

Here you go: #157

[effigies]

Amazing. Thanks for being so quick!

[JelleZijlstra]

@effigies may I ask why it's important to you that types are picklable? We have quite a lot of code to deal with pickling types and on first sight I don't really understand why people would care. This may be relevant to implementing new type system features in the future.

[effigies]

Sure. It's not a principled requirement so much as practically it's causing me problems while trying to add static type checking to an existing code-base that uses type annotations at runtime.

The context is a workflow engine where we use types to annotate inputs and outputs (basically attrs classes), for example:

@attr.s
class InputSpec:
    in_file: File = attr.ib(...)

@attr.s
class OutputSpec:
    out_file: File = attr.ib(...)

class MyTask(Task):
    input_spec = InputSpec
    output_spec = OutputSpec

    def run(self, runtime):
        # run task
        return runtime

This is a static example to give you the flavor. The MyTask class is sent to a runner along with a dictionary of inputs. We also have syntactic sugar for pure Python tasks:

@task
def mytask(in_file: File) -> File:
    ...

Here, the input/output specs are dynamically constructed. In order to dispatch them to run in separate interpreters or on other servers, some form of pickling is required. It's possible that there's a way to rewrite our code base to avoid it, but I found that when adding a NewType I had passing tests on Python 3.10 but errors like the following in Python 3.7-3.9:

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/concurrent/futures/process.py", line 205, in _sendback_result
    exception=exception))
  File "/Users/runner/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/multiprocessing/queues.py", line 358, in put
    obj = _ForkingPickler.dumps(obj)
  File "/Users/runner/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'NewType.<locals>.new_type'
"""

To be honest, I'm not even sure if we'll end up making File/Directory into NewTypes, so I may not need this feature, but I think having consistent implementations across Python versions would at least help determine if NewType is the right tool for the job.

[JelleZijlstra]

Thanks, that's helpful!