926 distributed training tests

[wyli]

Signed-off-by: Wenqi Li wenqil@nvidia.com

Fixes #926

Description

adds a utility for distributed tests

Status

Ready

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh --codeformat --coverage.
• Quick tests passed locally by running ./runtests.sh --quick.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[wyli]

- tested single machine multiprocess on macos and ubuntu w/o GPU

• implemented some utilities mainly for pickling test cases for multiprocessing. the decorator pattern could also be useful for unit tests with expected execution time #852

on windows CI there's an error for the distributed test OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading "C:\hostedtoolcache\windows\Python\3.8.6\x64\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll" or one of its dependencies. I think this is a limitation of the CI instance rather than a code issue. so I skip the tests on windows

updates:

• fixed windows issue with 3rd party github action al-cheb/configure-pagefile-action@v1.2
• tested single machine multiprocess on macos/ubuntu/windows w/o GPU

[wyli]

/integration-test

[Nic-Ma]

Let's use spawn for now, good enough for the first version, maybe some expert can help update it later.

Thanks.

[wyli]

Let's use spawn for now, good enough for the first version, maybe some expert can help update it later.

Thanks.

sure thanks! this also needs to be extended to multi-node test cases