experiments: support local parallel execution in temp directories

[pmrowla]

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.
  Maybe: see experiments: support local parallel execution in temp directories #4257 (comment) below.

• ❌ I will check DeepSource, CodeClimate, and other sanity checks below. (We consider them recommendatory and don't expect everything to be addressed. Please fix things that actually improve code or fix bugs.)

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

Related to #2799.

• Adds local temp directory executor and support for running experiments in parallel.
  • Only dvc repro is currently supported
  • Individual dvc run stages within a single repro command cannot be run in parallel
• dvc repro --experiment --queue can be used to add an experiment to the execution queue
• dvc repro --run-all will run all experiments currently in the queue
  • -j/--jobs can be used to run experiments in parallel, currently defaults to 1 (run sequentially), will need to decide if we should default to cpu count for this?
• Queued experiments are listed in dvc exp show and denoted with *

Next steps:

• Reorganize executor output collection (currently collection is done manually in the experiments classes rather than reading output from an executor's tree)
• Clean up executor classes so that they can be more easily extended to support other types of executors

[pmrowla]

[pmrowla]

Looks like there's a windows issue w/cleaning up the executor temp directories, I'll have to take a look into it tomorrow

[pmrowla]

The windows CI issue is resolved and this should be mergable.

Not sure what's going on with the lint build step, check_patch/pylint passes for me locally and in the travis build.
For some reason pylint in the lint step is reporting errors for pytest imports that I think are normally ignored:

pylint...................................................................Failed
- hook id: pylint
- exit code: 2

Registered custom plugin. Some checks will be disabled for tests.
************* Module tests.unit.repo.test_repo
tests/unit/repo/test_repo.py:15:1: E1102: pytest.mark.parametrize is not callable (not-callable)
tests/unit/repo/test_repo.py:33:1: E1102: pytest.mark.parametrize is not callable (not-callable)
************* Module tests.unit.remote.ssh.test_connection
tests/unit/remote/ssh/test_connection.py:87:1: E1102: pytest.mark.skipif is not callable (not-callable)
tests/unit/remote/ssh/test_connection.py:100:1: E1102: pytest.mark.skipif is not callable (not-callable)
...

[skshetry]

@pmrowla, could be related to pytest-dev/pytest#7558.
Pinning pytest to <6.0.0 should do. Let me create a PR.

[jorgeorpinel]

Hi! Wow, this looks cool 👍 👍

Docs-wise though, We have this section in the repro cmd ref, https://dvc.org/doc/command-reference/repro#parallel-stage-execution, reading:

Currently, dvc repro is not able to parallelize stage execution automatically. If you need to do this, you can launch dvc repro multiple times manually. For example...

Sounds like it needs an update?

dvc repro --experiment --queue can be used to add an experiment to the execution queue

In fact all this seems like several new options have been added, so def. needs docs ticket, I think.

Thanks

[pmrowla]

@jorgeorpinel yeah there will be docs tickets once behavior is finalized, but for now since everything is expected to continue to change before the feature is released (and since it's currently an experimental feature and disabled by default) I haven't submitted any docs PRs yet.

[jorgeorpinel]

Kk