Add random delay during multiple job launch to break tie

[IsaacYangSLA]

Signed-off-by: Isaac Yang isaacy@nvidia.com

Fixes #1236 .

Description

GPU idle detection may have a race condition if multiple jobs are launched at nearly the same time. Add some random delay to break the tie.

The logs from four jobs in reported failed pipeline showed all of them were assigned to GPU 0 and 3. That meant all of them reached the detection code at the same time and at the time, none of them was using GPU.

The number 16 can be larger so the probability of two jobs being launched side-by-side is reduced.
The number 60 is to give time for unittest code to start allocating GPU memory, and for other instance to be aware of GPU utilization by others.

Status

Ready

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh --codeformat --coverage.
• Quick tests passed locally by running ./runtests.sh --quick.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[wyli]

thanks!