Add heartbeat RPC to detect failed clients

[MichaelBrim]

Description

The request manager thread for each client will send a periodic heartbeat RPC to the client to make sure it hasn't gone away unexpectedly. Currently, the period is set to 30 seconds. When a failure is detected, the request manager thread will exit and the client state will be cleaned by the main server thread.

Motivation and Context

See issues: #567 #646

How Has This Been Tested?

Tested in Docker Ubuntu by killing one of the unit tests while it was still executing and making sure the failure was detected and cleanup occurred.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Testing (addition of new tests or update to current tests)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the UnifyFS code style requirements.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted.

[adammoody]

It's hard for me to tell from the code, but assuming the resource manager thread has no work to do, how often are we pinging the client?

The comments above suggest the pthread_cond times out after 10ms.

[MichaelBrim]

The code in rm_heartbeat() makes it so it only does the RPC once every 30 seconds.

[adammoody]

Ok, thanks. Now I see it.