Automatic restarts for Containers and Executables #32
Description
Motivation
There are cases when a Container or Executable used as a dependency service is not as resilient to transient errors as one would like it to be. In those cases an automatic restart upon failure is a big boost to developer productivity.
Initial design thoughts
If we do this, we should be very careful about avoiding thrashing, i.e. restarting in a tight loop. K8s does it well with "exponential backoff" for repeatedly failing restarts.
For containers we might be able to leverage what the container orchestrator supports, but the support may vary from orchestrator to orchestrator, so DCP might need to backfill in some cases and we might model the restart support as optional in our container orchestrator abstraction layer.