Improve retry policy

[r4victor]

Currently, dstack's retry policy is very limited – it only works for interrupted spot jobs (despite its description that the run is retried on failure). To make retry policy useful, it should cover common use cases, including the following:

1. When I run a production service, I want to always restart the job if it fails for any reason (possibly with some large duration limit).
2. When I run a one-time task, I want to retry provisioning only to wait for capacity. I don't want job being restarted if there is a problem with my code.

The current retry policy specification looks like this:

retry_policy:
  retry: true
  duration: 1h

We should introduce new values for retry:

• retry: always – always retries the job unless explicitly stopped
• retry: no-capacity – retries on no capacity/interruption but not if the job failed
• retry: never – default

So retry_policy could look like this:

retry_policy:
  retry: always
  duration: 1h

To specify different retry policies via CLI, we could allow specifying them in --retry:

dstack run . --retry=always --retry-duration=1h

The semantics of duration should also be changed and clarified. Currently, the duration is calculated from the job submission time. It should be calculated from the last failure time (or job submission time for new jobs) so that retry policy can be used to retry production services.

[peterschmidt85]

I'd suggest a bit more compact/human-friendly but explicit syntax for YAML (like on n GitHub):

retry:
  on: [no-capacity, interruption]
  duration: 1h

and

retry:
  on: no-capacity
  duration: 1h

[r4victor]

@peterschmidt85, how would I use the syntax you're suggesting to always retry the job? Do we need other on values besides [no-capacity, interruption] like error?

[peterschmidt85]

@peterschmidt85, how would I use the syntax you're suggesting to always retry the job? Do we need other on values besides [no-capacity, interruption] like error?

I personally, never had a need to restart a job on an error. We could support error later if there is such a need.

[r4victor]

There should be a way to always restart jobs to support running production services. It's in the issue description.

[peterschmidt85]

There should be a way to always restart jobs to support running production services. It's in the issue description.

Ahh, makes sense. Then let's support error too.

[r4victor]

@peterschmidt85, we're currently using pyyaml for YAML parsing that supports YAML 1.1 but not YAML 1.2 (see yaml/pyyaml#555)

In YAML 1.1, on is interpreted as boolean, so it needs to be quoted when using it as a property name.

I suggest we choose a different property name to avoid problems with YAML 1.1 such as on_events:

retry:
  on_events: [no-capacity, interruption]
  duration: 1h