updates kill a container before trying to download the replacement image?

[SvenDowideit]

looking at the events, it appears that we pull after the kill succeeds - leaving us with 2 problems

1. the pull might take time
2. the pull might fail

especially when a container takes time to shutdown, wouldn't it be nicer to start the pull earlier (perhaps even before the update-delay time)

[stelund]

I'm working on a PR to make it optional to pre-allocate new tasks before shutting down old ones. I feel that feature is missing. Maybe it's what you are asking for too?

[SvenDowideit]

yeah - that feels similar too

[aluzzardi]

@stelund @SvenDowideit We used to set the desired state to READY but then I reverted it to ACCEPTED in #715.

It caused problems and I think the proper solution requires an "on-failure" strategy (continue, abort, rollback).

I agree its annoying.

@aaronlehmann wdyt?

[aaronlehmann]

@aluzzardi: Can you remind me what the issue was with setting the desired state to READY? #715 says:

However, with an invalid image, READY can't happen since the agent transitions from ACCEPTED to REJECTED.

But I don't remember why we can't handle the REJECTED case. What's the complication with a task not reaching READY?

If we need an on-failure strategy, there's a related proposal in #486.

[aluzzardi]

@aaronlehmann The update got stuck forever since we never reached the desired state.

But maybe we should just wait for >= DESIRED, it would have the same effect.

I think the pattern was: service update --image typo -> the entire update got stuck forever. The task would transition to REJECTED and I think the reconciliation loop kicked in.

[aaronlehmann]

But maybe we should just wait for >= DESIRED, it would have the same effect.

Yeah, I was thinking that. Do you see any reason it wouldn't work?

[aluzzardi]

@aaronlehmann I either had a good reason or was very tired. Both are likely :)

I think ACCEPTED is simply time bound - it will almost certainly happen (eventually). Not sure how thins changes anything but it was related to that.

[aaronlehmann]

@aluzzardi: Do you think it's worth trying to switch to >= READY? I could open a PR if you think this is a reasonable change to do now.

[aaronlehmann]

Opened #1199 to put restarted tasks straight into the READY desired state. This doesn't fix the main problem raised in this issue (tasks are killed before they are replaced), but it does help a bit by starting the pull before waiting for the restart-delay to elapse.

[aluzzardi]

With #1199 being merged and with #1177 in the roadmap, can we close this one down @aaronlehmann ?

[aaronlehmann]

Yes, we can replace this with #1177.