Supporting graceful upgrades and zero-downtime dpeloys

[prologic]

Hi All,

I see there are talks about plans to support this in the future which is great. What have we come up with so far? I'm starting to think about this more with the stuff I do and would like to see something get started on this. I was at least initially thinking of an upgrade command that gracefully pulls down the latest image for a service, spawns a new instance and terminates the old one.

I think things like (typical web production stuff) like database migrations/upgrades, load balancer raining and switching are likely out-of-scope for Docker Compose and should probably be handled by some other decoupled tooling. But I'm not sure yet; I'm just thinking about the "first step".

Thoughts?

[prologic]

The only related issues I've found so far are:

• Not recommended for production use. Why? #1264
• Do you have any documentation about downtime during fig up? #1035

[ccll]

+1

@prologic
I'm facing the same problem as you, fortunately I figured out a approach today right after google directed me to this post.
Suppose we have 10 instances of container 'web' running in the cluster, and we are upgrading 3 instances at a time, now the basic steps for the rolling upgrade script:

1. docker-compose pull to pull the latest image.
2. docker-compose scale web=13 to create 3 instances of the latest version, which will be named as 'web_11', 'web_12' and 'web_13'.
3. Move the first 3 instances(web_1, web_2, web_3) out of LB, the traffic will be gradually drained to other instances.
4. Wait until netstat shows there are no connections to the first 3 instances, then we can safely docker rm them.
5. Repeat from step 2 until all 10 instances are swapped to latest version.
6. Final step, docker rename the 'web_11', 'web_12', ... to 'web_1', 'web_2' ..., make everything clean and tidy.

I've tested the process manually and it works great, and a basic automated script is not very hard to write, but in a production environment we need to handle failures, rollbacks or resumes of upgrading, which complicates things a bit, so an official support on this will be great.

A minor problem: I couldn't find a way to spawn new containers affinitied to corresponding old containers with docker-compose scale, e.g. spawn 'web_11' on the same host as 'web_1', 'web_12' on the same host as 'web_2', etc, I saw an article from quay.io did it this way, but I have no idea if this is important in some situation or not, if anybody knows please tell me.

[prologic]

Is something like this considered genreal purpose and useful enough? Or do we also need to think about making "draining load balancer" a plugable thing?

[prologic]

Not a lot of movement on this one (even from me!); How about something as simple as a docker-compose upgrade <service> (destructive). I'm thinking:

• pull latest image
• stop and re-create service

[jayfk]

The main problem I see right now is that links are not updated. If you want to do something like @ccll you need to add your own container discovery service.

[aanand]

@jayfk We shouldn't worry too much about links when designing future stuff (especially for production), as it's soon to be replaced by new networking constructs (see #1676).

[prologic]

@ccll Do you have a script/tool that does what you described that you're willing to share? Otherwise I'm going to prototype up something (be good to leverage of your efforts).

[ccll]

Do you have a script/tool that does what you described that you're willing to share? Otherwise I'm going to prototype up something (be good to leverage of your efforts).

@prologic
Not yet, our project is still far to production stage, so I didn't bother to write it yet, sorry!

[prologic]

In that case I've almost completed a general solution that will work with the autodock platform.

[dnephin]

Consolidating all production related issues into #1786