Fig for production deployment

[olalonde]

Hi fig team!

I'm currently exploring options for automating the deployment of my app into production (fig does an awesome job locally). I have looked into the following solutions: Deis, Flynn, Kubernetes, Terraform, Atlas, Bosh, CloudFoundry, etc. but they all operate either at a too low or too high level of abstraction.

My app consists of 10 git repositories, each with its own Dockerfile. Each service is a REST API and depends on a database service (so a minimum of 20 docker containers are needed to run the whole app). Only one service (the gateway API) is exposed externally. The REST API services can theoretically be replicated/load balanced automatically while it would be a bit harder to do for the database services.

Theoretically, the fig.yml file could be used to specify which services can safely be "auto-scaled/load balanced", which data needs to be persisted and there could be a deploy tool that understands the fig.yml and be able deploy my app on a IaaS provider (e.g. AWS, Kubernetes cluster). It would be even better if external self contained fig based projects could be referenced from a fig.yml file as logical services.

My question is, does such a tool exist? Or does the fig team plan to work on something along those lines? Just food for thought... sorry if this is a duplicate.

[dnephin]

For referencing external fig.yml, see #318 and #758

I think for production systems, you end up needing a lot more configuration. fig.yml covers the topology, but then you also need: replication level, monitoring, load balancing, deployment, etc. It might be interesting if a tool supported fig.yml, but accepted other configuration files to cover the rest. I don't know of anything supporting that yet.

[olalonde]

I think most of the points you mention could be automated using another tool that understands the fig.yml format (perhaps the format could be extended to specify which services can be safely replicated). Load balancing nodes could be added to the topology (e.g. via a nginx container). Deployment could be done with a tool that understands the fig.yml format and can materialize it on an IaaS provider using their API (e.g. Terraform, BOSH). Monitoring can be done at a high level using IaaS provider tools or at a lower level by adding some monitoring server containers to the topology and making sure the other containers are linked to it and know how to report their metrics/logs. I think having two separate sets of configuration files (one for development and one for production deployment) is against the DRY principle. Of course, it wouldn't cover all use cases, but probably 80% of them.

Anyways, just food for thought.

[funkyfuture]

@olalonde concerning automation, #988 is propably worth a read.
anyway, can this issue be closed?

[bfirsh]

@olalonde Thanks for the question!

We're currently working on including configs in other configs: #988

We'd really like to make the Compose format a standard format that can be consumed by deploy tools, as you suggest. We're talking to a bunch of toolmakers about how this could work, and also thinking about how we can do this with Compose and Swarm:

http://blog.docker.com/2015/02/orchestrating-docker-with-machine-swarm-and-compose/

Would be interested in your feedback on that. We're also going to document some of the ways Compose is useful in production: #944

Hope that helps!

[olalonde]

Thanks, will follow the conversation at #988