feature: including external docker-compose.yml

[dnephin]

The Problem

I'd like to use fig for acceptance testing of services

In the case of a single service with a single level of dependencies this is easy (ex: a webapp with a database). As soon as the dependency tree grows to depth > 1, it gets more complicated (ex: service-a requires service-b which requires a database). Currently I'd have to specify the service dependency tree in each fig.yml. This is not ideal because as the tree grows, we end up with a lot of duplication, and having to update a few different fig.yml's in different projects when a service adds/changes a dependency is not great.

Proposed Solution

Support an include section in the fig.yml which contains url/paths to other fig.yml files. These files would be included in a Project so that the top-level fig.yaml can refer to them using <proejct>_<service>.

Example config

include:
    # link using http
    servicea:
        http: http://localhost:8080/projects/servicea/fig.yml
    # local paths
    serviceb:
        path: ../projects/serviceb/fig.yml

webapp:
    ...
    links:
        - servicea_webapp
        - serviceb_webapp

[dnephin]

I'm looking to implement this features myself, but I would like to upstream it eventually, so I'm interested to see how you feel about this idea.

I've looked over the code, and it seems to be relatively easy to make this work.

Some unresolved issues are:

• including services which use image should be easy enough, but services with build would need to ensure that the naming remains consistent, and that the service was already built (and possibly pushed to a registry). See next two issues
• in the case of remote repos, the config may need to support a repo name as well (as part of the includes section).
• project naming. Instead of using the file path/directory for a project, it uses the key from the parent yaml config (the file with the include section). This poses a problem if the names don't match. I was considering adding a project_name key to the config to keep these consistent (this is not really critical)

For fetching the external configs. I was going to use requests since it's already a dependency, but I was also considering supporting git as another method, possibly using dulwich

[dnephin]

I have a working prototype with a single integration test in https://github.com/dnephin/fig/compare/include_remote_config

It still needs some cleanup and more tests, but I thought I'd link to the current progress.

[bfirsh]

Nice!

[suzaku]

@dnephin @bfirsh +1 for the new feature.

I'm looking for a way to reuse fig.yml from another project,
before something like include is available,
I'll have to fallback to using copy/paste/edit ;(

[pirelenito]

That is awesome! I'd love to get this feature as well. Any updates on merging this on master @dnephin ?

[dnephin]

Cool, glad you are interested! I haven't had much time to finish this off yet. I still plan on doing it, but it might still be a week or two out. I've been thinking it actually probably requires #457 to deal with including any services that use build instead of image (because you wont be able to build them without the whole project).

The idea would be to use one of the tags as the image, and assume the other project has properly pushed the images and they are available to be pulled (or already cached locally).

[pirelenito]

Nice!

For a remote or standalone fig.yml file your solution of using tags seems pretty reasonable.

What are your thoughts if the fig.yml file is available locally inside its own project (meaning everything would be there)? Could the build be triggered? It seems to make sense.

[dnephin]

I haven't thought much about using it with locally available fig.yml. In that kind of setup would you have different services in subdirectories and you want to include them so that each individual one is smaller?

I think defaulting to use the build if no tags are provided is probably reasonable. It might need to do something with relative paths (append the path given in the include section to any build paths in the included fig.yml).