Monorepo with multiple builds steps/Docker images

[seniorquico]

We have a large, existing monorepo that contains our frontend code (JavaScript transpiled and published via Node.js and npm) and our backend code (C# via Mono 5).

I'm jazzed about the approach presented by the Shippable team for detecting specific changes in a monorepo:

http://blog.shippable.com/build-test-and-deploy-applications-independently-from-a-monorepo

However, I'm not sure how to configure Shippable to build our frontend and backend as separate pipeline steps using different Shippable Docker images. It looks like some others were looking for a way to achieve this, but nobody provided a response (#3024). Is it possible to use multiple shippable.yml files in a monorepo? If not, is our only option to create a monolithic Docker image that contains every build tool needed for all components?

Thanks!

• Kyle

[ttrahan]

Hi @seniorquico: We actually just published a new blog on using Shippable with a mono repo that addresses this scenario. It contains separate Dockerfiles for each component and only builds and pushes the images that have code changes.

It demonstrates the following scenario:

Check it out here: http://blog.shippable.com/ci/cd-of-microservices-using-mono-repos

In regards to your question, it is not possible to use multiple shippable.yml files in a single repo. However, you can define multiple Shippable workflow jobs within a single repo, which is an option that allows for pretty much unlimited configuration options for how to set up your workflow, as seen in the blog example.

[seniorquico]

@ttrahan Ah, I missed that blog post.

OK... going the multiple Shippable workflow jobs route... would the approach be along the lines of ditching the ciRepo resource and runCI job in exchange for a gitRepo resource (say with "buildOnPullRequest: true") and multiple runSh jobs? But... can we use custom Docker images with runSh? Looks like this is discussed in #3717, but that's not available yet(?). Can this be accomplished without #3717?

Thanks!

• Kyle

[ttrahan]

@seniorquico: You could certainly go that route to use runSh jobs instead of runCI jobs for this, but you're correct that this won't work for you if you're using custom images for your CI jobs. This is in development for runSh, but not launched yet.

In the interim, you could try a few different approaches to get you close to what you need:

• Set up a runSh job that will evaluate which components had code changes and will generate an output that stores a variable or set of variables in a Shippable params resource that you'll pass into a runCI job. Use these params values to set the custom image used in the pre_ci_boot section (you can use env variables in that yml block) and which CI job to run. You can even do this with matrix builds and spin up running CI and build/push for each component in separate runs for your job.

For example:

  pre_ci_boot:
    image_name: mydockerrepo/${image_to_use_for_ci}
    image_tag: imagetag
    pull: true
    options:

• Identify a base image common to all components and install additional dependencies, as needed, depending on which component you want to build. You could pass in an Input parameter, generated from an initial runSh job that identifies the component to build, to the runCI job and use that variable to determine which component to run CI on (i.e. which dependencies to install, which tests to run, etc.).

[seniorquico]

@ttrahan: The approach described in the first bullet sounds awesome!

To configure this scenario, would we simply switch off "Process Webhooks" on the project's settings and then configure a job as described? Or would it be more appropriate to leave "Process Webhooks" on and switch off all of "Commits", "Pull Requests", "Tags", and "Releases"? It's unclear if "Process Webhooks" has other consequences... I noticed it causes the project to show as "Paused".

Finally, I'm assuming we'd still need to setup the gitRepo resource. Or is it possible to feed an existing ciRepo resource into a runSh job?

Thanks!

• Kyle

[ttrahan]

@seniorquico: Yes, you can use either method to stop triggering the CI job from the source control webhook that's fired when a commit is made. Other inputs to the runCI job (like the params resource you'll set up) can still trigger the CI job when the webhooks are paused.

And yes, you will need to create a gitRepo resource as an input to the runSh job. Currently, gitRepo and ciRepo can't be shared, though this will consolidate in the future.

[seniorquico]

Thanks, @ttrahan!

[ttrahan]

@seniorquico: I apologize, but just tested and it looks like turning off "Process Webhooks" will pause the runCI job for all inputs, so leave that on and instead turn off the switches for "Commits", "Pull Requests", "Tags", and "Releases" in Project Settings.

[seniorquico]

@ttrahan Got it! Thanks for the help! 😄

[ttrahan]

@seniorquico: One last clarification. When passing in the variables as inputs to the runCI job, these will be injected as global variables, not matrix variables. So, perhaps a better method, would be to trigger individual CI jobs for each component that has changed by calling the Shippable API from the runSh job. The process would be:

1. runSh job generates list of changed folders (i.e. components)
2. runSh job triggers a new build job via the Shippable API for each component in the list
3. A CI job will run for each component

[sheats]

@ttrahan this is good and almost what I was looking for... What if my microservices are in multiple languages -- seems like the language definition in the root shippable.yml only allows one language?

[sheats]

You can disregard, I found #1222

[seniorquico]

@ttrahan I'm finally building this solution, and I have a follow-up question to your final comment from Oct 18. Is it somehow possible to get back to the original approach you described? Namely:

You can even do this with matrix builds and spin up running CI and build/push for each component in separate runs for your job.

It sounds like input resources to a runCI job and global environment variables injected by a Shippable API call don't work. Is it possible to dynamically configure a matrix build? Or are matrix builds simply limited to the static configuration in shippable.yml?

I'm thinking of the nicety of having the various component builds roll up into a single entry in our build history. If we dispatch multiple build jobs using the Shippable API, won't we have to manually correlate multiple entries in our build history?

Thanks!

• Kyle

[seniorquico]

This looks related to #2714.