RFC: Monorepo

[bajtos]

In the past year or two, we have repeatedly came across the idea of using a single monorepo containing all core LoopBack modules (strong-remoting, juggler, loopback, possibly others like swagger, etc.). The last time this was mentioned by @ritch when discussing loopback-next kick-off.

In this thread, I'd like to discuss pros and cons of using monorepo approach for the development of loopback-next (and possibly beyond).

@raymondfeng @superkhau @ritch

[bajtos]

I have several concerns about the monorepo approach:

1) Modular design

So far, LoopBack was designed as a composition of smaller building blocks (juggler, remoting) that can be used on their own. In which case I think it's better for both strong-remoting and juggler to live in their own repository that can be versioned independently and have project-focused issue tracker where people interested only in a particular building block can follow the discussions without being overwhelmed by everything else happening in LoopBack core.

Having said that, it doesn't look like there are many people using strong-remoting or juggler without loopback (loopback downloads: 58k/month, juggler downloads: 58k/month, remoting: 52k/month, see also https://www.npmjs.com/browse/depended/strong-remoting and https://www.npmjs.com/browse/depended/loopback-datasource-juggler), so maybe it's time to get rid of this layered design and move all code into a single Node.js project/npm package?

2) Back/forward-porting changes, git history

With 3.x version living in multiple GitHub repositories, and the NEXT version living in a single monorepo, it will be difficult to back/forward port changes across major versions, as git cherry-pick will no longer work.

Also if we eventually decide to copy code from loopback-next monorepo into individual projects we have now, the commit history and links to valuable discussions in pull requests will be lost.

---

Here is the alternative solution I would like to propose for consideration:

• Create a monorepo in loopback-next that will be used only as a playground for trying out new ideas and getting feedback.
• Regularly sync code from master of individual loopback projects into this mono repo (but not the other way around!).
• Once an idea gets mature enough that we know what kind of changes we want/need to make in each individual repository, then the real changes will be sent in multiple pull requests to these repos.

[bajtos]

Related reading:

• https://medium.com/@bebraw/the-case-for-monorepos-907c1361708a#.90y1b43up
• http://danluu.com/monorepo/

The first article recommends https://lernajs.io/ as the tool to use for managing a monorepo containing multiple npmjs packages.

[ritch]

Here are my take aways from above:

1. no one uses our modules stand alone, even though they can
2. module design and monorepos are compatible concepts
3. switching to a monorepo is hard (mainly because of back porting and lost history)
4. @bajtos and I are willing to try this monorepo for the development of 4.x (but not sure about beyond)

---

So I think the things for consideration are:

1. Do the PROs of a monorepo offset 3 enough? I think we can determine this after we get deeper into 4.x development in this repo. It probably won't take long to see if its going to offset the other headaches.

2. Should we de-emphasize the idea of using these other components on their own (juggler / remoting)? Note: we can still keep our modular design.

[raymondfeng]

Please also consider the impact to CI our systems which may have assumptions about the module/repo layout.

[superkhau]

no one uses our modules stand alone, even though they can

I completely agree with this. IMO, if we're going to sell Juggler as it's own thing, why attach the LoopBack name to it. Could be like waterline from sails for example. Maybe people can swap ORMs in the future for one they prefer, etc. I'm down with unix philosophy of do one thing (which means more repos really). We shouldn't have to rewrite large portions because we coupled our components too much, we need better decouplling and better extensibility. What if they want to swap strong-remoting or boot itself. I think we need to design from a different perspective: extensibility first, which means end users will have to hook in via other GitHub repos (and we're going to dogfood our own extensibility API first).

Anyways, one huge benefit of monorepo I can see is centralizing issues to one github repo which will make keeping track and searching through issues much easier during triage/roadmapping compared to what we have now.

I'm also concerned about CI and the line-ups that I can already see forming. Less concerned when it's distributed over N repos like we do now and take advantage of free open source CI systems like traivs (which can run in parallel over N projects).

[ritch]

For now the approach is to try lernajs as part of the general 4.x refactor / rewrite in TS. If it ultimately does not work out lets open another discussion. Closing for now to keep focused on moving forward.