Unifying third-party dependency management (through npm)

[petetnt]

The issue

Currently there are at least four different ways managing third-party dependencies in the Brackets core:

1. Git submodules
   • https://github.com/adobe/brackets/tree/master/src/thirdparty
   • https://github.com/adobe/brackets/tree/master/src/extensions/default/JSLint/thirdparty
2. node_modules committed into the folders themselves
   • https://github.com/adobe/brackets/tree/master/src/extensibility/node/node_modules
   • most recently this: 224633c
3. Plain dumps of the files
   • https://github.com/adobe/brackets/tree/master/src/thirdparty
   • https://github.com/adobe/brackets/tree/master/src/extensions/default/InlineColorEditor/thirdparty
4. devDependencies for Brackets itself:
   • https://github.com/adobe/brackets/blob/master/package.json#L15

and so on. Obviously every method has it's merits, for example in #11726 (comment) we had discussion on the merits of having submodules.

But let's take @busykai's comment on the merits of having submodules in that issue:

@zaggino, if you don't have a fork you never have a space to maneuver. If I remember correctly, we needed a modified version of CodeMirror on numerous occasions. jslint is another example. If you're using an npm version (i.e. a publicly distributed version) -- you're stuck. By stuck I mean if you fix it in the dependency upstream, you have to wait for 1) your contribution to be reviewed and accepted; 2) new package to be promoted and published. Logistically, it does not make any sense to wait for any of that. I don't know why would you assume it would never happen, it happened many times now.

Nowadays you can install npm install from any repo, from any branch, from any commit, from a tarball, pretty much from everywhere. You don't have to wait for an upstream fix if you don't want to, just push your fork to your own repo and change the dependency in package.json. When the upstream fix is ready, you can just change the dependency back and run npm install again. Sometimes you have to cherry pick commits or parts of them, and that's fine too. But hiding the third party dependencies to subfolders under subfolders easily lead (in my opinion) to heavily outdated dependencies and code rot. Just look at the current JSLint fork under https://github.com/adobe/brackets/tree/master/src/extensions/default/JSLint/thirdparty

This branch is 1 commit ahead, 223 commits behind douglascrockford:master

Which then leads into issues such as #12000. Who knows at what versions are the other dependencies at? At least I don't know, I assume fairly recent but I cannot be sure: hell, I cannot be sure what Brackets depends on without going through all the folders searching for node_modules, thirdparty folders, ´thirdparty-foo.js` files...

Proposed solution

Plain and simple: picking the de-facto way and running with it: managing all the core dependencies of Brackets through npm.

Potential problems / drawbacks of handling everything through npm

Handling all the dependencies through npm does have some small drawbacks that I can think of:

Increased maintenance cost

The maintenance cost of Brackets core can potentially increase if the dependencies are constantly kept up-to-date: this of course applies to other ways of maintaining the thirdparty deps too, but the simplicity of actually maintaining the dependencies might actually encourage people keep them up-to-date, which could increase the said cost.

There might be some potential conflicts if some of the extensions / core features use different versions of the same dependencies to boot.

Increased amount of third party files

This mostly applies to the few examples where the actual third party files are just dumped into the folders, but this is mostly a non-issue, as most of the files are already submodules or fully committed node_modules anyway.

Potential problems with different npm versions

Something could be borked with npm@2 that's fixed with npm@3

Summary of the benefits

Then consider the benefits:

• Having all the current dependencies in one place
• Unifying the way of introducing new third party dependencies (and removing unneeded once)
• A lot smaller initial size of the repo
• Simplifying the hacking process to git clone url/to/brackets && npm install
• Not having to deal with with submodules (I once again had issues with them with Porting tern analysis of JS code to Node. #11948 when the submodule paths had changed)

npm@3 has neat deduping capabilities and flat tree methology, which also removes many pains from Windows users (see: #11988 (comment) and #11988 (comment))

Obviously this would need some of initial work but in the end would be totally worth it. Opinions?

[zaggino]

As stated a few times before, I'm all in for using npm, for all the reasons mentioned above. I don't believe git submodules bring any benefits since you can always point an npm dependency to a repo & commit if required.

[ficristo]

While I'm totally in favor for this to happen I'm trying to think some drawbacks.

AFAIK Brackets architecture is divided in two: a part that resembles the browser and the part using node.
Is it possible to use npm for everything? Or do you think to use bower too?
How this will interact with the @zaggino port of Brackets-Electron?
(I see this port as a long-term project but the right move forward)
Will the extensions be impacted by this change? If so, how?

For the pros I will add that if we want to support node 4 having the dependencies up-to-date should somewhat a requirement. (Maybe they will work as is but in recent versions the support should be more guaranteed)

[petetnt]

@ficristo

• It is possible to use npm for everything from backend to frontend, which is pretty much the reason the reason bower's popularity is diminishing by the minute.
  • After module loaders and bundlers got popular on the browser too, there isn't much reason for two package managers, when one can handle everything and the other only half of it.
  • npm@3 also implemented what many consider bowers best feature: flat dependency tree. There's no more node_modules under node_modules under node_modules, but just a single node_modules-folder with all the modules (and their dependencies) inside of it when ever possible (conflicting versions are nested only)
• For the port question, this change would have only benefits for the brackets-electron project at best, no harm at worst
• There are few ways extensions could be affected:
  • If the extension directly (as in by using an absolute path, not sure if this is even possible) requires some of the third-party modules, the extension would need to be updated
  • If they depend on deprecated / outdated version of some third-party module that is included and subsequently updated, the extension would need to be updated
    • Personally I don't consider either of these a negative: pretty much just regular maintenance work
    • Most extensions seem to use their own third party modules anyway, the ones in Brackets core are fairly stable as-is
  • Good thing would be that there would be a standard way to require node-modules from the Brackets Core itself

The actual changes needed would be:

• Take third party dependencies out of the src folders...
• and add them to the package.json file and running npm install
  • Taking the current versions would be the safest way
  • Taking the most recent versions would be optimal, but might require more work if there are heavily outdated third party extensions
• Updating the references in code (all the require("thirdparty... lines)
  • If the versions are newer, issues would have to be fixed
• Updating Hacking on Brackets and similar guides accordingly
• And fixing the build process: https://github.com/adobe/brackets/blob/master/Gruntfile.js#L46

[ficristo]

@petetnt Thank you for the long explanation.
So if I understand correctly the way Node.js is used will not change and extensions will continue to use the Brackets DomainManager to access Node.js features.
I was thinking the dependencies for the default extensions would be installed through the package.json in the root, but instead those will continue to be bundled inside the extension itself. Or those could use the global node_modules?

[petetnt]

@ficristo Default extensions would/should use the node_modules in the Brackets core. Other extensions (the ones from the Extension Manager) would continue to bundle their own modules (but would be able to use the dependencies from Brackets Core too).

This change wouldn't affect the Node-features of Brackets itself, it would just unify the way the third party dependencies are handled inside the core.

[busykai]

A git submodule is a fully-functional repository you can work in, change branches and commit to, it's not just a dependency. It is many time (not always though) the case that you need to do development in parallel (take tern as an example). I don't see any benefit of "unifying" to npm just for the sake of it.

We can manage requirejs and its plugins via npm or jquery, but we cannot do the same for CodeMirror or acorn or tern. So "unification to unify" is a non-starter: one needs the latter to be repositories since you actually work in them.

Nowadays you can install npm install from any repo, from any branch, from any commit, from a tarball, pretty much from everywhere.

I know that. It creates a problem rather than resolves one.

Little bit of IMHO: in my experience, npm is quite troublesome in large enough projects if you don't fix the versions (see shrinkwrap). Relaxing module versions creates true chaos and unexpected issues from one day to another (if the version of a dependency of a dependency has been incompatibly updated). All this in the end just beats the purpose.

Also, npm dependencies has to be loaded differently. At the very least requirejs configuration has to be changed for the modules managed by npm. npm is not exactly browser-friendly distribution mechanism, it was created to be used with nodejs, uses node-specific name resolution etc. If we were to use something, we'd use bower, but bower is an utter mess.

And finally, this piece runs in Brackets' node context and cannot be managed by package.json in the repository root.

1. node_modules committed into the folders themselves
   https://github.com/adobe/brackets/tree/master/src/extensibility/node/node_modules
   most recently this: 224633c

this piece is used for CI in a headless Linux environment and ran in node environment (grunt), not by Brackets itself (or its node).

1. devDependencies for Brackets itself:
   https://github.com/adobe/brackets/blob/master/package.json#L15

@petetnt, I would revisit the problem you're trying to solve. If the problem is Brackets uses multiple ways to include 3rd party dependencies, then it's not a problem. It's actually OK. Each way of including a dependency has its reason. If the problem is you not liking submodules, then it's not objective. To start off, we need a good and objective problem statement.

P.S. If you like to, you could try moving requirejs to npm and see what it entails, but still... there's no hope we could move all 3rd party code to npm anyways. At least to not my understanding of Brackets.

[petetnt]

@busykai , thanks for your reply.

@petetnt, I would revisit the problem you're trying to solve. If the problem is Brackets uses multiple ways to include 3rd party dependencies, then it's not a problem. It's actually OK. Each way of including a dependency has its reason. If the problem is you not liking submodules, then it's not objective. To start off, we need a good and objective problem statement.

A git submodule is a fully-functional repository you can work in, change branches and commit to, it's not just a dependency. It is many time (not always though) the case that you need to do development in parallel (take tern as an example). I don't see any benefit of "unifying" to npm just for the sake of it.

I'd like to emphasize that I am not championing using npm because I am biased towards it, but because I'd like there to be an unified way of handling a major thing in a software that I love. In my opinion, having multiple ways of handling third party dependencies is not OK by any means: it's actually quite terrible, because it makes actually handling them pretty darn hard. Which is the exact reason package managers were created in the first place. Obviously all the four methods I mentioned in the original post are capable of handling one (or all) of the dependencies: I would like to find (and use) the most optimal one of them.

We can manage requirejs and its plugins via npm or jquery, but we cannot do the same for CodeMirror or acorn or tern. So "unification to unify" is a non-starter: one needs the latter to be repositories since you actually work in them.

CodeMirror is on npm: https://www.npmjs.com/package/codemirror
Acorn is on npm: https://www.npmjs.com/package/acorn
Tern is on npm: https://www.npmjs.com/package/tern

All of them are also hosted online on a git based platform (in these instances, GitHub), which means we can also install them directly from the repos themselves: from any version, from any commit. If there's need for cherry picking, then we would host them on it's own repository, just like JSLint is right now. And then we can just do:

  'jslint': 'peterflynn/jslint#5a09b35'

to install that exact version. Only 3rd party dep that I can find that isn't hosted directly on npm is the path-utils, which can be, once again, installed directly from the repo.

Little bit of IMHO: in my experience, npm is quite troublesome in large enough projects if you don't fix the versions (see shrinkwrap). Relaxing module versions creates true chaos and unexpected issues from one day to another (if the version of a dependency of a dependency has been incompatibly updated). All this in the end just beats the purpose.

Yes, shrinkwrapping is an option (which is relatively easy to do) and yes, super relaxed module versions can create issues. Obviously caution must be used when specifying (and updating!) the versions. That said, there is very little overlap in the sub-dependencies in Brackets (at least right now). Like I said previously, one of my motivations here is to prevent code rotting by actually encouraging updating the said modules.

Also, npm dependencies has to be loaded differently. At the very least requirejs configuration has to be changed for the modules managed by npm. npm is not exactly browser-friendly distribution mechanism, it was created to be used with nodejs, uses node-specific name resolution etc. If we were to use something, we'd use bower, but bower is an utter mess.

Actually, let's look at the RequireJS docs on loading node_modules

Best practice: Use npm to install Node-only packages/modules into the projects node_modules directory, but do not configure RequireJS to look inside the node_modules directory. Also avoid using relative module IDs to reference modules that are Node-only modules. So, do not do something like require("./node_modules/foo/foo").

By default requirejs looks for the modules in the standard location, that being node_modules in the root. Which means that the configuration needed would be less than minimal, is in removing code from the current code base.

Furthermore, there needs to be an certain distinction made here about npm and node_modules in general: npm was created for handling modules for node but in 2015 it is the de-facto way of handling all kinds of modules. All modules that are made for node that use node specific features need to be bundled (with things such as Browserify and Webpack to be used in browser, but not all modules in npm are meant to be used with node programs only. npm doesn't take a stance on which kinds of packages it offers: there are CommonJS modules, AMD modules, UMD modules, scripts that are exposed as globals, jQuery plugins. Things written in TypeScript, things written in ES2015, things written in Coffeescript, things written in ES5. Some work straight in a browser, some don't (and need to be bundled). All of the third party modules we are using right now can be handled through npm: otherwise we wouldn't be using them at all.

And finally, this piece runs in Brackets' node context and cannot be managed by package.json in the repository root.

This might be an issue though, at which point I guess dumping the node_modules in to the folder is the best way to go. Or setting up a simple npm-command that just cd's into that folder and runs npm install there.

P.S. If you like to, you could try moving requirejs to npm and see what it entails, but still... there's no hope we could move all 3rd party code to npm anyways. At least to not my understanding of Brackets.

I'd love to try it, but as my time has been quite tight as of late, I would like more comments from the owners, collaborators (like you) and community members, if such thing would be desirable before diving in because comments like yours are extremely valuable (even if you don't agree with me... yet 😸). Obviously I think it would be worth it and I am championing npm because for me, as a full-time JavaScript developer, makes the most sense to me. If someone has a better idea, I am of course all ears. If someone wants to convince me that the current way is the best we can do, or it turns out that way after actually implementing this, then at least we know that npm is not the way 👍

[busykai]

I know npm offers all types of libraries (with a lot of garbage files that come with it, by the way). It's just consuming them from there does not bring any specific benefit. Having said that, one minor comment on your long part about using requirejs which meant to educate me:

Actually, let's look at the RequireJS docs on loading node_modules

The doc you're referring to is about loading node modules using requirejs executed in node context. What you'd have to do is to load npm-fetched module with browser-run (i.e. in Brackets) requirejs. As the very least, it does not understand the format of package.json to find the entry point. As the way of fetching files, npm is the most ridiculous of all that exist.

CodeMirror is on npm: https://www.npmjs.com/package/codemirror
Acorn is on npm: https://www.npmjs.com/package/acorn
Tern is on npm: https://www.npmjs.com/package/tern

You're missing my point. I know they are on npm. You cannot modify what you bring from npm registry and commit it. You'd have to use npm link and other extra and unnecessary stuff to actually contribute the change back. It's inconvenient.

I'd love to try it, but as my time has been quite tight as of late

So is mine. Hence until someone actually creates a prototype which demonstrates how it's better, I'll refrain from the further discussion.

Let me summarize what i think before I go: to me, the issue does not have a sound problem statement. You are trying to address a development-time configuration of Brackets repository which so far did not cause any inconvenience. IMHO, it's just not worth typing so much characters as we already did. I think it could be improved, I just don't see how it can be unified for all types of dependencies and execution contexts that Brackets has. I do not understand why it has to be unified in the first place.

[zaggino]

Guys, I'll be short, but this discussion shouldn't be about having only npm dependencies or having only git submodule dependencies. I believe we should have both, deciding on each separate case, what's more appropriate.

Why are we resistant against npm dependencies? Please check out #11988 (comment) - why do we want to commit 10MB of node_modules to git? Just to avoid having to do npm install after cloning a repository? Is that really such a big problem to execute that one command, you usually do anyway for 99% of the web projects these days?

[petetnt]

@busykai I understand where you are coming from, but I don't agree with things like difficulty of npm linking_. I specified the pros and cons of the said approach and I think that the inconvenience *is_ there (just look at @zaggino's comment). If you think that the pros are not worth it (the why part of your doubt), then I might have done a poor job explaining the reasons. The how part I can demonstrate by creating a branch that demonstrates the benefits, but I don't also understand the beef you have with npm in general: claiming that as the way of fetching files, npm is the most ridiculous of all that exist is like claiming that apt-get is the most ridiculous way of installing files on Debian based distros, Homebrew is a ridiculous way of installing apps on Mac OS or that gem is a ridiculous way of installing Ruby gems.

[busykai]

Guys, I'll be short, but this discussion shouldn't be about having only npm dependencies or having only git submodule dependencies. I believe we should have both, deciding on each separate case, what's more appropriate.

@zaggino, my point. I'm not against npm (or bower or whatever other package manager). I'm against unification (simply because it's impossible). So I do agree with you.

Why are we resistant against npm dependencies?

FWIW, ESLint extension should be pre-installed, not embedded. So should be any other linter. I agree, we don't need all the node_modules garbage in git in your ESLint example.

But let's assume assume it's in git, for the sake of argument... These are truly node dependencies. They are loaded in node (node domain of ESLint) and executed there. There are multiple places where that code is checked in git and we can throw it away and use shrinkwrap or similar to ensure the exact copy of the code is fetched. I'm all for that (using npm where it's appropriate). We can rely on the Brackets developer to run a post-checkout hook which would update all the modules in places where they are necessary (src/extensibility, ESLint etc). I'm not resistant to this, you have to understand. If something is executed in Brackets node, then it's OK (and natural to use npm+shrinkwrap+provide a post-checkout script).

A note of caution: where use pretty old version of node and npm (did you use these to fetch the dependencies, by the way?). We'll have to upgrade them (and all the dependencies) and keep upgrading periodically for a sustainable npm-based solution. Now that all that io.js vs node.js storm is over, I think we could relatively safely switch to some 4.x.x node version. It's a lot of work though.

Note, however, this is not what @petetnt argues. Just take a look at the title of this issue.

as the way of fetching files, npm is the most ridiculous of all that exist

I repeat this again. Because requirejs in browser context will not load from node_modules. Nor the modules will lookup and look up their dependencies. Again, @petetnt , the only thing you need to do is to try and materialize your ideas. If you're running in node context, the only role of npm is to fetch you the files or you have to do what this article suggests. Even if you do, the unification will not be achieved. I do believe that you have an over-simplified understanding of what needs to be done over supposed benefits it brings. Please show the code which does what you want, we can then spend our time more productively.

[zaggino]

Already made and attempt and will try to work on node 4.x.x -> adobe/brackets-shell#543 but ESLint thingy should be safe. You're right that I've downloaded dependencies with node4 but my brackets use 0.10 (as everyones) and everything works (there are no native dependencies in the tree).

[petetnt]

I had some spare time today so I made the proof-of-concept, which is available here:

• Branch petetnt/brackets/poc-thirdparty-with-npm

Information about the branch itself

Even if this will never materialize, I am rather happy with the PoC: I learned tons more of Brackets internals which I didn't know previously. This proof-of-concept is a 99% working, 100% backwards compatible and extension friendly version of the current master branch with 99% of the third party dependencies handled with npm.

• petetnt@0838a03 demonstrates loading node_modules in Node context from Brackets folder by passing in an absolute path for the requires.
• This is also how Extension module loading is handled in general. See the ExtensionUtils comment on the bottom.

Missing pieces

The missing pieces are as follows:

• path-utils still missing the modularization, see Move PathUtils to Brackets core code? #11726. I'll do this tomorrow just in case because the fix is trivial (and useful later) Landed in: Create UMD-style wrapper for PathUtils and add a package.json jblas/path-utils#1, fixed in my branch
• thirdparty/globmatch.js is a mystery: this most likely should have been a Brackets core feature that requires minimatch and fnmatch in the first place. The "fix" is once again trivial.
• thirdparty/murmurhash3_gc.js has a similar problem to path-utils: alternative implementations exist and should be trivial to swap with.
• Didn't have the time to swap test-only deps under tests/thirdparty
  • ⏰ Due to time issues I didn't run tests yet in general so there might be something hiding I don't know of yet
• There are third party modules under src/widgets I only noticed now
• JavascriptCodeHints runs tern code in a worker. This requires a similar hack that extensions and the Node Context uses, which I didn't do yet.
  • Funnily enough this will also be fixed with Porting tern analysis of JS code to Node. #11948 and the Node approach

What is broken

The 1% of the Brackets is broken part is due to me updating React to 0.14 which breaks FileTreeView#L362 (and maybe some other parts there) - fix should be trivial and I'll do a separate PR for updating React in the current branch anyway soon-ish. Mustache that is used is a very old version so extension manager needed some work too, this is fixed in my branch: I could send another PR for updating Mustache too.
Both of these have pending PR's for the core regardless of this issue (see #12034, #12035) (Both have now landed to core)

Other notes:

• Use npm install to install the packages.
• The package.json uses npm@3 formatting: for backwards compatibility, remove all the github: mentions from the package.json.
• Due to the the worker issue, .gitmodules is still there.
• I upgraded all the third party modules except CodeMirror to the latest versions: only react has breaking changes and JSLint needs some additional configuring. Edit 07/2016: React has been updated in Update React to 0.14.7 and Immutable to 3.7.6. #12035 and JSLint is most likely deprecated.
• The requirejs config can be further simplified a lot if needed.
• Build process hasn't been touched / fixed / implemented so don't bother trying to build a release version from this yet 🚒
• jQuery, Mustache and less are added with <script>-tags in index.html - Here they have been moved to src/main#L112 and then exposed to window.
  • These cannot be removed from the window scope any time soon as they will pretty much break all the extensions. 😿 Maybe for 2.0.0, heh.
• Deps are now required with require("foo") or getModule("foo") instead of require("thirdparty/foo");
  • Deprecation warning could be thrown if an extension requires thirdparty/foo - They are still mapped to thirdparty/* with requirejs.config to maintain backwards compatibility.
  • If the removal of "thirdparty/"-notation is too drastic in the requires, we can also just name the keys as "thirdparty/foo" in src/main.js
• ExtensionUtils has a long running problem with not using the global require context, see [ExtensionLoader#L164](https://github.com/adobe/brackets/blob/master/src/utils/ExtensionLoader.js#L164)
  • I fixed this in ExtensionLoader.js#L69 - It's kinda ugly so most likely it would be better off to do in utils/global and expose the info as brackets.getGlobalRequireConfig(). utils/global already uses the semi-private requirejs.s.contexts. there, so using it there is relatively safe (see James Burges comment here). Most likely worth a PR on it's own. Edit 07/2016: PR here: Use the global RequireJS config for ExtensionLoader. Fixes #1087 #12041
  • This is the hack/workaround I mentioned above that Workers need too. Basically importScripts("src/utils/global") but I didn't have the time to test it yet.
  • Fixed in master with Use the global RequireJS config for ExtensionLoader. Fixes #1087 #12041
• Notes on CodeMirror in the wiki is a bit outdated