New way to handle packaging

[amccollum]

Ok, so I've updated my previous changeset that implements a new Module class at runtime. The changes fall across 7 different modules:

• Ender
• ender-args-parser
• ender-builder
• ender-dependency-graph
• ender-package-util
• ender-js
• ender-commonjs

Each of the above has a new-packaging branch that includes the changes. The changes in most of the modules are fairly minor, with the lion's share of the changes coming in the ender-builder package in ender-js/ender-builder@4439aee

That commit is huge, so I've tried to comment on some of the larger changes inline to explain what's going on.

It's also worth looking at the new ender-commonjs module, which defines the new way packages are created using a Module class at runtime.

Here's a short summary what these changes bring:

• New Module object at runtime, supporting multi-file modules, internal/relative requires, requiring submodules, and better CommonJS compatibility
• Separated the CommonJS client lib from the Ender core
• Broken out assembly process, allowing for more control over the final build, or possibly replacing CommonJS with other package formats
• However, to better support CommonJS semantics, this change removes support for using an array of files for the main or bridge, the fix being to either use internal requires to mimic this behavior, or force packages to do their own concatenation

I updated all the current tests to work with these changes, but I haven't gotten around to writing tests for the new assemble.js file, yet. This leaves some holes in the test coverage, mainly around the specific content of the final build. It may take me a little while to get around to this, but I wanted to get the rest of the changes out for comment while I finished up the tests. There may be other bugs as well.

@rvagg, I'd love to hear your thoughts in particular, because I know you were a bit skeptical of some of these changes when I submitted them before.

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/969742-new-way-to-handle-packaging?utm_campaign=plugin&utm_content=tracker%2F165667&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F165667&utm_medium=issues&utm_source=github).

[ded]

just about to hit the sack for the evening until I read all these changed. I think the direction is pretty awesome given the stated goals. internal requires has been a feature even I have wanted for a long time.

I think one thing that should remain a goal is to keep the client interface, even if it isn't the default build (eg: the jQuery syntax).

Also, quite a number of popular modules (with old bridges) would have to get updated build processes

[amccollum]

Yeah, to be clear, I definitely think the client-interface is integral to Ender, even if technically it can be replaced. Part of what makes Ender great is that it is opinionated about a small number of core decisions, so other modules can build on those common assumptions. Another example of this behavior is that even with the new Module class, we auto-require every module (and bridge file) once on startup, in case modules modify the global context or need to do other initialization. While this isn't strictly following how Node behaves (you have to require a module once before its code runs), I think it's pretty important for striking a compromise for convenience in a browser environment.

I'm not sure how many modules use the array convenience for their main or bridge files, but I would guess the number is small. I did some analysis on all the Ender modules I could find in the wild back when I was contemplating some of these larger changes to the semantics, so maybe I should go back and update that to get a more precise count.

[ded]

would be great to know! As long as none are mine 😉

[luk-]

More control of the build is awesome, especially if it could make an easier road to getting source maps implemented.

[rvagg]

Repeating my comment here for clarity: IMO: at this stage, since we mostly seem to be time-poor, he who does the work, gets to decide on what gets done, so over to you! Thanks for picking up some of the slack.

@amccollum if you have a vision for how this should work then I'm behind you going for it, my vision is limited at the moment because I'm stretched across too many things.

[ded]

i tried reading thru each of the commits. does this finally give us support to include local files in packages....eg:

var parallel  = require('when/parallel')

parallel([a, b, c]).then(done)

[amccollum]

Yeah, it does. It supports both relative includes within a package and cross-package submodule includes.

On Tue, Aug 13, 2013 at 7:53 PM, Dustin Diaz notifications@github.com
wrote:

i tried reading thru each of the commits. does this finally give us support to include local files in packages....eg:

var parallel  = require('when/parallel')
parallel([a, b, c]).then(done)

---

Reply to this email directly or view it on GitHub:
#181 (comment)

[ded]

it would be great to have these branches published to npm as ender@2.0.0-devso can immediately get a feel for them.

[amccollum]

Ok, I think this branch is ready for others to try out. I had to remove some tests that were obsoleted by these changes, but there is currently only one test failing (a minifier test that seems unrelated, but I need to keep trying to track down the cause). I also added tests for the parts of ender-builder that were untested before.

A new regression in this build is the removal of the sans and noop arguments. These need to be rethought in a multiple base package world. In the case of sans, I think the solution is probably for you to be able to do something like --client-lib noop instead of --sans. The noop case is a little trickier since we're doing more than just concatenation now. My preference would probably be to just kill it, but there could be an alternate assembly mechanism that went back to the old behavior if it turns out this is a commonly-used option.

I've updated all the modified modules to have -dev versions, but I don't have permission to publish new versions of the modules to NPM.

[luk-]

@rvagg can you npm owner add amccollum ender please?

[rvagg]

done, and all the associated packages too, let me know if I've missed anything @amccollum, when you publish just remember to publish with --tag dev so they don't go under 'latest' and get auto-downloaded when people install (the actual versions are irrelevant) -- in case you didn't know.

[rvagg]

oh, and I think you might have to pin exact versions across the packages otherwise npm will try and match 'latest' versions. So you can't use fuzzy matching like '~0.1.2-dev', you have to '0.1.2-dev', which gets really annoying when you want to update stuff.

[amccollum]

Ok, I published this version under ender-2.0.0-dev. Let me know if you guys run to issues (which is definitely possible). There are a few aspects of the current code that could use some refactoring, so that's where my focus will be next.

[ded]

@amccollum @rvagg k. i'll have a look now given that i was pretty excited about some of the new features you introduced...

[amccollum]

I'm actually about to push some other improvements I've been working on that required a more major refactoring. For example, Bower support as the repository backend.

On Thu, Nov 7, 2013 at 1:52 PM, Dustin Diaz notifications@github.com
wrote:

@amccollum @rvagg k. i'll have a look now given that i was pretty excited about some of the new features you introduced...

Reply to this email directly or view it on GitHub:
#181 (comment)