imperative imports

[dherman]

ES2015 specifies a sequentialized evaluation order of modules, where the particular syntactic location of an import statement in a module body has no effect on the order of evaluation. Instead, the semantics simply determines a module's list of direct dependencies and recursively calls ModuleEvaluation() on each of them in order before evaluating the body. I'll refer to this semantics as declarative imports.

An alternative semantics would be an interleaved evaluation order. In this semantics, the evaluation order is sensitive to the particular syntactic location of an import statement. The evaluation semantics would be simply to execute the module body in order, and each ImportStatement would recursively call ModuleEvaluation() on that dependency. I'll refer to this semantics as imperative imports.

It's not too late to change

Existing implementations of module systems in production use today vary between these two semantics. As I understand it, AMD and YUI have declarative imports, CJS And Node have imperative imports, some ES6 transpilers follow the currently specified declarative semantics, and others such as Babel and TypeScript can configurably target AMD or CJS and inherit the semantics of the target system depending on that configuration. Meanwhile, no browsers are yet shipping native support for ES6 modules.

So my conclusion is that it's not too late to change this part of the spec.

Impact on top-level await

One consideration we would have to work out is what becomes of the semantics of top-level await. The only two options I can think of are:

1. the presence of any dependencies on asynchronous modules (i.e., modules that contain top-level await or that import from other asynchronous modules) automatically switches a module's evaluation order to declarative; or
2. imperative imports from asynchronous modules implicitly await.

(1) is pretty silly and a refactoring hazard to boot. I think (2) is natural and reasonable, but it does have the consequence of being a slightly nonobvious await point:

console.log("before");
import "some-async-module"; // implicitly blocks; remainder of module body does not execute in this turn
console.log("after");

The case for imperative imports

Compatibility with Node

Node is clearly the largest ecosystem, and consequently the most likely one to hit on compatibility problems with declarative imports. This is IMO probably the strongest argument for imperative imports.

Connecting side effects to statements

Since module initialization is effectful, imperative imports arguably give programmers clearer control over those side effects. I don't find this argument quite as compelling, since memoization means you can't actually predict whether the side effects are occurring now or have already occurred some time earlier. You can at least claim that it's more consistent with the familiar precedent of require() -- although this is roughly equivalent to the previous argument.

Increased expressivity

In the presence of cycles, imperative imports are technically more expressive. For example, two mutually recursive classes can coordinate to initialize their definitions and then add static instances of one another, without requiring a third module to orchestrate the initialization:

// module A
export default class A { ... } // create class A
import B from "B";             // SYNCHRONIZATION POINT
A.B_INSTANCE = new B();        // use class B

// module B
export default class B { ... } // create class B
import A from "A";             // SYNCHRONIZATION POINT
B.A_INSTANCE = new A();        // use class A

That said, this is an extremely subtle and fragile dependence on the execution order. In practice, most developers do not have a good mental model for the execution order of cycles and are unlikely to want to depend on these kinds of subtleties. So in that sense this argument is somewhat weak. But I suspect this means fewer cyclic dependency graphs will break.

The case for declarative imports

Simpler top-level await story

The story of top-level await is arguably more straightforward with declarative imports: the evaluation of the entire module body is simply blocked on completion of its dependencies. This doesn't introduce an implicit await, whereas imperative imports implicitly await at the point of importing an asynchronous module.

Refactoring

Since declarative imports do not depend on the order of imports, you can reorder them without changing the behavior of your program. This gives programmers more flexibility to group their module bodies the way feels best to them.

No harm done

Large ecosystems like the Ember community have been using the ES6 semantics for years without trouble, so it does not appear to have been a problem in practice.

Simpler spec

Declarative evaluation is simpler to spec than imperative evaluation, which requires the evaluation semantics of import statements to coordinate with the rest of the module semantics. I reject this argument entirely -- if @bterlson has to suffer to make JS developers' lives better, so be it! ;P

Conclusion

IMO, imperative imports come out ahead here: in particular, the compatibility and familiarity from existing module systems is the strongest argument I see. Of the arguments I've enumerated, the only one I can see that I find a bit concerning about imperative imports is the implicit await.

But I may well have missed arguments, so let's talk!

[bmeck]

imperative imports from asynchronous modules implicitly await.

Does this mean we must wait on the event loop to turn to finish an import sometimes (depending if the dependency is async/sync)?

[creationix]

The more I think about this, the more I'm concerned that top-level await would actually make interop with node harder. And if this makes node interop harder instead of easier, the declarative imports comes out a much bigger win.

[creationix]

Also as a bit of anecdote, I had added top-level yield to luvit modules (a lua clone of node.js) by re-implementing the require system in lua.

After several years, only a tiny fraction of users actually took advantage of the edge case you mentioned (namely my lit project). Eventually I decided to drop the hack to regain the more valuable interop with native lua require. The change to my code was painful, but quite straightforward luvit/lit@9b7ad7a.

If I could go back, I would have never added the ability to yield at the top level during require.

[caridy]

@bmeck this doesn't add considerable implications on the proposal to solve the interop with node, we talked about it, we should be good either way.

[allenwb]

@dherman
It seems to me that you actually made a stronger argument in favor of the current declarative model then you did for the imperative model.

The top-level await issues seem particular concerning as not really fully worked out.

Regarding, Node familiarity for the imperative model. Are the sorts of circularites where it makes a difference really very common? Aren't we really talking about initialization dependencies that involve multiple export/import bindings which really isn't something that the current node modules doesn't actually support.

Finally, I assume that in you imperative examples where you say "SYNCHRONIZATION POINT" all you really mean is run the other modules initialization (to completion) if it hasn't already run. Rather than some sort of corountine-like back and forth transfer of control. If so, then you can still get ordering related TDZ initialization failures.

[dherman]

@creationix We already have a good interop story for top-level await with Node. (Interop is a key constraint and not something to casually drop!) As @caridy says, the interop questions with top-level await are orthogonal to this issue.

[dherman]

@allenwb

It seems to me that you actually made a stronger argument in favor of the current declarative model then you did for the imperative model.

I was definitely trying to be balanced and bring out all the arguments I could think of, but I explained why I considered the arguments for declarative to be weak. If you believe they're stronger, it would help to explain which arguments you think are stronger than interop with Node and why.

Finally, I assume that in you imperative examples where you say "SYNCHRONIZATION POINT" all you really mean is run the other modules initialization (to completion) if it hasn't already run.

Hm, no, not if I understand what you're saying -- if you step through the semantics of imperative import, you'll see that whichever module starts executing first, it will start executing the other one when it hits the import line. Then the second one will run to its import line, which will be a no-op since the first module has already started evaluating (no coroutining here -- just a straightforward, on-the-fly memoizing semantics, just like require in Node). So by the time you get to the end of the "sync point" of either module, you know both modules have executed at least to their respective sync points.

[creationix]

Assuming the interop story is solved with top-level await: As a long-time node user, I would much prefer to gain the advantages noted for declarative imports. I place very little value on familiarity with the problematic semantics of node's module system.

I'm glad interop is top priority, but aside from that, easier refactoring and simpler spec are much more important than keeping semantics problematic.

[dherman]

@creationix

easier refactoring

The refactoring difference is quite minimal, which is why I don't weight that argument heavily. In practice people move around their top-level require statements in Node and don't worry about the different order of evaluation. Not only that, but the relative ordering of imports is still significant even in the declarative semantics (import "foo"; import "bar"; runs "foo" then "bar", whereas import "bar"; import "foo"; runs "bar" then "foo"). So the difference in degree of refactoring freedom is too small to really matter much in practice.

semantics problematic

Hm, but how is the semantics "problematic"? The extra spec complexity is really not large here (it's probably not much more than adding an evaluation semantics to ImportStatement and coordinating that with ModuleEvaluation), and regardless, spec complexity is often not at all in direct correlation with cognitive complexity for programmers. Consider teaching a newbie how the semantics works: it's a lot easier to say "everything runs top to bottom" than to say "most things run top to bottom but imports are actually hoisted and executed early."

[dherman]

@bmeck

Does this mean we must wait on the event loop to turn to finish an import sometimes (depending if the dependency is async/sync)?

Well yes, that's really what top-level await is for: it's a mechanism for standardizing modular, asynchronous, blocking startup logic. It's intended to block its dependents. Of course, require is an exception to that, because it's important not to violate JS's run-to-completion semantics, and it makes the interop and upgrade path for Node smooth and gradual. But the goal is to create a path to a future where there can be a standardized way to modularly execute asynchronous startup steps that don't lead to race conditions in applications' startup logic.

Of course, if you want to kick off some asynchronous logic at startup and you don't want to block the world, you just don't use top-level await; you can do all the same things you can do today.

But this is getting a bit off-topic -- I should probably do a gist to explain the story of top-level await as I see it so far.

[allenwb]

@dherman

If you believe they're stronger, it would help to explain which arguments you think are stronger than interop with Node and why.

I think all of the arguments in favor to the declarative approach are strong. It's the imperative arguments that I found weak.

You didn't actually identify any actual interop problems with node. You just postulated that there is a likelhood of such problems because of the difference in semantics. But you don't provide any actual examples of such. In particular, AFAIK, node doesn't have any way to declarative define circular import/export dependencies. If that is correct, then we aren't really talking about actual interop relating to existing code. Are we? Presumably, techniques used by node programmer to imperatively establish non-declarative dependencies between such modules would continue to work with the ES6 declarative model, as long as those techniques are operating at the property level (like in node) rather than the binding level.

Is your concern that node programmers will try to write new ES6 modules but won't fully understand them and run into unexpected initialization issues because they continue to follow Node patterns? If so, that's not an actual interop issue but rather a usability issue. With thinking about, but a new system does require new learning.

Finally, as you show in your "increased impressibility" example, it takes really careful thought to construct such examples where it actually makes a difference. I shudder at what it would take to meaning try to do this with more than two imported/exported symbols or more than two modules.

[dherman]

@bmeck

Just to add another point of clarification: the asynchronous API for executing a graph of module dependencies always returns a promise, and if there's any top-level await, it may not complete in a single turn, but, if there isn't any top-level await, everything evaluates to completion immediately, and the promise is immediately resolved. The Module.evaluate API (and/or some internal API provided by V8 that Node would use internally in the implementation of require for evaluating ES6 modules) is always async and produces a promise, but if there's no actual async top-level blocking in the module graph, it does all initialization synchronously.

[dherman]

@allenwb

OK, thanks, that's fair. It's true that I only have an intuition here and haven't really nailed the interop concerns. I mean, I think we both agree there's a learnability cost and a surprise for people used to the Node behavior. But my concern is more that code will break, and I should try to identify why.

So there's definitely an upgrade hazard. For example, you can implement the example I gave above in Node today:

// a.js
function A() { ... }
exports.A = A;
var B = require('./b').B;
A.B_INSTANCE = new B();

// b.js
function B() { ... }
exports.B = B;
var A = require('./a').A;
B.A_INSTANCE = new A();

This works because Node has the imperative semantics. As far as I can tell, the declarative semantics provides no way to implement this without a third module that updates the bindings. Something like:

// a.js
export default class A { ... }

// b.js
export default class B { ... }

// init.js
import A from "./a";
import B from "./b";
A.B_INSTANCE = new B();
B.A_INSTANCE = new A();

And then to ensure that init.js is always executed you would probably have to wrap them with "facade modules":

// a.js
import "./internal/a"; // ensure a is initialized
import "./internal/b"; // ensure b is initialized
import "./internal/init"; // ensure their statics are initialized
export { default } from "./internal/a"; // re-export a

// b.js
import "./internal/a"; // ensure a is initialized
import "./internal/b"; // ensure b is initialized
import "./internal/init"; // ensure their statics are initialized
export { default } from "./internal/b"; // re-export b

Now, it's actually easier to understand the control flow here, so you could argue people ought to do that anyway. And given how subtle the semantics of cyclic evaluation order is, maybe they'll want to. But it's a thing that works in Node today that would not work when upgrading.

I feel like there could be actual interop hazards too, but I have to think about it harder. Since we don't really expect cycles between CJS and ES6 to work anyway, maybe that diminishes the potential pitfalls. I'll have to think some more.

Presumably, techniques used by node programmer to imperatively establish non-declarative dependencies between such modules would continue to work with the ES6 declarative model, as long as those techniques are operating at the property level (like in node) rather than the binding level.

Can you explain this a little more? I had trouble unpacking it.

[bmeck]

@dherman So are you saying there is no way for circular deps to work in Declarative modules synchronously without using a 3rd module to provide the synchronization point?

Also, pretty neutral on this whole topic except I don't want it to be called an await if we re-enter code synchronously like the circular deps example.

I do think it is nicer for circular dependencies; but side of the parent module could affect children, which I think is a con. Both are pretty small to me though.