Proposal: Add support for full ES-Node interop

[ctjlewis]

• Rollup Plugin Name: node-resolve, commonjs, json

• Rollup Plugin Version: latest @ all

Adding support for full ES-Node interop

An important functionality for Rollup I believe is the ability to bundle ES modules which depend on Node packages, as this is a highly common development scenario. Interop with Node typically requires the inclusion of three plugins (above) which I will call the Node compatibility suite, and it presently has 86% coverage over the top ~100 NPM packages and the modules they depend on. Node interop is really the only reason I have ever needed to reach for Webpack anymore, and it would be nice to get us to the finish line on this.

Unfortunately, because a failure at any point in a dependency tree is disastrous to the developer experience, this risk compounds rapidly, and even 99% coverage is insufficient - however, I think it would be relatively easy to get 100% coverage over the top 1000 NPM packages within 30 days, as most issues seem to be caused by a few isolated issues.

Link: Node-Rollup Test Suite [PRs extremely welcome]

There are two types of errors that can occur with Node-Rollup interop:

1. There is a build-time failure, i.e. the output bundle cannot be generated.
2. There is a run-time failure, i.e. the output bundle does not behave in the same way as the source¹.

Some build-time headaches this team is already familiar with include the notable fsevents issue, which I would honestly place as a principal issue, if only because it actually prevents Rollup from bundling its own API via import { rollup } from rollup, which is incredibly depressing. As I understand it, chokidar can only bundle because a chokidar-specific wrapper was added, but this is insufficient for the obvious reason that it does not actually solve the problem - a better option would be adding the ability to ignore optionalDependencies, or even making them opt-in and ignore by default (what I would recommend), which is a future-proof and more general solution that will cover all packages.

---

¹ In the test suite, this is lazily evaluated as "if both fail, pass; if both exit 0, pass" when comparing source and bundle exit codes. Nonzero exits are generalized as "equivalent" for the purposes of these tests. This check can get more aggressive once we have 100% coverage, i.e. making sure console.log is 1:1 and so on.

Expected Behavior / Situation

It should be virtually impossible to find an NPM package which will not bundle and execute as expected.

Actual Behavior / Situation

Of the test sample (n=1303), 17 (1.3%) produce build-time errors, and 176 (13.5%) fail the runtime test, bringing total coverage to ~85.3%. Honestly, I would say this is pretty good coverage going into this issue.

Modification Proposal

Any adjustments could likely be limited to just the node-resolve plugin, and adding the node-resolve plugin to your build would just imply commonjs and json could be enabled with an option flag like nodeResolve({ legacyMode: true }).

Fix build-time failures

1. Add opt-in/opt-out ignore of optionalDependencies to handle fsevents-like scenarios. Preferred solution is to ignore optionalDependencies by default and add a flag for including them.

2. ✅ Fix hashbang directory issue Bundle fails for form require('../#/folder') due to hashbang-only directory #528 (in-progress at fix(babel): strip hash and query param in extension filter #533), which breaks for es5-ext package (and packages in the top 100 that depend on it, specifically d gulp es6-iterator es6-symbol es6-weak-map gulp gulp-cli last-run semver-greatest-satisfied-range sver-compat, see config/exclude.txt). [Update 11/13/20: This is fixed (presumably) with fix(babel): strip hash and query param in extension filter #533.]

3. Fix other build-time errors produced by the packages in config/exclude.txt. There's a "return outside of function" in there and a few other issues, but I have not bothered logging the build-time errors in the results/ dir yet as the runtime errors are much harder to test and outnumber them 10:1.

Packages which cannot build are added to config/exclude.txt so that the tests will build, any package in the test sample which refuses to build is listed in there.

Fix run-time failures

The following two issues cause 50% of the run-time errors:

1. util.inherits transpilations will fail in certain cases and get called with a null second argument (admittedly need help debugging this one), causing the error TypeError [ERR_INVALID_ARG_TYPE]: The "superCtor" argument must be of type function. Received undefined to be one of the two most common runtime errors, occurring for 64 packages that were tested for runtime errors (36% of build-time failures). See bundles/argparse.js:983: util.inherits(ArgumentGroup, action_container). This error seems most frequently produced by packages that rely on readable-stream.

2. __filename and __dirname shims are needed as CJS Node modules make use of these. I considered the node-polyfills plugin, but it completely conflicts with node-resolve and makes the build process unworkable. Errors due to __filename and __dirname being rolled into ES output cause 25 packages to fail (14% of build-time failures). This support will need to be added in a more native manner, see [commonjs] Output contains references to __filename #523.

---

I am happy to help as much as I can with this process, but I am a little out of my depth here and I would really appreciate support from the Rollup team in reaching 100% Node coverage.

[ctjlewis]

@Andarist @lukastaegert Thoughts?

[shellscape]

and adding the node-resolve plugin to your build would just imply commonjs and json.

I've got several builds where this would fail hard. Can't go into details, but I'm certainly not in favor of adding magic here. Config should be explicit imho.

Beyond that, we'd welcome any help from contributors on proposing these changes. Much of this seems reasonable though I think we're all fairly short on time at the moment.