node_modules after production build is too large

[Kaarel]

Description / Steps to reproduce / Feature proposal

As a developer I want a minimal necessary (as in bytes and number of dependencies) package of my loopback service for deployment.

1. docker run -ti node:alpine sh
2. npm i -g @loopback/cli
3. lb4 poc # Selecting all defaults by pressing Enter
4. cd poc
5. npm run build
6. rm -rf node_modules # To ensure no dev cruft
7. npm install --only=production

Current Behavior

du -sh node_modules is over 70M

Expected Behavior

I would have expected du -sh node_modules to be a few megs...10 at most?

du -s node_modules/* | sort -rn | head reveals node_modules/typescript (dependency of @loopback/openapi-v3) takes almost half that space.
Plus couple other random pics:

• node_modules/loopback-datasource-juggler/test 1Mb
• node_modules/strong-globalize 4Mb
• node_modules/@loopback/* modules contain unnecessary:
  • api-docs
  • src

A release/production build should not include test files, src files, module documentation files, build time utilities (eg typescript), dependencies that I don't use in my API service (eg globalization).

Action items

• get rid of typescript dependencies Replace type definition being imported by repository-json-schema #1132
• refactor strong-globalize: has too many dependencies strongloop/strong-globalize#97
• update dependencies to use the new smaller version of strong-globalize
  • Drop support for Node.js 4.x + update dependencies loopback-datasource-juggler#1594
  • Drop support for Node.js 4.x + update dependencies loopback-connector#126
• remove apidocs from production build
• get rid of nyc output

[shimks]

The issue of @loopback/openapi-v3 having typescript as a dependency is not intended and is tracked in issue #1132

[Kaarel]

Had a bit of time in my hands today so looked further in

• node_modules\loopback-datasource-juggler 4.0M, of which
  • .nyc_output 1.6M
  • test 1.2M
  • docs 380K
• node_modules/lodash/ 3.3M
  • surely not all of the 3+ megs of helper functions are being utilised?
• node_modules/@loopback/repository 1.8M
  • api-docs 1.2M
• node_modules/yamljs/ 812K
  • while node_modules/yamljs/dist/yaml.min.js is just 32K

This may well be just an education / documentation issue as I'm still grappling with Node and JS best practices. But I don't understand why, if I pull in a package into my application for its functionality, do I also get it's test code, benchmark data, code coverage results, documentation, original source (in case runtime files are transpiled from source), examples, esX versions (if I'm only interested esY) etc.

In my mind it is package maintainers responsibility to provide appropriate dev and release packages. But maybe I'm approaching this from a wrong perspective and the best practice in Node is to use Webpack et al to create the final deployable. Then perhaps this could be more prominently explained in the documentation (of which by the way loopack has done a great job)? Using a concrete example from above in case of yamljs, for the release deployable, of everything in the node_modules/yamljs/ I'm probably only interested in yaml.min.js.

[virkt25]

Thanks for the investigation @Kaarel !! This is great.

I agree that it's not helpful to have test coverage data and apidocs bundled in a final release of the package. apidocs are available online at http://apidocs.loopback.io but I believe the reasons for doing so is because the site pulls in the docs from the different packages directly. Maybe we should consider a different approach.

As for loopback-datasource-juggler, I'm not sure if there's a reason for bundling test and .nyc_output

yamljs is a external module that we depend on and as such we can't really control the size of their package expect raise an issue against their repository but in comparison to the other issues you've outlined, I think 812K isn't that bad.

[raymondfeng]

@Kaarel I just published loopback-datasource-juggler@3.15.5 to remove .nyc_output.

For the record, the test directory of loopback-datasource-juggler contains some shared tests to be run by other connectors. That's why it is included in the published module.

[raymondfeng]

We publish apidocs along with the modules and use npm to manage versions for apidocs.loopback.io.

[raymondfeng]

#1286 gets rid of a heavy dependency of @loopback/repository-json-schema.

[raymondfeng]

FYI: as of @loopback/cli@0.9.2, the size of node_modules is down to 35MB.

[raymondfeng]

I have a PR to clean up strong-globalize: strongloop/strong-globalize#132

[bajtos]

I have a PR to clean up strong-globalize: strongloop/strong-globalize#132

Now that the strong-globalize clean-up was merged, I think we need to publish a new release and update loopback-next to use the new strong-globalize version. Is that correct?

[raymondfeng]

We have released strong-globalize@4.0.0 and it's backward compatible. I'll open an issue to track how to update LB3 modules to use the new version.

[bajtos]

strong-globalize@4.0.0 does not support Node.js 6.x which LB 3.x needs to support, I opened strongloop/strong-globalize#134 to fix that.

[dhmlau]

@raymondfeng @bajtos , with the strong-globalize being refactored, does it mean this issue is already resolved?