diff --git a/active/0000.prebuild-addon.md b/active/0000.prebuild-addon.md new file mode 100644 index 0000000..fd6c045 --- /dev/null +++ b/active/0000.prebuild-addon.md @@ -0,0 +1,199 @@ +- Start Date: 2018-04-27 +- RFC PR: + +# Summary +The goal of this RFC is to reduce application build time of ember +applications by prebuilding addons. + +# Motivation +For improving developer productivity, reducing build time is +crucial. During cold and warm build (not rebuild) of an ember +application, both internal (in-repo, possibly being developed) and +external (in `node_modules`, generally not being developed) addons are +built on each build. In our large-application example, Babel takes up +the majority of the application build time, and transpiling addons +takes a significant portion of that (several minutes). Prebuilding +addons will help reduce the application build time. + +# Detailed design +Building an addon involves fetching the different broccoli trees +(`templates`, `app`, `vendor`, `addon`, `styles`, `public`, `test-support`, +`addon-test-support`, `src`) and transpiling them. + +Note that: +- External addons are never going to change unless we pull in a new version + of the addon or we directly modify the addon in `node_modules`. +- Internal addons may change if we are working on them or their + dependent addons. Addons must have the `isDevelopingAddon` + flag set if they are being actively worked on. + +The idea of this PR is that during the initial cold build `ember +build/serve` of an application or an addon, some addon +trees' transpiled output will be stored in a separate location so that +it can be reused for subsequent builds. + +# Key Considerations and Details +- Prebuild is an opt-in feature +- When you may not want to prebuild +- When prebuild is not possible +- Good default target browsers +- A cache key is generated for a prebuilt tree and prebuild uses it to check + if a given prebuilt tree is (still) valid +- Addons and apps can share prebuild responsibility +- Prebuild meta data information for each addon is persisted +- A prebuild summary log file is generated +- Clearing the prebuild cache + +Each of the above points is discussed below in detail. + +## Opt-in Feature +This is an opt-in feature. The environment variable PREBUILD must be set to store or +use the prebuilt trees. + +## When You May Not Want To Prebuild +You may not always want to prebuild every addon you can. These +exclusions can be defined either in the `package.json` of the +application or as an environment variable. +The addons to be excluded can be defined as an array or a string or a pattern of addon names. +- In `package.json`, the value is represented as shown below: +```json +"prebuild" : { + "excludeAddons" : "*", +} +``` +- The environment variable `EXCLUDEADDONS` can be used instead to contain the same value + +Having this ability provides flexibility in the following scenarios: + +- *When changing the addon directly in `node_modules`* - If you're modifying the addon code +directly inside `node_modules`, for example while troubleshooting issues, and the addon was previously prebuilt, +the prebuild will be used in preference to your changes during both build and rebuild. This +causes a couple of problems: + - *During rebuild* - Rebuild will not be triggered because the file-system watcher will + be watching the prebuild directories, not the changed code in `node_modules`. + - *During build* - Changes in `node_modules` will not be reflected if the prebuilt addon + is already available. The computed cache key for the addon does not include timestamp + data (it does not check that the code on which the key is based has changed), so the + build will continue to return the prebuilt addon tree. + + Both of these scenarios can be dealt with but the build-time gain from doing prebuild + will be lost handling them. +- *Features are removed from browsers* - + Features removed from browsers are a rare scenario. But in the future we can + consider `excluding` or `blacklisting` browsers as well, if `excluding` addons are + not enough. +- *Errors in Babel or any dependent library* - + If an error is introduced in a library and it is fixed in a newer version, + the prebuild must be cleared and regenerated to pick up the later version. + If the prebuild resides in `npm` then it also needs to be published there again. + +## When Prebuild Is Not Possible +Addons will not be prebuilt for the following scenarios: +- Addons that have the `isDevelopingAddon` flag set +- Addons that are symlinked. Addons are considered to be symlinked if the + addon's root is not contained in the project root or in project's node_modules. +- Addons which have been referred to using a path (e.g., to a github repo) instead of a + version in `package.json` of its parent + +In addition, not all trees can be prebuilt. Only the addon trees that do not +change between runs can be prebuilt. These include `templates`, `addon` and +`addon-test-support`. + +However, even one or more of these trees must sometimes be excluded from prebuild. + +An addon may need to be excluded from prebuild if one or more `treeFor` hooks for the +addon are being extended by the addon. If a hook has been extended, *it is the +responsibility of the addon developer* to decide if the tree is safe to prebuild. + +If not, currently the developer must opt out of prebuilding the addon. +Later we may support opting out of prebuilding particular trees of an addon, +not just the addon as a whole. + +## Good Default Target Browsers + +An ember application is generally targeted to be run on a particular +set of browsers (e.g, the last 2 versions of evergreen browsers). The +browser targets dictate the babel plugins that will be used to +transpile the code. If not specified for the addon being prebuilt, we +will use the set of browser targets defined in `ember-cli`. + +## Cache Key + +For each addon prebuild, a cache key is generated and the prebuild +directory path contains the cache key. The cache key for the addon +includes an MD5 hash of the addon name, package, babel options, +target browsers, etc. It is built as follows: +```js + cacheKeyParts = [ + addon.pkg && Object.keys(addon.pkg).sort(), + addon.name, + typeof addon.options.babel === 'function' ? addon.options.babel() : addon.options.babel, + addon.options['ember-cli-babel'], + targetBrowsers, + ]; + return crypto.createHash('md5').update(stringify(cacheKeyParts), 'utf8').digest('hex'); +``` + +Note: The cache key will be different for `last 2 Chrome versions` and `last 1 Chrome version`. +Even though the latter is a subset of the former the current design does not support using +the same prebuild. Whether this needs to be supported or not is a topic of discussion. + +## Sharing Prebuild Responsibility + +Typically, the responsibility of storing the addon prebuilt trees should be shared by both the addon and the app. +- The addon should publish its prebuild everytime there is a change to the addon. +- The app can also prebuild the addon and store it locally or publish it internally (e.g., to a company repository). + +Storing the addon prebuilt trees with the app is generally better because: +- Most of the time addons do not specify target browsers +- It's reasonably common that an addon and an app's target and babel options are different + +## Persistence +An addon can specify the location of its prebuilt directory in its +`package.json` shown below, +```json +"prebuild" : { + "prebuild-base-path" : "" +} +``` +By default, the prebuilt directory will be stored +inside the addon whether the addon is prebuilt by the app or the addon +itself. +The prebuilt directory will contain one prebuilt addon tree for each +target set. A target set is the attributes that make up the cache key. +The key will be in the format +`pre-built///`. + +The location to store the prebuild addons can differ from team to +team. Publishing the prebuilt addons either to `npm` or to a company +repository will make the prebuilds more widely available and help +in reducing the cold build time across the team. + +## Metadata +Each prebuilt addon for a particular target will have a metadata file +containing addon name, babel options and target browser versions. This +indicates the target configurations for which the addon has been +prebuilt. + +## Prebuild Summary Log File +At the end of the build a prebuild log file (json) will be generated. +It will contain the addon name, prebuildPath, treeType, usingPrebuild +(indicates whether prebuild is created or being used) for each of the +prebuilt addons. + +## Clear Prebuild Cache +The prebuilt addons can be cleared using the command `ember +prebuild:clear`. The command can take an addon name or glob pattern +as a parameter to indicate the addons for which prebuild data can be +cleared. By default the command clears the prebuild data for all addons in the +project. + +# Changes to Addon BluePrint +- `ember prebuild:clear` command should be added. +- Prebuilt directories should be ignored from being checkedin. + +# How We Teach This +This feature can be documented in ember-cli guides to teach. + +# Thanks +Many thanks to Stefan Penner for helping me drive this forward. \ No newline at end of file