RFC for Prebuilding Addons

active/0000.prebuild-addon.md

@@ -0,0 +1,199 @@
+- Start Date: 2018-04-27
+- RFC PR:
+
+# Summary
+The goal of this RFC is to reduce application build time of ember
+applications by prebuilding addons.
+
+# Motivation
+For improving developer productivity, reducing build time is
+crucial. During cold and warm build (not rebuild) of an ember
+application, both internal (in-repo, possibly being developed) and
+external (in `node_modules`, generally not being developed) addons are
+built on each build. In our large-application example, Babel takes up
+the majority of the application build time, and transpiling addons
+takes a significant portion of that (several minutes). Prebuilding
+addons will help reduce the application build time.
+
+# Detailed design
+Building an addon involves fetching the different broccoli trees
+(`templates`, `app`, `vendor`, `addon`, `styles`, `public`, `test-support`,
+`addon-test-support`, `src`) and transpiling them.
+
+Note that:
+- External addons are never going to change unless we pull in a new version
+  of the addon or we directly modify the addon in `node_modules`.
+- Internal addons may change if we are working on them or their
+  dependent addons. Addons must have the `isDevelopingAddon`
+  flag set if they are being actively worked on.
+
+The idea of this PR is that during the initial cold build `ember
+build/serve` of an application or an addon, some addon
+trees' transpiled output will be stored in a separate location so that
+it can be reused for subsequent builds.
+
+# Key Considerations and Details
+- Prebuild is an opt-in feature
+- When you may not want to prebuild
+- When prebuild is not possible
+- Good default target browsers
+- A cache key is generated for a prebuilt tree and prebuild uses it to check
+  if a given prebuilt tree is (still) valid
+- Addons and apps can share prebuild responsibility
+- Prebuild meta data information for each addon is persisted
+- A prebuild summary log file is generated
+- Clearing the prebuild cache
+
+Each of the above points is discussed below in detail.
+
+## Opt-in Feature
+This is an opt-in feature. The environment variable PREBUILD must be set to store or
+use the prebuilt trees.
+
+## When You May Not Want To Prebuild
+You may not always want to prebuild every addon you can. These
+exclusions can be defined either in the `package.json` of the
+application or as an environment variable.
+The addons to be excluded can be defined as an array or a string or a pattern of addon names.
+- In `package.json`, the value is represented as shown below:
+```json
+"prebuild" : {
+  "excludeAddons" : "*",
+}
+```
+- The environment variable `EXCLUDEADDONS` can be used instead to contain the same value
+
+Having this ability provides flexibility in the following scenarios:
+
+- *When changing the addon directly in `node_modules`* - If you're modifying the addon code
+directly inside `node_modules`, for example while troubleshooting issues, and the addon was previously prebuilt,
+the prebuild will be used in preference to your changes during both build and rebuild. This
+causes a couple of problems:
+    - *During rebuild* - Rebuild will not be triggered because the file-system watcher will
+      be watching the prebuild directories, not the changed code in `node_modules`.
+    - *During build* - Changes in `node_modules` will not be reflected if the prebuilt addon
+      is already available. The computed cache key for the addon does not include timestamp
+      data (it does not check that the code on which the key is based has changed), so the
+      build will continue to return the prebuilt addon tree.
+
+    Both of these scenarios can be dealt with but the build-time gain from doing prebuild
+    will be lost handling them.
+- *Features are removed from browsers* - 
+  Features removed from browsers are a rare scenario. But in the future we can
+  consider `excluding` or `blacklisting` browsers as well, if `excluding` addons are
+  not enough.
+- *Errors in Babel or any dependent library* -
+  If an error is introduced in a library and it is fixed in a newer version,
+  the prebuild must be cleared and regenerated to pick up the later version.
+  If the prebuild resides in `npm` then it also needs to be published there again.
+
+## When Prebuild Is Not Possible
+Addons will not be prebuilt for the following scenarios:
+- Addons that have the `isDevelopingAddon` flag set
+- Addons that are symlinked. Addons are considered to be symlinked if the
+  addon's root is not contained in the project root or in project's node_modules.
+- Addons which have been referred to using a path (e.g., to a github repo) instead of a
+  version in `package.json` of its parent
+
+In addition, not all trees can be prebuilt. Only the addon trees that do not
+change between runs can be prebuilt. These include `templates`, `addon` and
+`addon-test-support`.
+
+However, even one or more of these trees must sometimes be excluded from prebuild.
+
+An addon may need to be excluded from prebuild if one or more `treeFor` hooks for the 
+addon are being extended by the addon. If a hook has been extended, *it is the
+responsibility of the addon developer* to decide if the tree is safe to prebuild.
+
+If not, currently the developer must opt out of prebuilding the addon.
+Later we may support opting out of prebuilding particular trees of an addon,
+not just the addon as a whole.
+
+## Good Default Target Browsers
+
+An ember application is generally targeted to be run on a particular
+set of browsers (e.g, the last 2 versions of evergreen browsers). The
+browser targets dictate the babel plugins that will be used to
+transpile the code. If not specified for the addon being prebuilt, we
+will use the set of browser targets defined in `ember-cli`.
+
+## Cache Key 
+
+For each addon prebuild, a cache key is generated and the prebuild
+directory path contains the cache key. The cache key for the addon
+includes an MD5 hash of the addon name, package, babel options,
+target browsers, etc. It is built as follows:
+```js
+  cacheKeyParts = [
+    addon.pkg && Object.keys(addon.pkg).sort(),
+    addon.name,
+    typeof addon.options.babel === 'function' ? addon.options.babel() : addon.options.babel,
+    addon.options['ember-cli-babel'],
+    targetBrowsers,
+  ];
+  return crypto.createHash('md5').update(stringify(cacheKeyParts), 'utf8').digest('hex');
+```
+
+Note: The cache key will be different for `last 2 Chrome versions` and `last 1 Chrome version`.
+Even though the latter is a subset of the former the current design does not support using
+the same prebuild. Whether this needs to be supported or not is a topic of discussion.
+
+## Sharing Prebuild Responsibility
+
+Typically, the responsibility of storing the addon prebuilt trees should be shared by both the addon and the app.
+- The addon should publish its prebuild everytime there is a change to the addon.
+- The app can also prebuild the addon and store it locally or publish it internally (e.g., to a company repository).
+
+Storing the addon prebuilt trees with the app is generally better because:
+- Most of the time addons do not specify target browsers
+- It's reasonably common that an addon and an app's target and babel options are different
+
+## Persistence
+An addon can specify the location of its prebuilt directory in its
+`package.json` shown below, 
+```json
+"prebuild" : {
+  "prebuild-base-path" : "</path>"
+}
+```
+By default, the prebuilt directory will be stored
+inside the addon whether the addon is prebuilt by the app or the addon
+itself.
+The prebuilt directory will contain one prebuilt addon tree for each
+target set. A target set is the attributes that make up the cache key.
+The key will be in the format
+`pre-built/<addon-name>/<md5ChecksumOfTargetgroup>/<tree-type>`.
+
+The location to store the prebuild addons can differ from team to
+team. Publishing the prebuilt addons either to `npm` or to a company
+repository will make the prebuilds more widely available and  help
+in reducing the cold build time across the team.
+
+## Metadata
+Each prebuilt addon for a particular target will have a metadata file
+containing addon name, babel options and target browser versions. This
+indicates the target configurations for which the addon has been
+prebuilt.
+
+## Prebuild Summary Log File
+At the end of the build a prebuild log file (json) will be generated.
+It will contain the addon name, prebuildPath, treeType, usingPrebuild
+(indicates whether prebuild is created or being used) for each of the
+prebuilt addons.
+
+## Clear Prebuild Cache
+The prebuilt addons can be cleared using the command `ember
+prebuild:clear`.  The command can take an addon name or glob pattern
+as a parameter to indicate the addons for which prebuild data can be
+cleared. By default the command clears the prebuild data for all addons in the
+project.
+
+# Changes to Addon BluePrint
+- `ember prebuild:clear` command should be added.
+- Prebuilt directories should be ignored from being checkedin.
+
+# How We Teach This
+This feature can be documented in ember-cli guides to teach.
+
+# Thanks
+Many thanks to Stefan Penner for helping me drive this forward.