From 5f5906c3ca5bfd24ed03321513ded5f6d248e2c9 Mon Sep 17 00:00:00 2001 From: Arthi Ravi Shankar Date: Thu, 3 May 2018 12:11:57 -0700 Subject: [PATCH 1/3] RFC for Prebuilding Addons --- active/0000.prebuild-addon.md | 111 ++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 active/0000.prebuild-addon.md diff --git a/active/0000.prebuild-addon.md b/active/0000.prebuild-addon.md new file mode 100644 index 0000000..d749913 --- /dev/null +++ b/active/0000.prebuild-addon.md @@ -0,0 +1,111 @@ +- Start Date: 2018-04-27 +- RFC PR: + +# Summary + +Prebuild addons prior to building the ember app to reduce the application build time + +# Motivation + +Being able to build only the code that has changed is awesome for developer productivity. Currently, while building an ember application, the addons that are not being developed as part of application are also built during first time. This includes the addons from both node_modules and in-repo. Building addons contribute to majority of the application build time which mainly comes from transpiling the addons. For a large application this might be in the order of few mins. Hence, prebuilding the addons prior to building the application will reduce the app build time significantly. + +# Detailed design + +Building an addon involves fetching 8 trees (templates, app, vendor, addon, styles, public, test-support, addon-test-support) and compiling them. Trees here map to physical directories. + +Not all trees can be prebuilt. Only the addon trees that do not change dynamically between runs can be prebuilt. Hence the trees that are safe to prebuild involve templates, addon, addon-test-support, vendor. + +Prebuilding is done in a separate addon `ember-cli-prebuild-addon` by introducing new commands `ember prebuild` (current addon) and `ember prebuild:all` (all addons in current app) which builds the addons and stores it in a prebuilt location. + +Under the hood the `ember-cli-prebuild-addon` does, + +-Gets the list of addons (current addon or all addons in current app) to build based on the command +-Gets the list of target groups to prebuild +-Find the location to store the prebuilt addons +-Calls the `prebuild` function which invokes the `treeFor` hook for the requested tree type to compile the trees that needs to be prebuilt +-Merges all trees for the addon and passes it to `broccoli-builder` +-Store the resulting build in the prebuilt location + +Then during application build, the `treeFor` hook in `ember-cli` will + +-Check if the `isDevelopingAddon` flag is not set +-Check if the requested tree can be prebuilt (any of the default trees that can be prebuilt) use the prebuilt addon if it exists for the same target as the app +-Finds the md5 checksum of the application target and forms a prebuiltdirectory name by appending checksum of target, addon version, tree name +-Checks if the prebuilt directory for the application target exists +-Returns the prebuilt directory if it exists + +## Good Defaults + +The application is generally targeted to be run on a set of browsers (e.g, last 2 versions of evergreen browsers). The browser targets dictate the babel plugins that will be used to transpile the code. And the addons will also be transpiled using the targets specified by the app. Hence it is necessary to prebuild the addons for a set of target group and the app can choose to use the prebuilt addon for the target group it needs. + +By default, addons will be prebuilt for the default targets specified by ember-cli. + +## Onus of prebuilding + +Typically, the responsibility of prebuilding the addons should be shared by both the addon and the app. + +-The addon can be prebuilt for default targets every time before publishing to npm registry. +-The application should also be able to prebuild all the addons it uses for the target it needs. If the addon has already been prebuilt for the target the application needs it can skip prebuilding it again. +-The application should also be able to blacklist a set of addons that should be opted out of prebuilding while prebuilding all the addons. + +## Commands to prebuild + +-ember prebuild command to prebuild the current addon +-ember prebuild:all command prebuilds all the ember addons in the current app. Note: If an addon tree should be prebuilt it should be opted out else the build will break. + +Optional parameters include + +-`prebuildTarget` - location to fetch the target files for prebuilding. By default, it will be `config/prebuild`. + + +## Trees that are safe to prebuild + +By default, the templates, addon, addon-test-support, vendor trees will be prebuilt. A tree can be safely prebuilt if the `treeFor` hooks for the said trees are not being extended by the addon. If the hook has been extended, it is the responsibility of the addon owner to decide if the tree is safe to prebuilt else opt out of prebuilding for any tree by specifying it in the index.js of the addon. + +As a next step, `ember-cli-prebuild-addon` should be able to figure out if the addon has extended `treeFor` hooks and opt out the respective trees from being prebuilt. + +## Location to store the prebuilt addons + +The addon can specify the location the prebuilt directory can be stored in its index.js. By default, the prebuilt directory will be stored inside the addon whether the addon is prebuilt by the app or the addon itself. + +The prebuilt directory will contain the prebuilt addon one for each target. Md5 checksum of each target file is used to generate the directory name. + +The checksum will be calculated by sorting the array of browsers to ensure stability. + +It will in the format `pre-built//addon`. +Ember-cli will check if the addon is prebuilt for the same target group by calculating the md5 checksum of app's targets. + +## Configuring Target Groups + +The addons should be prebuilt for multiple target group. A target group will contain a list of browsers. Prebuilding for multiple target group is necessary because if the application developer decides to use only chrome for building the app then shipping the build code for other browsers to the client is unnecessary and will also impact the build time significantly. Hence having the flexibility to be able to prebuild for different target group is highly important. + +Currently `ember prebuild` will look for target group in + +-`/prebuild` which can contain a list of target files. The package.json of the addon or app should be able to specify the list of target files that should be used to prebuild. +-If no target files are present in `config/prebuild`, then it will look for `/target.js` +-If the above ones are not successful, the default ember-cli target will be used. + +## Changes in Addon BluePrint + +For the addons to be prebuilt by default, + +-The addons should have `ember-cli-prebuild-addon` in its package.json +-`ember prebuild` command should be included in npm `prepack` hook so that the addon will prebuilt before publishing to npm registry. + +# Drawbacks + +If the application is using the incorrect prebuilt addon, the application build will break + +-The latest version of target browser changes continuously hence the prebuilt directory might not necessarily be built for the latest version of the browser. But since current last version will always be a subset of future last versions the prebuilt directory will always be valid. +-If the addon trees that change dynamically is prebuilt because it was not opted out of prebuilding, it will result in breaking the app build. +-Since the prebuilt addon is stored inside the addon itself, if the app is prebuilding all the addons which stores the prebuilt directory inside node_modules. So, when the node_modules directory is deleted the addons need to be prebuilt again. + +# Open Questions + +-Better way to specify the target groups and prebuild location +-Better way to identify which prebuilt directory belongs to a target group +-Ensure stability of checksum + +# Thanks + +Many thanks to Stefan Penner for helping me drive this forward. \ No newline at end of file From f6f78c8c57737b2d7700bbda2818acd49e8a6e73 Mon Sep 17 00:00:00 2001 From: Arthi Ravi Shankar Date: Thu, 3 May 2018 12:11:57 -0700 Subject: [PATCH 2/3] RFC for Prebuilding Addons --- active/0000.prebuild-addon.md | 131 ++++++++++++++++++---------------- 1 file changed, 71 insertions(+), 60 deletions(-) diff --git a/active/0000.prebuild-addon.md b/active/0000.prebuild-addon.md index d749913..fa173f2 100644 --- a/active/0000.prebuild-addon.md +++ b/active/0000.prebuild-addon.md @@ -3,109 +3,120 @@ # Summary -Prebuild addons prior to building the ember app to reduce the application build time +The goal of this RFC is to reduce the build time by avoiding to build the addons each time the application is built # Motivation -Being able to build only the code that has changed is awesome for developer productivity. Currently, while building an ember application, the addons that are not being developed as part of application are also built during first time. This includes the addons from both node_modules and in-repo. Building addons contribute to majority of the application build time which mainly comes from transpiling the addons. For a large application this might be in the order of few mins. Hence, prebuilding the addons prior to building the application will reduce the app build time significantly. +For improving developer productivity reducing build time is crucial. During initial build of an ember application the addons that are not being developed as part of application are also built. This includes the addons from both node_modules and in-repo. Transpiling addons contribute to majority of the application build time. For large applications this is in the order of mins. Prebuilding addons will help reducing the application build time. # Detailed design -Building an addon involves fetching 8 trees (templates, app, vendor, addon, styles, public, test-support, addon-test-support) and compiling them. Trees here map to physical directories. +Building an addon involves fetching the different trees (templates, app, vendor, addon, styles, public, test-support, addon-test-support, src) and transpiling them. -Not all trees can be prebuilt. Only the addon trees that do not change dynamically between runs can be prebuilt. Hence the trees that are safe to prebuild involve templates, addon, addon-test-support, vendor. +The addons can be in-repo or external addons. +The external addons (present in node_modules) are never going to change unless we pull in the new version of the addon or we are directly modifying the addon in node_modules. +The in-repo addons may change if we are working on it or its dependent addons. In general it may have `isDeveloping` flag set if its being actively worked on. -Prebuilding is done in a separate addon `ember-cli-prebuild-addon` by introducing new commands `ember prebuild` (current addon) and `ember prebuild:all` (all addons in current app) which builds the addons and stores it in a prebuilt location. +Here we can see that inspite of not working on most of the addons we are building them every time. +Hence the idea is that during the initial cold build `ember build/serve` of an application or an addon, the result of each addon tree's transpiled output will be stored in a separate location so that it can be used for subsequent builds. -Under the hood the `ember-cli-prebuild-addon` does, +Key Considerations and Details, --Gets the list of addons (current addon or all addons in current app) to build based on the command --Gets the list of target groups to prebuild --Find the location to store the prebuilt addons --Calls the `prebuild` function which invokes the `treeFor` hook for the requested tree type to compile the trees that needs to be prebuilt --Merges all trees for the addon and passes it to `broccoli-builder` --Store the resulting build in the prebuilt location +-This is an Opt-In feature. +-A cache key is generated for prebuild and uses it to check if a prebuilt tree is valid. The cache key is built using the following parameters addon name, package, babel options, target browsers. +-Exclusion of addon(s) from prebuild is supported. +-The addons that are being developed will not be prebuilt/using prebuild. It can be identified from `isDeveloping` flag of an addon. +-Not all trees can be prebuilt. Only the addon trees that do not change between runs can be prebuilt. Hence the trees that are safe to prebuild are templates, addon, addon-test-support. +-Prebuild meta data information for each addon is persisted. +-Prebuild summaries are stored in a log file. +-Cleaning the prebuilt trees are supported by `ember prebuild:clearAll` -Then during application build, the `treeFor` hook in `ember-cli` will +Each of the above point is discussed below in detail --Check if the `isDevelopingAddon` flag is not set --Check if the requested tree can be prebuilt (any of the default trees that can be prebuilt) use the prebuilt addon if it exists for the same target as the app --Finds the md5 checksum of the application target and forms a prebuiltdirectory name by appending checksum of target, addon version, tree name --Checks if the prebuilt directory for the application target exists --Returns the prebuilt directory if it exists +## Opt-in Feature + +This is an Opt in feature. The environment variable PREBUILD should be set to store or use the prebuilt trees. ## Good Defaults The application is generally targeted to be run on a set of browsers (e.g, last 2 versions of evergreen browsers). The browser targets dictate the babel plugins that will be used to transpile the code. And the addons will also be transpiled using the targets specified by the app. Hence it is necessary to prebuild the addons for a set of target group and the app can choose to use the prebuilt addon for the target group it needs. -By default, addons will be prebuilt for the default targets specified by ember-cli. - -## Onus of prebuilding +For each addon prebuild, a cache key is generated and the prebuild directory name represents the cache key. +The cache key for the addon involves addon name, package, babel options, target browsers etc. The prebuild tree will be used if the cache key matches. -Typically, the responsibility of prebuilding the addons should be shared by both the addon and the app. +```js + cacheKeyParts = [ + addon.pkg && Object.keys(addon.pkg).sort(), + addon.name, + typeof addon.options.babel === 'function' ? addon.options.babel() : addon.options.babel, + addon.options['ember-cli-babel'], + targetBrowsers, + ]; + return crypto.createHash('md5').update(stringify(cacheKeyParts), 'utf8').digest('hex'); +``` --The addon can be prebuilt for default targets every time before publishing to npm registry. --The application should also be able to prebuild all the addons it uses for the target it needs. If the addon has already been prebuilt for the target the application needs it can skip prebuilding it again. --The application should also be able to blacklist a set of addons that should be opted out of prebuilding while prebuilding all the addons. +The cache key will be different for `last 2 Chrome versions` and `last 1 Chrome version`. Even though the latter is a subset of the former the current design does not support using the same prebuild. Whether this needs to be supported or not is a topic of disucssion. -## Commands to prebuild +## Addon(s) Exclusion --ember prebuild command to prebuild the current addon --ember prebuild:all command prebuilds all the ember addons in the current app. Note: If an addon tree should be prebuilt it should be opted out else the build will break. +Any addon(s) can be excluded from using this feature. It can be defined either in the package.json of the application or as an environment variable. The addons to be excluded can be defined as an array or a string or a pattern of addon names. -Optional parameters include +Having this ability provides flexibility in the following scenarios. --`prebuildTarget` - location to fetch the target files for prebuilding. By default, it will be `config/prebuild`. +-Changing the addon directly in node_modules + During rebuild: Rebuild will not be triggered since the prebuild trees would be returned during the initial build hence the watchman will not be watching the actual addons but the prebuild addon. + During Build: The change will not be reflected if the prebuild addon is already available, since the cache key does not check the code change and it will continue to return the prebuilt addon. + Both of the scenarios can be handled but the build time gain will be lost handling these. Hence `EXCLUDEADDONS` can be used in this case. -## Trees that are safe to prebuild +-Features are removed from browsers, Errors in Babel or any dependent library + Features removed from browsers are a rare scenario. But in the future we can consider `excluding` or `blacklisting` browsers as well if `exluding` addons are not enough. + If an error is introduced in a library and it is fixed in the new version, the prebuild should be cleared and regenerated again. If the prebuild resides in npm then it needs to be published again. -By default, the templates, addon, addon-test-support, vendor trees will be prebuilt. A tree can be safely prebuilt if the `treeFor` hooks for the said trees are not being extended by the addon. If the hook has been extended, it is the responsibility of the addon owner to decide if the tree is safe to prebuilt else opt out of prebuilding for any tree by specifying it in the index.js of the addon. +## Trees for prebuild -As a next step, `ember-cli-prebuild-addon` should be able to figure out if the addon has extended `treeFor` hooks and opt out the respective trees from being prebuilt. +By default, the templates, addon, addon-test-support, vendor trees will be prebuilt. A tree can be safely prebuilt if the `treeFor` hooks for the said trees are not being extended by the addon. If the hook has been extended, it is the responsibility of the addon owner to decide if the tree is safe to prebuilt else opt out of prebuilding the addon. As a next step, we can opt out of prebuilding a particular tree of an addon. -## Location to store the prebuilt addons +## Onus of prebuilding -The addon can specify the location the prebuilt directory can be stored in its index.js. By default, the prebuilt directory will be stored inside the addon whether the addon is prebuilt by the app or the addon itself. +Typically, the responsibility of storing the addon build should be shared by both the addon and the app. -The prebuilt directory will contain the prebuilt addon one for each target. Md5 checksum of each target file is used to generate the directory name. +-The addon should publish its prebuild everytime there is a change to the addon. +-The app can also prebuild and store locally or publish internally to company repoository. -The checksum will be calculated by sorting the array of browsers to ensure stability. +Though the advantage of storing the addon build from an app is better because + Most of the addons do not specify target browsers + The chances of addon and app's target and babel options to be different are reasonably common. -It will in the format `pre-built//addon`. -Ember-cli will check if the addon is prebuilt for the same target group by calculating the md5 checksum of app's targets. +## Persistence -## Configuring Target Groups +The addon can specify the location of prebuilt directory in its package.json. By default, the prebuilt directory will be stored inside the addon whether the addon is prebuilt by the app or the addon itself. -The addons should be prebuilt for multiple target group. A target group will contain a list of browsers. Prebuilding for multiple target group is necessary because if the application developer decides to use only chrome for building the app then shipping the build code for other browsers to the client is unnecessary and will also impact the build time significantly. Hence having the flexibility to be able to prebuild for different target group is highly important. +The prebuilt directory will contain one prebuilt addon tree for each target. Md5 checksum of each target file is used to generate the directory name. It will be in the format `pre-built//`. -Currently `ember prebuild` will look for target group in +The location to store the prebuild addons can differ from team to team. Publishing the prebuilt addons either to npm or to a company repository will help in reducing the cold build. --`/prebuild` which can contain a list of target files. The package.json of the addon or app should be able to specify the list of target files that should be used to prebuild. --If no target files are present in `config/prebuild`, then it will look for `/target.js` --If the above ones are not successful, the default ember-cli target will be used. +## Metadata -## Changes in Addon BluePrint +Each prebuilt addon for a particular target will have a metadata file containing addon name, babel options and target browser versions. This indicates the target configurations for which the addon has been prebuilt. -For the addons to be prebuilt by default, +## Summary --The addons should have `ember-cli-prebuild-addon` in its package.json --`ember prebuild` command should be included in npm `prepack` hook so that the addon will prebuilt before publishing to npm registry. +At the end of the build a prebuild log file (json) will be generated. +It will contain the addon name, prebuildPath, treeType, using prebuild (indicates whether prebuild is created or being used) for each of the addon tree created or used prebuild. -# Drawbacks +## Clear Prebuild -If the application is using the incorrect prebuilt addon, the application build will break +The prebuilt addons can be cleared using the command `ember prebuild:clearAll`. +It can take addon glob pattern as parameter to indicate the addons for which prebuild can be cleared. By default clears prebuild for all addons in the project. --The latest version of target browser changes continuously hence the prebuilt directory might not necessarily be built for the latest version of the browser. But since current last version will always be a subset of future last versions the prebuilt directory will always be valid. --If the addon trees that change dynamically is prebuilt because it was not opted out of prebuilding, it will result in breaking the app build. --Since the prebuilt addon is stored inside the addon itself, if the app is prebuilding all the addons which stores the prebuilt directory inside node_modules. So, when the node_modules directory is deleted the addons need to be prebuilt again. +## Changes to Addon BluePrint -# Open Questions +-`ember prebuild:clearAll` command should be added. +-Prebuilt directories should be ignored from being checkedin. --Better way to specify the target groups and prebuild location --Better way to identify which prebuilt directory belongs to a target group --Ensure stability of checksum +# How We Teach This -# Thanks +This feature can be documented in ember-cli guides to teach. -Many thanks to Stefan Penner for helping me drive this forward. \ No newline at end of file +# Thanks From 11e15676214151ad50d938508017982259c4fc69 Mon Sep 17 00:00:00 2001 From: Arthi Ravi Shankar Date: Mon, 16 Jul 2018 14:29:57 -0700 Subject: [PATCH 3/3] Updated based on feedback for prebuild addons --- active/0000.prebuild-addon.md | 247 ++++++++++++++++++++++------------ 1 file changed, 162 insertions(+), 85 deletions(-) diff --git a/active/0000.prebuild-addon.md b/active/0000.prebuild-addon.md index fa173f2..fd6c045 100644 --- a/active/0000.prebuild-addon.md +++ b/active/0000.prebuild-addon.md @@ -2,48 +2,127 @@ - RFC PR: # Summary - -The goal of this RFC is to reduce the build time by avoiding to build the addons each time the application is built +The goal of this RFC is to reduce application build time of ember +applications by prebuilding addons. # Motivation - -For improving developer productivity reducing build time is crucial. During initial build of an ember application the addons that are not being developed as part of application are also built. This includes the addons from both node_modules and in-repo. Transpiling addons contribute to majority of the application build time. For large applications this is in the order of mins. Prebuilding addons will help reducing the application build time. +For improving developer productivity, reducing build time is +crucial. During cold and warm build (not rebuild) of an ember +application, both internal (in-repo, possibly being developed) and +external (in `node_modules`, generally not being developed) addons are +built on each build. In our large-application example, Babel takes up +the majority of the application build time, and transpiling addons +takes a significant portion of that (several minutes). Prebuilding +addons will help reduce the application build time. # Detailed design - -Building an addon involves fetching the different trees (templates, app, vendor, addon, styles, public, test-support, addon-test-support, src) and transpiling them. - -The addons can be in-repo or external addons. -The external addons (present in node_modules) are never going to change unless we pull in the new version of the addon or we are directly modifying the addon in node_modules. -The in-repo addons may change if we are working on it or its dependent addons. In general it may have `isDeveloping` flag set if its being actively worked on. - -Here we can see that inspite of not working on most of the addons we are building them every time. -Hence the idea is that during the initial cold build `ember build/serve` of an application or an addon, the result of each addon tree's transpiled output will be stored in a separate location so that it can be used for subsequent builds. - -Key Considerations and Details, - --This is an Opt-In feature. --A cache key is generated for prebuild and uses it to check if a prebuilt tree is valid. The cache key is built using the following parameters addon name, package, babel options, target browsers. --Exclusion of addon(s) from prebuild is supported. --The addons that are being developed will not be prebuilt/using prebuild. It can be identified from `isDeveloping` flag of an addon. --Not all trees can be prebuilt. Only the addon trees that do not change between runs can be prebuilt. Hence the trees that are safe to prebuild are templates, addon, addon-test-support. --Prebuild meta data information for each addon is persisted. --Prebuild summaries are stored in a log file. --Cleaning the prebuilt trees are supported by `ember prebuild:clearAll` - -Each of the above point is discussed below in detail +Building an addon involves fetching the different broccoli trees +(`templates`, `app`, `vendor`, `addon`, `styles`, `public`, `test-support`, +`addon-test-support`, `src`) and transpiling them. + +Note that: +- External addons are never going to change unless we pull in a new version + of the addon or we directly modify the addon in `node_modules`. +- Internal addons may change if we are working on them or their + dependent addons. Addons must have the `isDevelopingAddon` + flag set if they are being actively worked on. + +The idea of this PR is that during the initial cold build `ember +build/serve` of an application or an addon, some addon +trees' transpiled output will be stored in a separate location so that +it can be reused for subsequent builds. + +# Key Considerations and Details +- Prebuild is an opt-in feature +- When you may not want to prebuild +- When prebuild is not possible +- Good default target browsers +- A cache key is generated for a prebuilt tree and prebuild uses it to check + if a given prebuilt tree is (still) valid +- Addons and apps can share prebuild responsibility +- Prebuild meta data information for each addon is persisted +- A prebuild summary log file is generated +- Clearing the prebuild cache + +Each of the above points is discussed below in detail. ## Opt-in Feature - -This is an Opt in feature. The environment variable PREBUILD should be set to store or use the prebuilt trees. - -## Good Defaults - -The application is generally targeted to be run on a set of browsers (e.g, last 2 versions of evergreen browsers). The browser targets dictate the babel plugins that will be used to transpile the code. And the addons will also be transpiled using the targets specified by the app. Hence it is necessary to prebuild the addons for a set of target group and the app can choose to use the prebuilt addon for the target group it needs. - -For each addon prebuild, a cache key is generated and the prebuild directory name represents the cache key. -The cache key for the addon involves addon name, package, babel options, target browsers etc. The prebuild tree will be used if the cache key matches. - +This is an opt-in feature. The environment variable PREBUILD must be set to store or +use the prebuilt trees. + +## When You May Not Want To Prebuild +You may not always want to prebuild every addon you can. These +exclusions can be defined either in the `package.json` of the +application or as an environment variable. +The addons to be excluded can be defined as an array or a string or a pattern of addon names. +- In `package.json`, the value is represented as shown below: +```json +"prebuild" : { + "excludeAddons" : "*", +} +``` +- The environment variable `EXCLUDEADDONS` can be used instead to contain the same value + +Having this ability provides flexibility in the following scenarios: + +- *When changing the addon directly in `node_modules`* - If you're modifying the addon code +directly inside `node_modules`, for example while troubleshooting issues, and the addon was previously prebuilt, +the prebuild will be used in preference to your changes during both build and rebuild. This +causes a couple of problems: + - *During rebuild* - Rebuild will not be triggered because the file-system watcher will + be watching the prebuild directories, not the changed code in `node_modules`. + - *During build* - Changes in `node_modules` will not be reflected if the prebuilt addon + is already available. The computed cache key for the addon does not include timestamp + data (it does not check that the code on which the key is based has changed), so the + build will continue to return the prebuilt addon tree. + + Both of these scenarios can be dealt with but the build-time gain from doing prebuild + will be lost handling them. +- *Features are removed from browsers* - + Features removed from browsers are a rare scenario. But in the future we can + consider `excluding` or `blacklisting` browsers as well, if `excluding` addons are + not enough. +- *Errors in Babel or any dependent library* - + If an error is introduced in a library and it is fixed in a newer version, + the prebuild must be cleared and regenerated to pick up the later version. + If the prebuild resides in `npm` then it also needs to be published there again. + +## When Prebuild Is Not Possible +Addons will not be prebuilt for the following scenarios: +- Addons that have the `isDevelopingAddon` flag set +- Addons that are symlinked. Addons are considered to be symlinked if the + addon's root is not contained in the project root or in project's node_modules. +- Addons which have been referred to using a path (e.g., to a github repo) instead of a + version in `package.json` of its parent + +In addition, not all trees can be prebuilt. Only the addon trees that do not +change between runs can be prebuilt. These include `templates`, `addon` and +`addon-test-support`. + +However, even one or more of these trees must sometimes be excluded from prebuild. + +An addon may need to be excluded from prebuild if one or more `treeFor` hooks for the +addon are being extended by the addon. If a hook has been extended, *it is the +responsibility of the addon developer* to decide if the tree is safe to prebuild. + +If not, currently the developer must opt out of prebuilding the addon. +Later we may support opting out of prebuilding particular trees of an addon, +not just the addon as a whole. + +## Good Default Target Browsers + +An ember application is generally targeted to be run on a particular +set of browsers (e.g, the last 2 versions of evergreen browsers). The +browser targets dictate the babel plugins that will be used to +transpile the code. If not specified for the addon being prebuilt, we +will use the set of browser targets defined in `ember-cli`. + +## Cache Key + +For each addon prebuild, a cache key is generated and the prebuild +directory path contains the cache key. The cache key for the addon +includes an MD5 hash of the addon name, package, babel options, +target browsers, etc. It is built as follows: ```js cacheKeyParts = [ addon.pkg && Object.keys(addon.pkg).sort(), @@ -55,68 +134,66 @@ The cache key for the addon involves addon name, package, babel options, target return crypto.createHash('md5').update(stringify(cacheKeyParts), 'utf8').digest('hex'); ``` -The cache key will be different for `last 2 Chrome versions` and `last 1 Chrome version`. Even though the latter is a subset of the former the current design does not support using the same prebuild. Whether this needs to be supported or not is a topic of disucssion. - -## Addon(s) Exclusion - -Any addon(s) can be excluded from using this feature. It can be defined either in the package.json of the application or as an environment variable. The addons to be excluded can be defined as an array or a string or a pattern of addon names. +Note: The cache key will be different for `last 2 Chrome versions` and `last 1 Chrome version`. +Even though the latter is a subset of the former the current design does not support using +the same prebuild. Whether this needs to be supported or not is a topic of discussion. -Having this ability provides flexibility in the following scenarios. +## Sharing Prebuild Responsibility --Changing the addon directly in node_modules - During rebuild: Rebuild will not be triggered since the prebuild trees would be returned during the initial build hence the watchman will not be watching the actual addons but the prebuild addon. - During Build: The change will not be reflected if the prebuild addon is already available, since the cache key does not check the code change and it will continue to return the prebuilt addon. +Typically, the responsibility of storing the addon prebuilt trees should be shared by both the addon and the app. +- The addon should publish its prebuild everytime there is a change to the addon. +- The app can also prebuild the addon and store it locally or publish it internally (e.g., to a company repository). - Both of the scenarios can be handled but the build time gain will be lost handling these. Hence `EXCLUDEADDONS` can be used in this case. - --Features are removed from browsers, Errors in Babel or any dependent library - Features removed from browsers are a rare scenario. But in the future we can consider `excluding` or `blacklisting` browsers as well if `exluding` addons are not enough. - If an error is introduced in a library and it is fixed in the new version, the prebuild should be cleared and regenerated again. If the prebuild resides in npm then it needs to be published again. - -## Trees for prebuild - -By default, the templates, addon, addon-test-support, vendor trees will be prebuilt. A tree can be safely prebuilt if the `treeFor` hooks for the said trees are not being extended by the addon. If the hook has been extended, it is the responsibility of the addon owner to decide if the tree is safe to prebuilt else opt out of prebuilding the addon. As a next step, we can opt out of prebuilding a particular tree of an addon. - -## Onus of prebuilding - -Typically, the responsibility of storing the addon build should be shared by both the addon and the app. - --The addon should publish its prebuild everytime there is a change to the addon. --The app can also prebuild and store locally or publish internally to company repoository. - -Though the advantage of storing the addon build from an app is better because - Most of the addons do not specify target browsers - The chances of addon and app's target and babel options to be different are reasonably common. +Storing the addon prebuilt trees with the app is generally better because: +- Most of the time addons do not specify target browsers +- It's reasonably common that an addon and an app's target and babel options are different ## Persistence - -The addon can specify the location of prebuilt directory in its package.json. By default, the prebuilt directory will be stored inside the addon whether the addon is prebuilt by the app or the addon itself. - -The prebuilt directory will contain one prebuilt addon tree for each target. Md5 checksum of each target file is used to generate the directory name. It will be in the format `pre-built//`. - -The location to store the prebuild addons can differ from team to team. Publishing the prebuilt addons either to npm or to a company repository will help in reducing the cold build. +An addon can specify the location of its prebuilt directory in its +`package.json` shown below, +```json +"prebuild" : { + "prebuild-base-path" : "" +} +``` +By default, the prebuilt directory will be stored +inside the addon whether the addon is prebuilt by the app or the addon +itself. +The prebuilt directory will contain one prebuilt addon tree for each +target set. A target set is the attributes that make up the cache key. +The key will be in the format +`pre-built///`. + +The location to store the prebuild addons can differ from team to +team. Publishing the prebuilt addons either to `npm` or to a company +repository will make the prebuilds more widely available and help +in reducing the cold build time across the team. ## Metadata +Each prebuilt addon for a particular target will have a metadata file +containing addon name, babel options and target browser versions. This +indicates the target configurations for which the addon has been +prebuilt. -Each prebuilt addon for a particular target will have a metadata file containing addon name, babel options and target browser versions. This indicates the target configurations for which the addon has been prebuilt. - -## Summary - +## Prebuild Summary Log File At the end of the build a prebuild log file (json) will be generated. -It will contain the addon name, prebuildPath, treeType, using prebuild (indicates whether prebuild is created or being used) for each of the addon tree created or used prebuild. - -## Clear Prebuild +It will contain the addon name, prebuildPath, treeType, usingPrebuild +(indicates whether prebuild is created or being used) for each of the +prebuilt addons. -The prebuilt addons can be cleared using the command `ember prebuild:clearAll`. -It can take addon glob pattern as parameter to indicate the addons for which prebuild can be cleared. By default clears prebuild for all addons in the project. +## Clear Prebuild Cache +The prebuilt addons can be cleared using the command `ember +prebuild:clear`. The command can take an addon name or glob pattern +as a parameter to indicate the addons for which prebuild data can be +cleared. By default the command clears the prebuild data for all addons in the +project. -## Changes to Addon BluePrint - --`ember prebuild:clearAll` command should be added. --Prebuilt directories should be ignored from being checkedin. +# Changes to Addon BluePrint +- `ember prebuild:clear` command should be added. +- Prebuilt directories should be ignored from being checkedin. # How We Teach This - This feature can be documented in ember-cli guides to teach. # Thanks +Many thanks to Stefan Penner for helping me drive this forward. \ No newline at end of file