Investigate integration/awareness of N-API with node pre-gyp

[mhdawson]

One of the wins for using N-API is not needing a binary for every version of Node. pre-gyp current assumes there is a different one so some investigation/change are needed to optimize for an N-API world.

[jschlight]

The module in question is mapbox/node-pre-gyp. There is a nice background piece, Cross-platform addons with node-pre-gyp.

node-pre-gyp essentially enhances nodejs/node-gyp with the ability to automatically upload binaries as part of the build process. These pre-built binaries can then be downloaded when the native module is deployed.

I am researching the current interoperability between node-pre-gyp and N-API with an eye towards understanding steps that can be taken to make the interaction seamless.

[jschlight]

I've updated my N-API-based module to use node-pre-gyp instead of node-gyp. I'm able to build and upload binaries to S3 and have these binaries detected and automatically downloaded at deployment.

There may be a straightforward solution to using node-pre-gyp with N-API that I'd like to discuss tomorrow during the call.

[addaleax]

Okay, so what I’m thinking based off the discussion we’re having right now:

• We should add process.versions.napi that exports NAPI_MODULE_VERSION (Michael)
• We should consider adding {node_abi_napi} field to node-pre-gyp that yields:
  • If N-API is available, node_abi_napi resolves to napi
  • If N-API is not available, node_abi_napi resolves to the same as node_abi (e.g. 49) to accommodate users of node-addon-api which use the built-in version of N-API
• We should consider adding a {napi_version} field to node-pre-gyp that yields NAPI_MODULE_VERSION
  • Along with it, a new package.json field for the node-pre-gyp options that could be called e.g. "napi-versions": which would contain an array of N-API versions that the author is building for, and that consequently are available for download

[jschlight]

Considering that the process.versions.napi value is just now being introduced, what is the best way to reliably determine, from JavaScript, if the currently running Node instance supports N-API?

node-pre-gyp maintains an abi_crosswalk.json file that maps Node versions to versions for the Node ABI and V8. They look like this:

  "8.9.0": {
    "node_abi": 57,
    "v8": "6.1"
  },
  "9.0.0": {
    "node_abi": 59,
    "v8": "6.2"
  }

We could piggyback on this file to include N-API version numbers. But I need to know which Node versions support which N-API version.

[mhdawson]

Right now there are only 2 N-API versions so it should be easy to figure that out, more complicated is that we were only planning to update once we exited experimental which may complicate things.

[mhdawson]

Once option may be to consider that only the first version of 8.X, 6.X etc where N-API is non-experimental is supported but I'm not sure how practical that might be.

[jschlight]

The value of {node_abi_napi} described above, can be determined directly from the value of {napi_version}. I'm thinking we have a couple of options for the task of determining {napi_version}:

OPTION 1

If the process.versions.napi value is present, then the N-API version is directly determined.

If the process.versions.napi value is not present, N-API is not supported.

This approach essentially says node-pre-gyp is supported only for those versions of Node with a non-experimental N-API implementation.

OPTION 2

If the process.versions.napi value is present, then the N-API version is directly determined as it is for option 1.

If the process.versions.napi value is not present, the N-API version is determined heuristically from the Node version. (Think hard-coded table.)

In this option, the heuristic needs to be determined only once and should be accurate indefinitely. As deployments of Node supporting process.versions.napi become more prevalent, the heuristic code will be relied upon less and less.

[jschlight]

Note: This proposal was last updated 2018-01-18. It's been superseded by the actual Implementation description found as a comment farther down.

I'm posting this proposed implementation for everyone's review. I'm anxious to receive any feedback you may have.

Proposed implementation

New path substitution variables

Add two new substitution variables for the binary properties module_path, remote_path, and package_name.

1. {napi_version} If N-API is supported, this value is the N-API version number supported by Node. If N-API is not supported, this value is an empty string.

2. {node_abi_napi} If the value returned for {napi_version} is non empty, this value is 'napi'. If the value returned for {napi_version} is empty, this value is the value returned for {node_abi}.

New package.json variable

Add an optional new package.json property under node-pre-gyp's binary section.

1. napi-versions An array of N-API versions that the author is building for, and that consequently are available for download.

Changes to publishing

Add code to support the napi-versions package.json property.

1. Inspect the binary property found on the package.json file. If the napi-versions property is present and the {napi_version} substitution string is present in any of module_path, remote_path, and package_name properties, inspect the values found in the napi-versions property. For each value found in the array, use node-gyp to compile a binary for that N-API version and upload the resulting binary using the remote_path, and package_name properties.

   If either the napi-versions property or the {napi_version} substitution string is absent, compile and upload as is currently implemented.

Changes to installing

Add code to support the napi-versions package.json property.

1. Inspect the binary property found on the package.json file.

   If the napi-versions property is present and the {napi_version} substitution string is present in any of module_path, remote_path, and package_name properties, inspect the values found in the napi-versions property. From the array, determine the largest version greater than or equal to the Node N-API version of the installing machine. Use this value for the {napi_version} substitution string and download the binary file using the current implementation. If no value is found in the array matching the criteria, the module does not support the Node N-API version of the installing machine.

   If the {napi_version} substitution string is present, and the napi-versions property is absent, for the purposes of the download, set the {napi_version} value to the Node N-API version of the installing machine, and attempt to download the binary with that N-API version. If the download fails, decrement N-API version by one and retry the download. The download fails when the N-API version reaches zero.

   If both the napi-versions property and the {napi_version} substitution string are absent, download the binary as is currently implemented.

Concerns

1. There can be an issue in the case where the napi-versions property is absent and the {napi_version} substitution string is present. For example, the module developer may be developing with a version of Node that supports N-API version 3. In the case where the napi-versions property is absent, the published binary is tagged for N-API version 3 whether it requires the features of N-API version 3 or not. A user of the module might be running a version of Node that supports N-API version 2. In this case, the user would not be able to download the binary because it appears to require N-API version 3.

   This issue could be resolved if we require the napi-versions property to be present if the {napi_version} substitution string is used. It would also greatly simplify the node-pre-gyp installation code by eliminating the need for sequential download requests at installation time.

[mhdawson]

The case that I think the array could handle (not with the current description though since it only uses the lowest) is:

• Module maintainer publishes 2 versions of their module. Version 1 (supports N-API A) and Version 2 (supports N-API B, where B >A). Version 2 is much faster because it uses some new functions only available in the later version of N-API
• Module is used with Node.js version that only supports N-API A. Version 1 of the module is downloaded and everything runs ok.
• Module is used with Node.js version that supports N-API B. Version 2 is downloaded and it runs much faster than if it had used Version 1.

If package.json just had the lowest N-API version supported, at install time how would we know Version 2 was available for use ?

[jschlight]

I think I'm getting greater clarity here.

If a module specifies "napi-versions":[3,4,5] it is declaring that the code will work under any of those three versions. How does it achieve this? Is the C/C++ code using conditional compilation?

If so, the node-pre-gyp publisher will need to fire off three builds and uploads, one targeting each of the three supported versions. Then the correct binary can be located and downloaded at installation time.

In the example shown here, the node-pre-gyp publishing system will need to be running a version of Node supporting N-API version 5 or later. I'm assuming that a system running a version of Node supporting N-API version 5 can compile C/C++ code targeting N-API version 4 and lower? Correct?

[mhdawson]

If a module specifies "napi-versions":[3,4,5] it is declaring that the code will work under any of those three versions. How does it achieve this? Is the C/C++ code using conditional compilation?

The module owner would have to provide 3 different binaries. Conditional compilation would be one way they could structure their code to support that.

In the example shown here, the node-pre-gyp publishing system will need to be running a version of Node supporting N-API version 5 or later. I'm assuming that a system running a version of Node supporting N-API version 5 can compile C/C++ code targeting N-API version 4 and lower? Correct?

It can compile code by avoiding functions that are not available in the earlier versions. In addition Kyle has an issue about adding #define that can be used to specify the level of functions you want to be visible (ie if you defined NAPI_API_VERION_4 ( I know I have the name wrong but you get the idea) then only the functions in version 4 and lower would be available when compiling.

[jschlight]

I've updated the proposed implementation shown above. The changes are:

1. When publishing, one binary is compiled and uploaded for each version found in the napi-versions array.
2. When installing, the best matching version from the napi-versions array is downloaded.
3. The concern about the napi-versions array was removed.

[mhdawson]

Overall looks reasonable to me.

In terms of

This issue could be resolved if we require the napi-versions property to be present if the {napi_version} substitution string is used. It would also greatly simplify the node-pre-gyp installation code by eliminating the need for sequential download requests at installation time.

are there any downsides of requiring the napi-versions property to be present?