Support NPM5

[joaomoreno]

More details on why this still isn't supported in #29576

[joaomoreno]

And as I suspected: npm i will always update package-lock.json with whatever it really installed. Given that fsevents won't install on Windows, running npm i on Windows (after locking all dependencies in macOS) updates the package-lock.json and removes it from there:

We can't really take this right now since we have optional dependencies which are platform specific. We'd have constant changes in these files all the time.

I see some activity going on in npm/npm#16839, which seems related, let's see if that fixes it.

[felixfbecker]

@joaomoreno npm 5.1 was just released and has fixes that might be relevant:

• This is a big PR that fixes a variety of issues when installing from package locks. If you were previously having issues with missing dependencies or unwanted removals, this might have fixed it:
  • It introduces a new package-lock.json field, called requires, which tracks which modules a given module requires.
  • It fixes Update Seti-icons to match Seti-UI #16839 which was caused by not having this information available, particularly when git dependencies were involved.
  • It fixes "Toggle Activity Bar Visibility" in Zen Mode Does Nothing #16866, allowing the package.json to trump the package-lock.json.
    npm ls now loads the shrinkwrap, which opens the door to showing a full tree of dependencies even when nothing is yet installed. (It doesn't do that yet though.)

https://github.com/npm/npm/releases/tag/v5.1.0

also related:

Regarding cross platformness, I'll make sure it's clearer, but yes, ALL optional dependencies, ALL dev dependencies should be in lock/shrinkwrap files even if you installed with --no-optional and --only=production.

npm/npm#16441 (comment)

[joaomoreno]

Just in time for a lazy friday afternoon! 🍺

[joaomoreno]

After rerunning ./scripts/npm.sh i a few times on macOS until it stabilised, I noticed that the optional dependencies still don't show up in the package-lock.json files.

Still, I pushed it to the branch. Then, jumped to Windows, and now it somehow fails to compile v8-profiler, no idea why:

joao@JOAOMORENO46E4 C:\Users\joao\Work\vscode
> .\scripts\npm.bat i

> code-oss-dev@1.15.0 preinstall C:\Users\joao\Work\vscode
> node build/npm/preinstall.js

npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but npm-shrinkwrap.json was generated for lockfileVersion@0. I'll try to do my best with it!
npm WARN deprecated node-uuid@1.4.8: Use uuid module instead
npm WARN deprecated to-iso-string@0.0.2: to-iso-string has been deprecated, use @segment/to-iso-string instead.
npm WARN deprecated jade@0.26.3: Jade has been renamed to pug, please install the latest version of pug instead of jade

> v8-profiler@5.6.5 preinstall C:\Users\joao\Work\vscode\node_modules\v8-profiler
> node -e 'process.exit(0)'


> typechecker@2.0.8 preinstall C:\Users\joao\Work\vscode\node_modules\typechecker
> node ./cyclic.js


> windows-mutex@0.2.0 install C:\Users\joao\Work\vscode\node_modules\windows-mutex
> node-gyp rebuild


joao@JOAOMORENO46E4 C:\Users\joao\Work\vscode\node_modules\windows-mutex
> if not defined npm_config_node_gyp (node "C:\Users\joao\AppData\Roaming\nvm\v7.10.0\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "" rebuild )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  main.cc
  mutex.cc
  win_delay_load_hook.cc
..\src\mutex.cc(70): warning C4996: 'v8::Function::NewInstance': was declared deprecated [C:\Users\joao\Work\vscode\node_modul
es\windows-mutex\build\CreateMutex.vcxproj]
  C:\Users\joao\.node-gyp\iojs-1.6.6\deps\v8\include\v8.h(3644): note: see declaration of 'v8::Function::NewInstance'
     Creating library C:\Users\joao\Work\vscode\node_modules\windows-mutex\build\Release\CreateMutex.lib and object C:\Users\j
  oao\Work\vscode\node_modules\windows-mutex\build\Release\CreateMutex.exp
  Generating code
  Finished generating code
  CreateMutex.vcxproj -> C:\Users\joao\Work\vscode\node_modules\windows-mutex\build\Release\\CreateMutex.node
  CreateMutex.vcxproj -> C:\Users\joao\Work\vscode\node_modules\windows-mutex\build\Release\CreateMutex.pdb (Full PDB)

> windows-foreground-love@0.1.0 install C:\Users\joao\Work\vscode\node_modules\windows-foreground-love
> node-gyp rebuild


joao@JOAOMORENO46E4 C:\Users\joao\Work\vscode\node_modules\windows-foreground-love
> if not defined npm_config_node_gyp (node "C:\Users\joao\AppData\Roaming\nvm\v7.10.0\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "" rebuild )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  foreground-love.cc
  win_delay_load_hook.cc
     Creating library C:\Users\joao\Work\vscode\node_modules\windows-foreground-love\build\Release\foreground-love.lib and obj
  ect C:\Users\joao\Work\vscode\node_modules\windows-foreground-love\build\Release\foreground-love.exp
  Generating code
  Finished generating code
  foreground-love.vcxproj -> C:\Users\joao\Work\vscode\node_modules\windows-foreground-love\build\Release\\foreground-love.nod
  e
  foreground-love.vcxproj -> C:\Users\joao\Work\vscode\node_modules\windows-foreground-love\build\Release\foreground-love.pdb
  (Full PDB)

> v8-profiler@5.6.5 install C:\Users\joao\Work\vscode\node_modules\v8-profiler
> node-pre-gyp install --fallback-to-build

node-pre-gyp ERR! Tried to download(404): https://node-inspector.s3.amazonaws.com/profiler/v5.6.5/electron-v1.6-win32-x64.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for v8-profiler@5.6.5 and electron@1.6.6 (electron-v1.6 ABI) (falling back to source compile with node-gyp)

joao@JOAOMORENO46E4 C:\Users\joao\Work\vscode\node_modules\v8-profiler
> if not defined npm_config_node_gyp (node "C:\Users\joao\AppData\Roaming\nvm\v7.10.0\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" clean )  else (node "" clean )

joao@JOAOMORENO46E4 C:\Users\joao\Work\vscode\node_modules\v8-profiler
> if not defined npm_config_node_gyp (node "C:\Users\joao\AppData\Roaming\nvm\v7.10.0\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" configure --fallback-to-build --module=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64\profiler.node --module_name=profiler --module_path=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64 )  else (node "" configure --fallback-to-build --module=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64\profiler.node --module_name=profiler --module_path=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64 )

joao@JOAOMORENO46E4 C:\Users\joao\Work\vscode\node_modules\v8-profiler
> if not defined npm_config_node_gyp (node "C:\Users\joao\AppData\Roaming\nvm\v7.10.0\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" build --fallback-to-build --module=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64\profiler.node --module_name=profiler --module_path=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64 )  else (node "" build --fallback-to-build --module=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64\profiler.node --module_name=profiler --module_path=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64 )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  profiler.cc
  cpu_profiler.cc
  cpu_profile.cc
  cpu_profile_node.cc
c:\users\joao\work\vscode\node_modules\nan\nan_new.h(208): warning C4244: 'argument': conversion from 'int64_t' to 'double', p
ossible loss of data (compiling source file ..\src\cpu_profile.cc) [C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\p
rofiler.vcxproj]
  ..\src\cpu_profile.cc(82): note: see reference to function template instantiation 'v8::Local<v8::Number> Nan::New<v8::Number
  ,int64_t>(A0)' being compiled
          with
          [
              A0=int64_t
          ]
  heap_profiler.cc
  heap_snapshot.cc
  heap_output_stream.cc
  heap_graph_node.cc
..\src\heap_snapshot.cc(46): error C2039: 'GetHiddenValue': is not a member of 'v8::Object' [C:\Users\joao\Work\vscode\node_mo
dules\v8-profiler\build\profiler.vcxproj]
  c:\users\joao\.node-gyp\iojs-1.6.6\deps\v8\include\v8.h(2881): note: see declaration of 'v8::Object'
..\src\heap_snapshot.cc(50): error C2039: 'SetHiddenValue': is not a member of 'v8::Object' [C:\Users\joao\Work\vscode\node_mo
dules\v8-profiler\build\profiler.vcxproj]
  c:\users\joao\.node-gyp\iojs-1.6.6\deps\v8\include\v8.h(2881): note: see declaration of 'v8::Object'
..\src\heap_snapshot.cc(101): warning C4244: 'argument': conversion from 'int64_t' to 'uint32_t', possible loss of data [C:\Us
ers\joao\Work\vscode\node_modules\v8-profiler\build\profiler.vcxproj]
..\src\heap_graph_node.cc(69): warning C4996: 'v8::Object::Has': was declared deprecated [C:\Users\joao\Work\vscode\node_modul
es\v8-profiler\build\profiler.vcxproj]
  c:\users\joao\.node-gyp\iojs-1.6.6\deps\v8\include\v8.h(2998): note: see declaration of 'v8::Object::Has'
  heap_graph_edge.cc
c:\users\joao\work\vscode\node_modules\nan\nan_new.h(208): warning C4244: 'argument': conversion from 'std::size_t' to 'double
', possible loss of data (compiling source file ..\src\heap_graph_node.cc) [C:\Users\joao\Work\vscode\node_modules\v8-profiler
\build\profiler.vcxproj]
  ..\src\heap_graph_node.cc(126): note: see reference to function template instantiation 'v8::Local<v8::Number> Nan::New<v8::N
  umber,std::size_t>(std::A0)' being compiled
          with
          [
              A0=std::size_t
          ]
  win_delay_load_hook.cc
gyp ERR! build error
gyp ERR! stack Error: `C:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Users\joao\AppData\Roaming\nvm\v7.10.0\node_modules\npm\node_modules\node-gyp\lib\build.js:258:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:194:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
gyp ERR! System Windows_NT 10.0.15063
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\joao\\AppData\\Roaming\\nvm\\v7.10.0\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "build" "--fallback-to-build" "--module=C:\\Users\\joao\\Work\\vscode\\node_modules\\v8-profiler\\build\\profiler\\v5.6.5\\electron-v1.6-win32-x64\\profiler.node" "--module_name=profiler" "--module_path=C:\\Users\\joao\\Work\\vscode\\node_modules\\v8-profiler\\build\\profiler\\v5.6.5\\electron-v1.6-win32-x64"
gyp ERR! cwd C:\Users\joao\Work\vscode\node_modules\v8-profiler
gyp ERR! node -v v7.10.0
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute 'node-gyp.cmd build --fallback-to-build --module=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64\profiler.node --module_name=profiler --module_path=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (C:\Users\joao\Work\vscode\node_modules\node-pre-gyp\lib\util\compile.js:83:29)
node-pre-gyp ERR! stack     at emitTwo (events.js:106:13)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:194:7)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:899:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)
node-pre-gyp ERR! System Windows_NT 10.0.15063
node-pre-gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\joao\\Work\\vscode\\node_modules\\node-pre-gyp\\bin\\node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd C:\Users\joao\Work\vscode\node_modules\v8-profiler
node-pre-gyp ERR! node -v v7.10.0
node-pre-gyp ERR! node-pre-gyp -v v0.6.36
node-pre-gyp ERR! not ok
Failed to execute 'node-gyp.cmd build --fallback-to-build --module=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64\profiler.node --module_name=profiler --module_path=C:\Users\joao\Work\vscode\node_modules\v8-profiler\build\profiler\v5.6.5\electron-v1.6-win32-x64' (1)
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@0.3.8 (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@0.3.8: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! v8-profiler@5.6.5 install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the v8-profiler@5.6.5 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\joao\.npm-electron\_logs\2017-07-07T14_33_10_633Z-debug.log

[felixfbecker]

Just to make sure the cause is not broken old shrinkwraps or broken migration code, does the same error happen if you delete the shrinkwraps prior to installing?

This feels like it might be worth to open an issue for at npm

[joaomoreno]

This was already without any shrinkwrap file around, simply having the package-lock.json files.

https://github.com/Microsoft/vscode/tree/joao/npm5

But now that you mention this, maybe I should try to migrate the shrinkwraps from scratch on 5.1

[joaomoreno]

Hm... just got the exact same error in macOS. Welcome to the Twilight Zone.

[ivanz]

I've managed to get npm install to work with NPM 5.3 with the changes in #30759 . That fixes the build error that @joaomoreno mentions, which I hit as well.

[ivanz]

Relevant issue from NPM for the optional dependency stuff: npm/npm#17722 . Can confirm that fsevents still gets removed on Windows from the lock file with npm 5.3