[BUG] `node_modules/.bin` directory and symlinks no longer being created after upgrading from v6 to v8

[busticated]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

After upgrading my project from v6.14.16 to v8.3.2, the node_modules/.bin directory and associated symlinks are no longer created when running npm i

Expected Behavior

node_modules/.bin directory and associated symlinks are created after running npm i, bare executable names are recognized when used within scripts - e.g. using a script like:

  "scripts": {
    "postinstall": "lerna bootstrap"
  },

...running npm run postinstall directly (or implicitly after npm i) works, the lerna name resolves to node_modules/.bin/lerna which inturn poinst to ../lerna/cli.js

Steps To Reproduce

1. using npm@v6.14.16 and node@v16.13.2 on macOS v12.0.1 (intel)

node --eval 'console.log(process.versions)'

{
  node: '16.13.2',
  v8: '9.4.146.24-node.14',
  uv: '1.43.0',
  zlib: '1.2.11',
  brotli: '1.0.9',
  ares: '1.18.1',
  modules: '93',
  nghttp2: '1.46.0',
  napi: '8',
  llhttp: '6.0.4',
  openssl: '1.1.1m',
  cldr: '39.0',
  icu: '69.1',
  tz: '2021a',
  unicode: '13.0'
}

2. run npm i npm@8 --global
3. run rm -rf node_modules
4. run npm i
5. run ls -la node_modules/.bin
6. observe the error: No such file or directory

Environment

• npm: v8.3.2
• Node.js: v16.13.2
• OS Name: macOS
• System Model Name: Monterey (v12.0.1 intel)
• npm config:

; "builtin" config from /usr/local/lib/node_modules/npm/npmrc

prefix = "/usr/local" 

; "user" config from /Users/me/.npmrc

//registry.npmjs.org/:_authToken = (protected) 

; node bin location = /usr/local/Cellar/node@16/16.13.2/bin/node
; cwd = /Users/me/code/particle/js-utils
; HOME = /Users/me
; Run `npm config ls -l` to show all defaults.

[busticated]

ah, whoops - meant to also note that upgrading from v6.14.16 to v7.24.2, and then to v8.3.2 fixes the issue. when i diff the npm-shrinkwrap.json file, i see a number of changes like:

<       "dev": true
---
>       "dev": true,
>       "bin": {
>         "parser": "bin/babel-parser.js"
>       },
>       "engines": {
>         "node": ">=6.0.0"
>       }

...which seems to indicate that npm@7 is properly handling the migration but npm@8 is not

[ruyadorno]

hi @busticated thanks for the thorough report! unfortunately I can't reproduce it, I set up a minimal example such as:

$ cat package.json
{
  "name": "test-lockfile",
  "version": "1.0.0",
  "dependencies": {
    "ruy": "^1.0.0"
  }
}

$ cat package-lock.json
{
  "name": "test-lockfile",
  "version": "1.0.0",
  "lockfileVersion": 1,
  "requires": true,
  "dependencies": {
    "ruy": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/ruy/-/ruy-1.0.0.tgz",
      "integrity": "sha512-VYppDTCM6INWUMKlWiKws4nVMuCNU5h+xjF6lj/0y90rLq017/m8aEpNy4zQSZFV2qz66U/hRZwwlSLJ5l5JMQ=="
    }
  }
}

When I try to npm install (tried with multiple v7 - v8 versions) that also includes the current development branch:

$ npm install
npm WARN old lockfile
npm WARN old lockfile The package-lock.json file was created with an old version of npm,
npm WARN old lockfile so supplemental metadata must be fetched from the registry.
npm WARN old lockfile
npm WARN old lockfile This is a one-time fix-up, please be patient...
npm WARN old lockfile

added 1 package, and audited 2 packages in 558ms

found 0 vulnerabilities

$ cat package-lock.json
{
  "name": "test-lockfile",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "name": "test-lockfile",
      "version": "1.0.0",
      "dependencies": {
        "ruy": "^1.0.0"
      }
    },
    "node_modules/ruy": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/ruy/-/ruy-1.0.0.tgz",
      "integrity": "sha512-VYppDTCM6INWUMKlWiKws4nVMuCNU5h+xjF6lj/0y90rLq017/m8aEpNy4zQSZFV2qz66U/hRZwwlSLJ5l5JMQ==",
      "bin": {
        "ruy": "bin/index.js"
      }
    }
  },
  "dependencies": {
    "ruy": {
      "version": "1.0.0",
      "resolved": "https://registry.npmjs.org/ruy/-/ruy-1.0.0.tgz",
      "integrity": "sha512-VYppDTCM6INWUMKlWiKws4nVMuCNU5h+xjF6lj/0y90rLq017/m8aEpNy4zQSZFV2qz66U/hRZwwlSLJ5l5JMQ=="
    }
  }
}

I can see that the "bin" entry was properly generated in the lockfile and I can also confirm the file is there as expected:

$ ls node_modules/.bin/ruy
node_modules/.bin/ruy

---

So maybe there's something else particular to your environment and/or that project that is causing this? It would be super helpful if you could try to reproduce your problem using a minimal example (such as this one here) so that we could possibly have something to work with and possibly git bisect to find when the problem started.

[busticated]

thanks for taking a look 🙏

maybe there's something else particular to your environment and/or that project that is causing this?

it's always a possibility but given that i'm able to "fix" things by going from v6.14.16 to v7.24.2, and then to v8.3.2 (as opposed to going from v6.14.16 directly to v8.3.2) and that this is not without precedent (#2147, #3791, #4035) seems to cast some doubt on it being exclusively due to something unique in my environment.

that said, anything i should look for specifically? i'm working out of a lerna-driven monorepo (docs) but this issue happened in the root package which doesn't behave differently than a single-package repo (lerna bootstrap is run via the postinstall hook) as far as i can tell. npm is installed with node@16 via brew. pretty standard setup i think 🤷‍♂️

if you could try to reproduce your problem using a minimal example

i'll see what i can do but i'm unfortunately a bit overbooked here. one thing right off the top - i'm using npm-shrinkwrap.json not package-lock.json maybe that's playing a role?

[ruyadorno]

it'd be super helpful if you can track some of the names of deps (possibly providing the full package-lock.json file if that's something you can do) that end up without the bin files (is it all of them?), from reading this comment it seems to me the problem might be related to a registry not returning a valid bin entry - I'd be interested to inspect the packument values returned by the registry for your packages that are missing .bin files

[TalMat]

Can confirm over here. Upgraded directly from node 14/npm 6 to node 16/npm 8. Running npm i -g no longer symlinks the packages in the global npm/node_modules directory and did not create the command, command.cmd, and command.ps1 files in the /npm. (note: this is for a custom script added to the package.json bin + preferGlobal: true, with a shebang in the .js file)

Uninstalled node. Reinstalled at v14. Upgrade npm to v7 and tested: still works. Upgraded to v8: now works properly. Upgraded node to v16 (npm already at 8.4.0) and it still works. Something about going through v7 sidestepped the problem.