Requiring a dynamically added dependency fails when the package has no `index.js` file

[zkochan]

Version

v7.1.0 and newer (last working version was v6.17.1)

Platform

Linux

Subsystem

No response

What steps will reproduce the bug?

Require a dependency that doesn't exist. Catch the error. Dynamically add the missing package to the node_modules. It should not contain an index.js file but should use a custom entry point instead. Require the now not missing package again.

Expected result: the modules is loaded.
Actual result: module not found error is thrown.

I have create a repro repo: https://github.com/zkochan/require-issue

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior?

No response

What do you see instead?

On the second require the module is loaded

Additional information

cc @GiladShoham @davidfirst

[DavidWeiss2]

I found a working workaround:
github fork

[zkochan]

@DavidWeiss2 your workaround doesn't seem to work on my PC

[DavidWeiss2]

Can you please increase the timeout time? In my opinion the bug root cause is an i/o opertion of the write. And it worth a shot

[zkochan]

Even an increase to 10 seconds doesn't help.

[DavidWeiss2]

@zkochan I uploaded another workaround at the same fork.
github fork

[DavidWeiss2]

Just published an npm package that solves this issue:
requirsafe

[zkochan]

This workaround will work only on direct dependencies, not on subdependencies. I guess we could hook into require to fix it for subdeps.

[zkochan]

Alright, this also works:

try {
  console.log(require("pkg"));
} catch (err) {
  console.error("failed require from package json");
  const data = require('pkg/package.json')
  console.log("this works", require('pkg/' + data.main));
}

We can use it to patch require. @DavidWeiss2 thanks for the workaround!

But of course it still needs to be fixed in Node.js.

[aduh95]

Why do you use node:child_process instead of node:fs for file system operations? If you switch to using node:fs, it looks like everything works as expected:

mkdir repro && cd repro
echo 'try { console.log(require("pkg")) } catch { console.log("first try failed") }' > entry.js
echo 'const fs = require("node:fs");' >> entry.js
echo 'fs.mkdirSync("node_modules/pkg/lib", { recursive:true });' >> entry.js 
echo 'fs.writeFileSync("node_modules/pkg/lib/main.js", "module.exports = `lib/main was loaded`");' >> entry.js
echo 'fs.writeFileSync("node_modules/pkg/package.json", JSON.stringify({main:"lib/main.js"}));' >> entry.js
echo 'try { console.log(require("pkg")) } catch { console.log("second try failed") }' >> entry.js
node entry.js
cd .. && rm -r repro

If I paste the above in my terminal, it outputs:

first try failed
lib/main was loaded

I'm going to close this as it doesn't seem to be a bug in Node.js, don't hesitate to re-open if you think I missed something or continue the discussion if you have more questions.

[zkochan]

it doesn't matter whether I use child process or use node:fs. It doesn't work. I have updated the repo to use fs. It fails.

don't hesitate to re-open if you think I missed something or continue the discussion if you have more questions.

I can't reopen it.

Your code also doesn't work if I move require two line below:

const fs = require("node:fs");
fs.mkdirSync("node_modules/pkg/lib", { recursive:true });
try { console.log(require("pkg")) } catch { console.log("first try failed") }
fs.writeFileSync("node_modules/pkg/lib/main.js", "module.exports = `lib/main was loaded`");
fs.writeFileSync("node_modules/pkg/package.json", JSON.stringify({main:"lib/main.js"}));
try { console.log(require("pkg")) } catch { console.log("second try failed") }

[aduh95]

Ah right, the issue is that Node.js doesn't re-check the FS if there was no package.json found. If you write the package.json before the first require, it doesn't reproduce:

mkdir repro && cd repro
echo 'const fs = require("node:fs");' >> entry.js
echo 'fs.mkdirSync("node_modules/pkg/lib", { recursive:true });' >> entry.js
echo 'fs.writeFileSync("node_modules/pkg/package.json", JSON.stringify({main:"lib/main.js"}));' >> entry.js
echo 'try { console.log(require("pkg")) } catch { console.log("first try failed") }' >> entry.js
echo 'fs.writeFileSync("node_modules/pkg/lib/main.js", "module.exports = `lib/main was loaded`");' >> entry.js
echo 'try { console.log(require("pkg")) } catch { console.log("second try failed") }' >> entry.js
node entry.js
cd .. && rm -r repro

I'm not sure it's worth fixing, having to re-request the FS and re-parse the package.json for each require call would be very inefficient. /cc @nodejs/modules

[ljharb]

I agree - has the use case of dynamically creating requireable files been one node has ever intentionally supported?

[davidfirst]

@aduh95 , if it's not worth fixing, it would be great to have the option to "clear" this kind-of cache of the non-found package.json files. (similar to the ability to clear require.cache).