Slow packaging of pnpm workspaces project

[jonim8or]

What version of pkg are you using?

6.0.0, 5.16.0

Which version(s) of pkg work for you?

5.15.0

What version of Node.js are you using?

20.18.0

What operating system are you using?

Windows

What CPU architecture are you using?

x86_64

What Node versions, OSs and CPU architectures are you building for?

node20-linux-x64

Describe the Bug

I have a pnpm project using workspaces. build time is significantly slower in pkg 5.16 onwards. Our project builds in 22 seconds using pkg 5.15, and 35 minutes when using pkg 5.16 or later.

Expected Behavior

Build time should be similar for both. Or at least in the same order of magnitude.

To Reproduce

Unzip the reproduce.zip folder.

reproduce.zip

run these commands:

pnpm install
time pnpm -r artifacts

then in packages/foo/package.json change the version to ~5.16.0, and repeat the above commands.
For me this are the execution times:

pkg version	build time
~5.15.0	0m5.618s
~5.16.0	0m28.320s
~6.0.0	0m30.630s

[robertsLando]

@SuperchupuDev could this be due to globby/minimatch replacement?

[SuperchupuDev]

probably, there are some cases where optimizations can't be applied due to the glob patterns used, a workaround can be ignoring node_modules from globbing. sorry about that. there's some ongoing work in tinyglobby to greatly improve optimizations

[robertsLando]

@SuperchupuDev could you submit a PR with the suggested optimization please? Otherwise I would revert the glob replacement for now as the time is 5x and is too high

[SuperchupuDev]

should be a one line fix with the ignore option, can do, but first i must ask, is the collect function ever supposed to return files from node_modules? if yes i'll have to fix the problem at its core (finishing the better optimizations in tinyglobby), which shouldn't take long as most of the work is done

[robertsLando]

@SuperchupuDev Yeah that could return files from node_modules

[SuperchupuDev]

ah, then i'll try to finish the optimizations and publish a new tinyglobby version. meanwhile, can you try to give me a list of glob patterns commonly used by pkg? just to make sure

[robertsLando]

@SuperchupuDev Glob patterns used there can be provided by users, them are fetched from scripts and assets provided in pkg configuration in package.json

[SuperchupuDev]

interesting. what's the pattern used in the repro provided in this issue? is it the same one that's actually passed to tinyglobby or is it transformed in some way beforehand?

[robertsLando]

I have no clue, I think you can try with the provided zip in the issue above?

I also have a feel that the reason caused issue listed in #112 test-80-compression-node-opcua, maybe you can give a try with that? Actually that test takes forever and uses pnpm and has assets. See development.md to understand how to spin tests locally

[SuperchupuDev]

small update: SuperchupuDev/tinyglobby#69 is in theory done, but it breaks vite's tests. will investigate more tomorrow, and merge once i can make sure it doesn't cause any regressions

[SuperchupuDev]

debugging the repro zip file made me see the issue with the glob usage done in this package: it is calling globSync many times, when once should be enough. improving this should solve this issue without needing to wait for tinyglobby

[SuperchupuDev]

i've opened #121 which should solve this issue, but the usage of globSync is still not optimal here until this project stops calling globSync more than once, regardless of the glob library used