node_modules linker refactor

[Silic0nS0ldier]

The existing linker has the following issues;

1. Not thread-safe
   Symlinks are created in shared spaces that point to action specific runfiles symlink forests and
   leading to race conditions (linkers for different executables writing conflicting symlinks). The
   latter results in actions running wihtout access to their node_modules inputs.
2. Non-portable symlink paths
   Symlink targets are computed in a separate action which may not be using the same execution
   strategy or will simply never match if sandboxing is enabled, resulting in invalid symlinks.
3. Leaking inputs
   The node_modules created closest to cwd points to the originally generated node_modules
   external repository. This contains all packages for the owning package.json/yarn.lock pair
   and in conjunction with managed_directories allows these packages to use files in the user
   workspace.

This PR refactors the linker so symlinks are written under [name].runfiles/ in a thread-safe manner.

Compared to the previous implementation;

• node_modules symlinks are written inside the runfiles directory of the current executable.
• Symlink destinations are rooted in the runfiles directory, preventing certain classes of input leaks via the managed_directories Bazel feature.

  Before:

  • /__cwd__/web/node_modules ↩
    /__output_base__/external/web_node_modules/node_modules
    (with managed_directories, this is a symlink into the user workspace)
  • /__output_base__/execroot/com_canva_canva/bazel-out/_/bin/___/_.runfiles/com_canva_canva/web/node_modules ↩
    /__output_base__/execroot/com_canva_canva/web/node_modules
  • /__cwd__/bazel-out/_/bin/web/node_modules ↩
    /__output_base__/execroot/com_canva_canva/web/node_modules

  After

  • /__output_base__/execroot/com_canva_canva/bazel-out/_/bin/___/_.runfiles/com_canva_canva/web/node_modules ↩
    /__output_base__/execroot/com_canva_canva/bazel-out/_/bin/___/_.runfiles/web_node_modules/node_modules

• Elimination of cross-contamination issues due to writing symlinks in directories visible to different executables.
• Sandboxing support (--worker_sandboxing and sandboxed spawn strategies).
• A big step towards remote execution support.

[christianscott]

LGTM, nice work

[Silic0nS0ldier]

Remaining issue is due to flaws around how Rollup is used. The same issue that had to be solved internally when this change was first rolled out via a .patch file.

I'll need to rework existing Rollup usage to work with this.

[Silic0nS0ldier]

More issues were discovered. e.g. TypeScript rules (ts_project and the tsc generated rule) are unable to resolve node_modules, so types from packages like @types/node cannot be found.

Given sufficient time these issues can be overcome, however the goal here was to simplify patch management. To that end, repo source will be replaced with v3.8.0 release artefact contents.

Closing PR.