Skip to content

nix-required-mounts: support deep symlinks; refactor & test#500971

Merged
SomeoneSerge merged 1 commit into
NixOS:masterfrom
tfc:unwhack-nrm
Apr 8, 2026
Merged

nix-required-mounts: support deep symlinks; refactor & test#500971
SomeoneSerge merged 1 commit into
NixOS:masterfrom
tfc:unwhack-nrm

Conversation

@tfc
Copy link
Copy Markdown
Contributor

@tfc tfc commented Mar 18, 2026

I have been running cuda apps on both NVIDIA and AMD (with ZLUDA) graphics cards and found that nix-required-mounts produced some problems:

  • created paths that don't exist
  • infinite looping
  • extremely slow execution because there were too many paths (N*N complexity instead of N buried somewhere)

While fixing these issues, i decided that the scenario is complex enough for test cases, so i wrote those and restructured the app a bit to make it more testable.

Tested these changes again on two NVIDIA and AMD machines, works very nicely.

Fixes #497824

CC @SomeoneSerge @ConnorBaker @kmein @jfly

Things done

  • Built on platform:
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • Tested, as applicable:
  • Ran nixpkgs-review on this PR. See nixpkgs-review usage.
  • Tested basic functionality of all binary files, usually in ./result/bin/.
  • Nixpkgs Release Notes
    • Package update: when the change is major or breaking.
  • NixOS Release Notes
    • Module addition: when adding a new NixOS module.
    • Module update: when the change is significant.
  • Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

@nixpkgs-ci nixpkgs-ci Bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. 2.status: merge-bot eligible This PR can be merged by commenting "@NixOS/nixpkgs-merge-bot merge". labels Mar 18, 2026
@tfc tfc force-pushed the unwhack-nrm branch 3 times, most recently from 4acdde7 to 67747fd Compare March 18, 2026 11:24
@kmein
Copy link
Copy Markdown
Member

kmein commented Mar 18, 2026

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 500971
Commit: 67747fd2b7659d6fe4dcd7dde694c53b392518a1


x86_64-linux

✅ 2 packages built:
  • nix-required-mounts
  • nix-required-mounts.dist

Copy link
Copy Markdown
Member

@kmein kmein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functionality-wise, everything looks really good. I've left a few inline comments throughout the patch—mostly just some naming questions and minor nitpicks around type hints and a small optimization.

Let me know what you think of those, but overall this is in great shape!

Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/test.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/test.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
@tfc
Copy link
Copy Markdown
Contributor Author

tfc commented Mar 18, 2026

TODO:

  • i will add a testcase that proves why we need to resolve symlink parents
  • split up "paths" into "paths" and "storepaths" so we don't do the recursion needlessly on nix store paths whose closure has long been pre-calculated already. maybe next PR

Comment thread pkgs/by-name/ni/nix-required-mounts/package.nix Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py
@tfc tfc force-pushed the unwhack-nrm branch 2 times, most recently from d6e144a to f2797ce Compare March 18, 2026 16:19
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still a git mv. With py_modules in pyproject.toml it should be fine to just keep the top-level module

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on the one hand there is the git mv, on the other hand it's much more ergonomic during development to have all files that belong to the actual tool without the extra stuff in one folder. Especially because you don't have to explain black, mypy, and pytest, which files to ignore.

I understand preserving blame history is useful, but is it more useful than tool ergonomy while maintaining the app?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this somehow affects ergonomics in your environment, there is something off about your environment. Top-level modules are perfectly normal and should not cause any issues.

Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py
Comment thread pkgs/by-name/ni/nix-required-mounts/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py
Comment thread pkgs/by-name/ni/nix-required-mounts/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/test_nix_required_mounts.py
@SomeoneSerge
Copy link
Copy Markdown
Contributor

SomeoneSerge commented Apr 7, 2026 via email

@tfc
Copy link
Copy Markdown
Contributor Author

tfc commented Apr 7, 2026

@SomeoneSerge

  • pushed the latest change request
  • squashed all commits (don't panic if you were not through yet, i parked the unsquashed changes on another branch if you want that back for your convenience)
  • added you as co-author in the commit message

@SomeoneSerge
Copy link
Copy Markdown
Contributor

I'd say

  1. ...: fix paths -> something like ...: mount symlink children's targets & add tests
    Breaking change: instead of expanding a single symlink chain,
    unsafeFollowSymlinks now walks directories recursively and
    mounts all symlink descendants' targets.
    This accommodates usecases like:
    /sys/dev/char/226:128 -> ../../devices/pci0000:00/0000:00:02.0/drm/renderD128
    /sys/dev/char/226:128/subsystem -> ../../../../../class/drm
    
  2. Undo git mv

And we're done.

@tfc tfc force-pushed the unwhack-nrm branch 2 times, most recently from ad2f868 to e08b757 Compare April 7, 2026 17:15
@tfc
Copy link
Copy Markdown
Contributor Author

tfc commented Apr 7, 2026

@SomeoneSerge done

@tfc
Copy link
Copy Markdown
Contributor Author

tfc commented Apr 7, 2026

ok there are pipeline errors because i used lib.fileset to filter everything that does not belong to the app and its tests. As the error message suggests, instead of filtering the app should go into a subfolder.

As the app has only been touched 3 times in the last 2 years i would rather drop the git history and go back to the subfolder. This way, the closure .py and .nix files and everything else does not become part of the package closure.

Please advise.

@SomeoneSerge SomeoneSerge changed the title nix-required-mounts: fix paths nix-required-mounts: support deep symlinks; refactor & test Apr 7, 2026
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
@SomeoneSerge
Copy link
Copy Markdown
Contributor

SomeoneSerge commented Apr 7, 2026

RE: fileset

Wow this warning is ludicrous, fileset has been the primary and default method for defining this kind of src for 1+ years.

EDIT: So one more "nix doesn't vfs" issue, breaking a documented nixpkgs functionality...

@nixpkgs-ci nixpkgs-ci Bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Apr 7, 2026
@SomeoneSerge
Copy link
Copy Markdown
Contributor

@tfc I can't deny I'm partially motivated by spite, but I want this to be lib.sources then (which uses relative paths; lib.sources.sourceFilesBySuffices or lib.sources.sourceByRegex), so far as the github action isn't reverted.

@tfc
Copy link
Copy Markdown
Contributor Author

tfc commented Apr 8, 2026

@SomeoneSerge understandable. done.

@nixpkgs-ci nixpkgs-ci Bot added the 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages. label Apr 8, 2026
Comment thread pkgs/by-name/ni/nix-required-mounts/nix_required_mounts.py Outdated
Breaking change: instead of expanding a single symlink chain,
unsafeFollowSymlinks now walks directories recursively and
mounts all symlink descendants' targets.
This accommodates usecases like:
/sys/dev/char/226:128 -> ../../devices/pci0000:00/0000:00:02.0/drm/renderD128
/sys/dev/char/226:128/subsystem -> ../../../../../class/drm

Fixes issue NixOS#497824

Co-authored-by: SomeoneSerge <else@someonex.net>
@SomeoneSerge SomeoneSerge added this pull request to the merge queue Apr 8, 2026
Merged via the queue into NixOS:master with commit 3a24d4b Apr 8, 2026
26 checks passed
@SomeoneSerge
Copy link
Copy Markdown
Contributor

Thank you @tfc, @kmein, @jfly! I know this has taken quite a bit longer and more friction than you aimed for... That said, I liked and learned a lot from all the discussions, and appreciate the thought you've put into this

@tfc tfc deleted the unwhack-nrm branch April 8, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 12.approvals: 1 This PR was reviewed and approved by one person. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nix_required_mounts.py::symlink_parents returns non-existent paths

5 participants