Advise on HPC conda environments consuming home directory quota

[thompsonmj]

It's been a repeated occurrence that conda environments, which default to living in a user's home directory, causes the user's quota to exceed the file count limit on OSC.

To fix this, conda can be configured to store environments and packages in a custom location, scratch storage being an appropriate option I use and have recommended. E.g., in a ~/.condarc file:

solver: libmamba
envs_dirs:
  - /fs/scratch/PAS2136/$USER/conda-envs
pkgs_dirs:
  - /fs/scratch/PAS2136/$USER/conda-pkgs

channels:
  - conda-forge
  - defaults

Would it make sense to do the following:

• Add a general discussion of this issue under the Conda section of Virtual Environments
• Provide a generic .condarc template that can be copied using /path/to/desired/user/conda-envs and /path/to/desired/user/conda-pkgs

Should an entry be made into the internal guidelines wiki for Imageomics using the actual paths?

[hlapp]

Yes, putting conda environments into your home directory on a shared cluster is a bad idea, including in major ways for the reason you say. Another is that they are then essentially not shareable.

The simple practice I use and that has worked entirely fine is to put conda environments into places where there is enough storage, and that is accessible to a whole lab group, which is easily done using the -p argument to conda create. I.e., when creating a confide environment, use paths, not names. The former is much more reproducible than the former.

That said, this still doesn't address the location of the package cache, which also defaults to your home directory. So a mechanism is still needed to ensure the package cache goes somewhere else, and arguably it should be scratch (in contrast to conda environments, which arguably should not go to scratch space, where they are at risk of getting purged after some time).