Skip to content

slow module loads and purge #927

@puneet336

Description

@puneet336

Hi ,
We are currently in a situation where in we installed a R bundle using easybuild. but with the generated module files, the module load and module purge are taking more than 15 seconds.
I f this was a code, I would have attached a debugger to see which line of code is taking longer. Could you please help me to understand how and thich tools can I debug the root cause of slowness here ?
Here is the command I use to reproduce the slowness -

So far we have observed slowness issues with only this particular module.

Example:
[singhpuv@server ~]$ time module load R-bundle-Novartis

real 0m16.866s
user 0m15.105s
sys 0m0.082s
[singhpuv@Pserver~]$ module list
Currently Loaded Modulefiles:

  1. GCCcore/11.2.0 83) Mesa/22.0.3-GCCcore-11.2.0
  2. zlib/1.2.11-GCCcore-11.2.0 84) libGLU/9.0.2-GCCcore-11.2.0
  3. binutils/2.37-GCCcore-11.2.0 85) nodejs/18.17.1-GCCcore-11.2.0
  4. GCC/11.2.0 86) MPFR/4.1.0-GCCcore-11.2.0
  5. numactl/2.0.14-GCCcore-11.2.0 87) OpenJPEG/2.5.0-GCCcore-11.2.0
  6. XZ/5.2.5-GCCcore-11.2.0 88) Pandoc/3.1.2
  7. libxml2/2.9.13-GCCcore-11.2.0 89) NSPR/4.34-GCCcore-11.2.0
  8. libpciaccess/0.16-GCCcore-11.2.0-BuildNAOS 90) NSS/3.79-GCCcore-11.2.0
  9. hwloc/2.7.1-GCCcore-11.2.0 91) double-conversion/3.2.0-GCCcore-11.2.0
  10. OpenSSL/1.1 92) graphite2/1.3.14-GCCcore-11.2.0
  11. libevent/2.1.12-GCCcore-11.2.0 93) DBus/1.14.0-GCCcore-11.2.0
  12. UCX/1.12.1-GCCcore-11.2.0 94) snappy/1.1.9-GCCcore-11.2.0
  13. libfabric/1.15.1-GCCcore-11.2.0 95) Qt5/5.15.5-GCCcore-11.2.0-MEISSA
  14. PMIx/4.1.2-GCCcore-11.2.0 96) Boost/1.81.0-GCC-11.2.0
  15. UCC/1.0.0-GCCcore-11.2.0 97) poppler/22.01.0-GCC-11.2.0-MEISSA
  16. OpenMPI/4.1.4-GCC-11.2.0 98) OpenPGM/5.2.122-GCCcore-11.2.0
  17. gompi/2022a 99) libsodium/1.0.18-GCCcore-11.2.0
  18. imkl/2022.1.0-gompi-2022a 100) ZeroMQ/4.3.4-GCCcore-11.2.0-BuildNAOS
  19. gomkl/2022a 101) FFTW/3.3.10-gompi-2022a
  20. OracleInstantClient/21.12-el9 102) Szip/2.1.1-GCCcore-11.2.0
  21. ncurses/6.2-GCCcore-11.2.0 103) HDF5/1.12.1-gompi-2022a
  22. libreadline/8.1-GCCcore-11.2.0 104) libogg/1.3.5-GCCcore-11.2.0
  23. bzip2/1.0.8-GCCcore-11.2.0 105) FLAC/1.3.4-GCCcore-11.2.0
  24. PCRE2/10.40-GCCcore-11.2.0 106) libvorbis/1.3.7-GCCcore-11.2.0
  25. cURL/7.78.0-GCCcore-11.2.0 107) libopus/1.3.1-GCCcore-11.2.0
  26. M4/1.4.20-GCCcore-11.2.0 108) LAME/3.100-GCCcore-11.2.0
  27. expat/2.4.8-GCCcore-11.2.0 109) libsndfile/1.1.0-GCCcore-11.2.0
  28. DB/18.1.40-GCCcore-11.2.0 110) libpipeline/1.5.7-GCC-11.2.0
  29. Perl/5.34.0-GCCcore-11.2.0-BuildNAOS 111) man-db/2.11.2-GCC-11.2.0
  30. Autoconf/2.71-GCCcore-11.2.0-BuildNAOS 112) NLopt/2.7.1-GCCcore-11.2.0
  31. Automake/1.16.5-GCCcore-11.2.0-BuildNAOS 113) UDUNITS/2.2.28-GCCcore-11.2.0
  32. libtool/2.4.6-GCCcore-11.2.0 114) netCDF/4.9.0-gompi-2022a
  33. Autotools/20220317-GCCcore-11.2.0 115) GEOS/3.11.1-GCC-11.2.0
  34. libpng/1.6.37-GCCcore-11.2.0 116) nlohmann_json/3.11.2-GCCcore-11.2.0
  35. Brotli/1.0.9-GCCcore-11.2.0 117) PROJ/9.0.0-GCCcore-11.2.0
  36. freetype/2.12.1-GCCcore-11.2.0 118) libgeotiff/1.7.1-GCCcore-11.2.0
  37. pixman/0.40.0-GCCcore-11.2.0 119) pybind11/2.10.3-GCCcore-11.2.0
  38. libffi/3.4.4-GCCcore-11.2.0 120) hypothesis/6.14.6-gomkl-2022a
  39. gettext/0.21-GCCcore-11.2.0-BuildNAOS 121) numpy/1.23.5-gomkl-2022a
  40. PCRE/8.45-GCCcore-11.2.0 122) SciPy-bundle/2022.05-gomkl-2022a
  41. util-linux/2.38-GCCcore-11.2.0 123) libtirpc/1.3.2-GCCcore-11.2.0
  42. GLib/2.72.1-GCCcore-11.2.0 124) HDF/4.2.15-GCCcore-11.2.0
  43. fontconfig/2.14.0-GCCcore-11.2.0 125) Eigen/3.4.0-GCCcore-11.2.0
  44. xorg-macros/1.19.3-GCCcore-11.2.0-BuildNAOS 126) arpack-ng/3.8.0-gomkl-2022a
  45. X11/20220504-GCCcore-11.2.0 127) Armadillo/11.4.3-gomkl-2022a-MEISSA
  46. cairo/1.17.4-GCCcore-11.2.0 128) CFITSIO/4.2.0-GCCcore-11.2.0
  47. Java/17.0.6 129) giflib/5.2.1-GCCcore-11.2.0
  48. ICU/74.1-GCCcore-11.2.0 130) json-c/0.16-GCCcore-11.2.0
  49. NASM/2.15.05-GCCcore-11.2.0 131) Xerces-C++/3.2.4-GCCcore-11.2.0
  50. libjpeg-turbo/2.1.3-GCCcore-11.2.0 132) Imath/3.1.5-GCCcore-11.2.0
  51. jbigkit/2.1-GCCcore-11.2.0 133) OpenEXR/3.1.5-GCCcore-11.2.0
  52. gzip/1.12-GCCcore-11.2.0 134) Highway/1.0.3-GCCcore-11.2.0
  53. lz4/1.9.3-GCCcore-11.2.0 135) Brunsli/0.1-GCCcore-11.2.0
  54. zstd/1.5.2-GCCcore-11.2.0 136) Qhull/2020.2-GCCcore-11.2.0
  55. LibTIFF/4.3.0-GCCcore-11.2.0 137) LERC/4.0.0-GCCcore-11.2.0
  56. HarfBuzz/8.2.2-GCCcore-11.2.0-MEISSA 138) libwebp/1.3.1-GCCcore-11.2.0
  57. FriBidi/1.0.12-GCCcore-11.2.0 139) GDAL/3.6.2-gomkl-2022a-MEISSA
  58. Pango/1.50.7-GCCcore-11.2.0-MEISSA 140) unixODBC/2.3.11-gomkl-2022a
  59. Tcl/8.6.12-GCCcore-11.2.0 141) GObject-Introspection/1.72.0-GCCcore-11.2.0
  60. Tk/8.6.12-GCCcore-11.2.0 142) pkg-config/0.29.2-GCCcore-11.2.0
  61. Zip/3.0-GCCcore-11.2.0 143) librsvg/2.54.2-gomkl-2022a-MEISSA
  62. CMake/3.25.0-GCCcore-11.2.0 144) phantomjs/2.1.1-gomkl-2022a
  63. cmdstan/2.32.2-gomkl-2022a 145) MPC/1.2.1-GCCcore-11.2.0
  64. ATK/2.38.0-GCCcore-11.2.0 146) gmpy2/2.1.2-GCC-11.2.0
  65. SQLite/3.39.4-GCCcore-11.2.0 147) sympy/1.9-gomkl-2022a
  66. GMP/6.2.1-GCCcore-11.2.0 148) PostgreSQL/14.4-GCCcore-11.2.0
  67. Python/3.10.7-GCCcore-11.2.0-BuildNAOS 149) libpq/13.20-el9_5
  68. Gdk-Pixbuf/2.42.8-GCCcore-11.2.0 150) texlive/20210325-GCCcore-11.2.0-MEISSA
  69. six/1.16.0-gomkl-2022a 151) rpmrebuild/2.11
  70. GTK2/2.24.33-GCCcore-11.2.0-MEISSA 152) shinycannon/1.1.3-linux-x86_64
  71. Ghostscript/9.56.1-GCCcore-11.2.0-MEISSA 153) MariaDB-connector-c/3.3.5-gomkl-2022a
  72. JasPer/4.0.0-GCCcore-11.2.0 154) quartocli/1.6.40
  73. LittleCMS/2.13.1-GCCcore-11.2.0 155) tbb/2018_U5-GCCcore-11.2.0
  74. ImageMagick/7.1.0-4-GCCcore-11.2.0-MEISSA 156) libarchive/3.7.7-GCCcore-11.2.0
  75. GLPK/5.0-GCCcore-11.2.0 157) texinfo/7.1-GCCcore-11.2.0
  76. GSL/2.7-GCC-11.2.0 158) hiredis/1.3.0-GCCcore-11.2.0
  77. JAGS/4.3.2-gomkl-2022a 159) libxslt/1.1.43-GCCcore-11.2.0
  78. libgit2/1.4.3-GCCcore-11.2.0 160) Abseil/20250512.1-GCCcore-11.2.0
  79. libdrm/2.4.110-GCCcore-11.2.0 161) protobuf/31.1-GCCcore-11.2.0
  80. libglvnd/1.4.0-GCCcore-11.2.0 162) R/4.3.1-gomkl-2022a-0.4
  81. libunwind/1.6.2-GCCcore-11.2.0 163) R-bundle-Novartis/0.5-gomkl-2022a-R-4.3.1
  82. LLVM/12.0.1-GCCcore-11.2.0

Key:
auto-loaded
[singhpuv@server ~]$ time module purge

real 0m18.310s
user 0m17.923s
sys 0m0.045s
[singhpuv@server ~]$

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions