UCX ERROR Failed to allocate memory pool

[orensg1]

Code of Conduct

• I have read and agree to Star HPC's Code of Conduct

What is the problem?

Default value for max memory pool too low.
When running pytorch and allocating large amounts of memory (>8M) the following error appears:

[1740171758.025024] [gpu6:1025296:0]           mpool.c:269  UCX  ERROR Failed to allocate memory pool (name=ud_recv_skb) chunk: Input/output error
[1740171758.025504] [gpu6:1025296:0]           ib_md.c:282  UCX  ERROR ibv_reg_mr(address=0x7ffdd8600000, length=20971520, access=0x10000f) failed: Cannot allocate memory : Please set max locked memory (ulimit -l) to 'unlimited' (current: 8192 kbytes)

Steps to Reproduce

ulimit -a

real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 3092588
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 513293
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

ulimit -l unlimited

ulimit -a

real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 3092588
max locked memory           (kbytes, -l) unlimited
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 513293
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
``

### Expected Results

See above

### Additional information

See above

[orensg1]

To solve the problem the user currently needs to run:
ulimit -l unlimited

[alex-rosenberg35]

@orensg1 We could change the default value, but the value requires further consideration. I doubt it would be in everyone's best interest to set this to unlimited by default.

[orensg1]

@alex-rosenberg35, makes sense but we should add to the documentation a blurb about how to deal with this problem because I suspect I will not be the only one having to deal with this issue.
In any case, it is not urgent but you should be aware of this issue if/when users start complaining about this error when running their software tools.

[MKhlystun]

Hi @orensg1,
Thank you for pointing out the issue. Alex and I discussed it, and we found that 8K for "max locked memory" is too low for modern workloads. I have increased the default value to 1GB. Could you verify whether your workload still encounters this issue with the new default setting?

[orensg1]

Hi @orensg1, Thank you for pointing out the issue. Alex and I discussed it, and we found that 8K for "max locked memory" is too low for modern workloads. I have increased the default value to 1GB. Could you verify whether your workload still encounters this issue with the new default setting?

I ran a couple of heavy workloads and setting the value to 1GB solved the issue. Thank you.