Use all applicable NUMA Memory information on a machine to determine available Memory resources before dispatching a job to a machine

We were noticing that jobs were being terminated by LSF with the message:

TERM_MEMLIMIT: job killed after reaching LSF memory usage limit.
Exited with exit code 137.

Yet, this was happening before the job had even consumed the amount of memory reserved for the job. After working with IBM support, we learned that currently LSF only considers MemTotal when doing memory affinity scheduling, but not MemFree, and this is a limitation of LSF.

Example

$ numastat -cnm | grep -B 3 MemFree
Node 0 Node 1 Node 2 Node 3 Total
------ ------ ------ ------ ------
MemTotal 192144 193525 193484 193515 772669
MemFree 182323 185487 117121 129386 614317

We would like to request an update to the logic in LSF that calculates available NUMA memory on a node to consider all applicable NUMA memory statistics/data before dispatching a job to a machine, to make certain that jobs have the memory they need during execution.

Needed By

Quarter

Post comment

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.

Please enter your email address

RELATED IDEAS

Use all applicable NUMA Memory information on a machine to determine available Memory resources before dispatching a job to a machine