Automatic making host closed by reservation memory

To prevent another running job suspend on a execution server, admin requests new LSF feature.

It is, all job's total reserved memory value is same or exceeds real physical memory size, admin want the host could be closed automatically.
(This doesn't care extra slots on the server)

Let's suppose there is a server which has 100G memory and 4 slots.
UserA submits a job with 50G mem reservation and 1 slot.
UserB submits a job with 50G mem reservation and 1 slot.

In this situation, 2 slots are remained on this host.
And because these two jobs doesn't use whole 50G MEM from starting point.
In LSF side, this host recognizes that the host has 98-99G mem.
Additionally, lsload and bhosts, this host status would be 'ok'

if userC submit a job with 2 slot without mem reservation, this job will be dispatch to this host.
from begin time to somewhere(we don't know exact point), these 4 jobs are run well

but, from these 4 jobs start to use memory gradually, situation becomes wrong.
So, admin's requirement is,
if users reserve memory and whole reserved memory size is matched with real physical memory, admin want to make that host as "Closed"

Post comment

Guest

Reply
| Aug 20, 2019

This RFE asks for "So customer needs, LSF can make host 'Closed' status when user reserves almost memory of server's, as like NC's 'FULL'."
This functionality can be achieved by setting the loadsched memory threshold for the host(s). The host(s) will then be closed when reserved memory falls below the threshold.

[user001@chjtlnx2048 ~]$ bhosts -l chjtlnx2048
HOST chjtlnx2048
STATUS CPUF JL/U MAX NJOBS RUN SSUSP USUSP RSV DISPATCH_WINDOW
ok 12.50 - 4 0 0 0 0 0 -

CURRENT LOAD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem slots
Total 1.0 0.0 0.0 0% 0.0 1 1 28 39G 0M 2.8G 4
Reserved 0.0 0.0 0.0 0% 0.0 0 0 0 0M 0M 0M -

LOAD THRESHOLD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - 100M
loadStop - - - - - - - - - - 10M

CONFIGURED AFFINITY CPU LIST: all

[user001@chjtlnx2048 ~]$ bsub -R "rusage[mem=1400]" -J j1 /home/user001/work/src/cpp/mem -m 100 ===>this job will consume 100M memory and reserve 1400M memory
Job <522> is submitted to default queue .
[user001@chjtlnx2048 ~]$ bsub -R "rusage[mem=1400]" -J j2 /home/user001/work/src/cpp/mem -m 100 ===>this job will consume 100M memory and reserve 1400M memory
Job <523> is submitted to default queue .
[user001@chjtlnx2048 ~]$ bhosts -l chjtlnx2048
HOST chjtlnx2048
STATUS CPUF JL/U MAX NJOBS RUN SSUSP USUSP RSV DISPATCH_WINDOW
closed_Busy 12.50 - 4 2 2 0 0 0 -

CURRENT LOAD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem slots
Total 1.0 0.0 0.0 1% 0.0 4 1 0 39G 0M *86M 2 ====>loadSched threshold is not met (86 < 100) now and the host status becomes closed (close_busy), no job can dispatch to this host
Reserved 0.0 0.0 0.0 0% 0.0 0 0 0 0M 0M 2.5G - ====>we can see that the available memory (total + reserved) reserved by jobs 522 and 523

LOAD THRESHOLD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - *100M
loadStop - - - - - - - - - - 10M

CONFIGURED AFFINITY CPU LIST: all

[user001@chjtlnx2048 ~]$ bsub -J j3 sleep 10000
Job <524> is submitted to default queue .
[user001@chjtlnx2048 ~]$ bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
522 user001 RUN normal chjtlnx2048 chjtlnx2048 j1 Aug 19 23:13
523 user001 RUN normal chjtlnx2048 chjtlnx2048 j2 Aug 19 23:13
524 user001 PEND normal chjtlnx2048 j3 Aug 19 23:16
[user001@chjtlnx2048 ~]$ bjobs -l 524

Job <524>, Job Name , User , Project , Status , Que
ue , Command
Mon Aug 19 23:16:45: Submitted from host , CWD <$HOME>;
PENDING REASONS:
The available memory (mem) is low: 1 host;
....

[user001@chjtlnx2048 ~]$bkill 522
Job <522> is being terminated
[user001@chjtlnx2048 ~]$ bhosts -l ====>the host status became OK now (1.4G > 100M)
HOST chjtlnx2048
STATUS CPUF JL/U MAX NJOBS RUN SSUSP USUSP RSV DISPATCH_WINDOW
ok 12.50 - 4 2 2 0 0 0 -

CURRENT LOAD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem slots
Total 0.9 0.0 0.0 1% 0.0 10 1 0 39G 0M 1.4G 2
Reserved 0.0 0.0 0.0 0% 0.0 0 0 0 0M 0M 1.2G -

LOAD THRESHOLD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - 100M
loadStop - - - - - - - - - - 10M

CONFIGURED AFFINITY CPU LIST: all

[user001@chjtlnx2048 ~]$

0 reply Hide replies

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.

Please enter your email address

RELATED IDEAS

Automatic making host closed by reservation memory