The function that can close host automatically by reserved memory

One of competition scheduler, Runtime NC(Network Computing) has a useful function.
It is to make execution server "FULL" status automatically when user submits a job with reserved memory.
If do so, it doesn't dispatch a job to that host any more.

I tried to find similar function and implement it in LSF. But, unfortunately, there is no one in LSF.

In LSF, if user reserves memory and then run 'lsload', lsload displays just available memory that reserved memory isn't applied.
And if we run 'bhosts -l hostname', we can see reserved memory value. But, because it is just reserved memory, even if we reserve whole mem, 'bhosts -l hostname' displays 'ok' status to us.

Sometimes, this could make some issues.
For example, let's suppose there is hostA server that has 100G memory size.
If userA submits a job with reserved memory 90G and this job is dispatched to hostA, 'lsload' displays 100G memory as available size and 'bhosts' displays, Total 10G and Reserved 90G. Both of them displays 'ok' status for hostA.

In this situation, userB submits a job without memory reservation and this job will be dispatched to hostA. But actually userB's job doesn't use much memory starting time. It uses memory gradually.

Due to using memory for two jobs, hostA server sometimes becomes big problem and two job also get big affect.

Currently, customer's LSF cluster has 1,700-1,800 hosts. For so many servers, it is so complicated and difficult for us to make elim and threshold. additionally, there are a lot of applications and they has different application's characteristics.

So customer needs, LSF can make host 'Closed' status when user reserves almost memory of server's, as like NC's 'FULL'.

Post comment

Guest

Reply
| Aug 20, 2019

This RFE asks for "So customer needs, LSF can make host 'Closed' status when user reserves almost memory of server's, as like NC's 'FULL'."

This functionality can be achieved by setting the loadsched memory threshold for the host(s). The host(s) will then be closed when reserved memory falls below the threshold.

[user001@chjtlnx2048 ~]$ bhosts -l chjtlnx2048
HOST chjtlnx2048
STATUS CPUF JL/U MAX NJOBS RUN SSUSP USUSP RSV DISPATCH_WINDOW
ok 12.50 - 4 0 0 0 0 0 -

CURRENT LOAD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem slots
Total 1.0 0.0 0.0 0% 0.0 1 1 28 39G 0M 2.8G 4
Reserved 0.0 0.0 0.0 0% 0.0 0 0 0 0M 0M 0M -

LOAD THRESHOLD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - 100M
loadStop - - - - - - - - - - 10M

CONFIGURED AFFINITY CPU LIST: all

[user001@chjtlnx2048 ~]$ bsub -R "rusage[mem=1400]" -J j1 /home/user001/work/src/cpp/mem -m 100 ===>this job will consume 100M memory and reserve 1400M memory
Job <522> is submitted to default queue .
[user001@chjtlnx2048 ~]$ bsub -R "rusage[mem=1400]" -J j2 /home/user001/work/src/cpp/mem -m 100 ===>this job will consume 100M memory and reserve 1400M memory
Job <523> is submitted to default queue .
[user001@chjtlnx2048 ~]$ bhosts -l chjtlnx2048
HOST chjtlnx2048
STATUS CPUF JL/U MAX NJOBS RUN SSUSP USUSP RSV DISPATCH_WINDOW
closed_Busy 12.50 - 4 2 2 0 0 0 -

CURRENT LOAD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem slots
Total 1.0 0.0 0.0 1% 0.0 4 1 0 39G 0M *86M 2 ====>loadSched threshold is not met (86 < 100) now and the host status becomes closed (close_busy), no job can dispatch to this host
Reserved 0.0 0.0 0.0 0% 0.0 0 0 0 0M 0M 2.5G - ====>we can see that the available memory (total + reserved) reserved by jobs 522 and 523

LOAD THRESHOLD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - *100M
loadStop - - - - - - - - - - 10M

CONFIGURED AFFINITY CPU LIST: all

[user001@chjtlnx2048 ~]$ bsub -J j3 sleep 10000
Job <524> is submitted to default queue .
[user001@chjtlnx2048 ~]$ bjobs
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
522 user001 RUN normal chjtlnx2048 chjtlnx2048 j1 Aug 19 23:13
523 user001 RUN normal chjtlnx2048 chjtlnx2048 j2 Aug 19 23:13
524 user001 PEND normal chjtlnx2048 j3 Aug 19 23:16
[user001@chjtlnx2048 ~]$ bjobs -l 524

Job <524>, Job Name , User , Project , Status , Que
ue , Command
Mon Aug 19 23:16:45: Submitted from host , CWD <$HOME>;
PENDING REASONS:
The available memory (mem) is low: 1 host;
....

[user001@chjtlnx2048 ~]$bkill 522
Job <522> is being terminated
[user001@chjtlnx2048 ~]$ bhosts -l ====>the host status became OK now (1.4G > 100M)
HOST chjtlnx2048
STATUS CPUF JL/U MAX NJOBS RUN SSUSP USUSP RSV DISPATCH_WINDOW
ok 12.50 - 4 2 2 0 0 0 -

CURRENT LOAD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem slots
Total 0.9 0.0 0.0 1% 0.0 10 1 0 39G 0M 1.4G 2
Reserved 0.0 0.0 0.0 0% 0.0 0 0 0 0M 0M 1.2G -

LOAD THRESHOLD USED FOR SCHEDULING:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - 100M
loadStop - - - - - - - - - - 10M

CONFIGURED AFFINITY CPU LIST: all

[user001@chjtlnx2048 ~]$

0 reply Hide replies

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.

Please enter your email address

RELATED IDEAS

The function that can close host automatically by reserved memory