Jobs that get into PSUSP state because they reached MAX_JOB_REQUEUE cannot be killed by an appropriate JOB_CONTROL mechanism

Answer from PMR:

For the PSUSP jobs JOB_CONTROL can not work on PSUSP job. We have checked the source code in these days and found that

1. LSF doesn't set any environment variable when a job reaches its requeue limit and be put into PSUSP status

2. When LSF set job status to PSUSP, the job state is changed directly, no post proc stage will be involved.

Therefore, we couldn't identify a job has reached its requeue limit outside LSF, and also even we could, we don't have a place to execute the bkill command since post exec won't be invoked.

Post comment

Guest

Reply
| Jan 15, 2018

The job controls and post-exec are executed by the sbatchd process on the execution host.

In the case of a job that cannot be dispatched, it is set to PSUSP by the mbatchd - it has never made it to an execution host, so it is impossible to run the job controls to be executed on the "execution host". Having the mbatchd execute this could potentially have significant performance side effects - e.g. if a lot of jobs were all put into psusp at the same time.

An alternative could be to introduce a new field in bjobs -o which would highlight the difference between jobs that are in psusp due to the user and jobs that are in psusp due to the scheduler. Would that address your issue?

0 reply Hide replies

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.

Please enter your email address

RELATED IDEAS

Jobs that get into PSUSP state because they reached MAX_JOB_REQUEUE cannot be killed by an appropriate JOB_CONTROL mechanism