About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
One solution is to use the "WATCHDOG" parameter in LSF application profile.
LSF runs this watchdog script regularly and passes $LSB_JOB_PIDS to the script. The script can go through this pid list and check if job res is in it. If no, kill them all.
https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=health-monitor-applications-by-using-external-scripts
Given the script above, it is unlikely that this will be make it to the top of the priority list in the near future. The idea will be left open and may be reconsidered if there is additional interest.
There is a "zapit" utility that is included with LSF that can be used to clean up any processes that a job has created and still has lying around.
It's typically added as the last line of the job script