LSF does not immediately requeue work with a host becomes unavailable. There are many checks and parameters that control how long sbd, mbd and mlim will wait before declaring a host gone, and triggering requeue.

Admittedly this information is scattered throughout the documentation:

LSF does not immediately requeue the affected jobs - there are multiple steps that are taken to determine whether a host in unavailable (LSF daemons gone) or unreachable (network connectivity). Each of these has multiple timeouts and retries, and when these are exhausted there is a final time out before the job killed.

🖥️ How LSF Detects Host Failures: The Complete End‑to‑End Process (and Every Timeout That Matters)

When an LSF cluster is healthy, it feels almost invisible—jobs flow, hosts report load, and everything “just works.” But when a host becomes unreachable, the behaviour can feel mysterious: jobs flip to UNKNOWN, hosts bounce between UNREACH and UNAVAIL, and administrators scramble to understand what’s actually happening.

This blog demystifies the entire process. We’ll walk step‑by‑step through:

• How LSF monitors host health

• How it decides a host is unreachable vs unavailable

• What happens to running jobs

• Every timeout and heartbeat that influences the process

• How to tune these values safely

🧭 1. The Three Daemons That Matter Most

LSF’s host‑health logic depends on three core daemons:

• LIM — Load Information Manager

• sbatchd — Job execution daemon

• mbatchd — Central scheduler

Each daemon has its own communication pattern and timeout logic.

To understand host failure behaviour, you must understand LIM heartbeats, sbatchd RPC calls, and mbatchd retry logic.

🔌 2. Step‑by‑Step: How LSF Detects a Host Failure

Below is the exact sequence LSF follows when a host begins to fail.

Step 1 — LIM Heartbeats Begin Failing

LIMs exchange heartbeats every few seconds. If the master LIM stops hearing from a host’s LIM, it begins a retry cycle.

Influencing parameters:

• LSF_LIM_PING_INTERVAL

• LSF_LIM_TIMEOUT

If the timeout expires:

➡️ Host becomes UNAVAIL ➡️ LSF assumes the host is effectively down ➡️ Jobs on that host become UNKNOWN

This is the strongest failure state.

Step 2 — sbatchd RPC Calls Begin Failing

Even if LIM is alive, mbatchd must still communicate with sbatchd to:

• start jobs

• signal jobs

• query job status

If sbatchd stops responding:

➡️ Host becomes UNREACH ➡️ Jobs become UNKNOWN

This is a partial failure: the host is alive, but job control is broken.

Influencing parameters:

• LSF_SBD_TIMEOUT

• LSF_SBD_RETRY_INTERVAL

• LSF_SBD_RETRY_LIMIT

These determine how long LSF keeps trying before giving up.

Step 3 — Jobs Transition to UNKNOWN

Once a host is UNREACH or UNAVAIL:

• LSF cannot confirm job state

• Jobs are marked UNKNOWN

• Resources remain allocated

• Jobs continue running on the host (LSF does not kill them)

Influencing parameters:

• REMOVE_HUNG_JOBS_FOR (optional cleanup)

• WAIT_TIME inside REMOVE_HUNG_JOBS_FOR

If cleanup is enabled:

➡️ After WAIT_TIME, LSF kills the job and frees resources.

If not:

➡️ Jobs remain UNKNOWN indefinitely.

Step 4 — LSF Continues Retrying Communication

LSF never gives up permanently.

• LIM keeps sending heartbeats

• mbatchd keeps retrying sbatchd RPCs

• sbatchd attempts to reconnect to mbatchd

If communication returns:

➡️ Host returns to ok ➡️ Jobs return to RUN or EXIT depending on real state

⚙️ All Tuning Parameters That Influence Host Failure Behaviour

Below is a complete list of every parameter that affects host reachability, job state transitions, and cleanup.

🟦 1. LIM Heartbeat & Availability Parameters

These determine when a host becomes UNAVAIL.

Recommended ranges:

• Stable networks: 20–40s

• Cloud/variable networks: 40–90s

🟩 2. sbatchd RPC Timeout Parameters

These determine when a host becomes UNREACH.

Recommended ranges:

• Stable networks: retry limit 5–8

• Cloud/variable networks: retry limit 8–12

🟥 3. Hung‑Job Cleanup Parameters

These determine how long LSF waits before killing jobs on unreachable hosts.

Recommended ranges:

• HPC/scientific workloads: 30–120 minutes

• Shared clusters: 10–20 minutes

🧠 Putting It All Together: The Full Lifecycle

Here’s the complete chain of events, with the parameters that influence each step:

1. Network issue begins

   • LIM heartbeats fail → LSF_LIM_TIMEOUT

   • sbatchd RPCs fail → LSF_SBD_TIMEOUT

2. LSF retries communication

   • LIM retries → LSF_LIM_PING_INTERVAL

   • sbatchd retries → LSF_SBD_RETRY_INTERVAL, LSF_SBD_RETRY_LIMIT

3. Host marked UNREACH or UNAVAIL

   • sbatchd failure → UNREACH

   • LIM failure → UNAVAIL

4. Jobs marked UNKNOWN

   • No parameters influence this step

5. Optional cleanup

   • REMOVE_HUNG_JOBS_FOR

   • WAIT_TIME

6. Host recovers

   • LIM heartbeats resume

   • sbatchd reconnects

   • Jobs return to RUN/EXIT

Rerunnable jobs are handled differently, but only after LSF decides the host is UNREACH or UNAVAIL. The detection process and timeouts do not change, but the job‑handling logic does.

Below is the full, admin‑level explanation.

🎯 Core takeaway

LSF does not use different heartbeat or timeout logic for rerunnable jobs. However, once a host is unreachable, LSF may automatically requeue a rerunnable job instead of leaving it in UNKNOWN — but only if certain conditions are met.

This is where rerunnable jobs diverge from normal jobs.

🧩 What stays the same (no difference for rerunnable jobs)

The following mechanisms behave identically whether a job is rerunnable or not:

• LIM heartbeats → host UNAVAIL

• sbatchd RPC failures → host UNREACH

• mbatchd retry logic

• Host state transitions

• Timeouts and heartbeat intervals

• UNKNOWN job state assignment

All of these are controlled by the same parameters:

• LSF_LIM_TIMEOUT

• LSF_LIM_PING_INTERVAL

• LSF_SBD_TIMEOUT

• LSF_SBD_RETRY_INTERVAL

• LSF_SBD_RETRY_LIMIT

Rerunnable jobs do not influence these.

🔀 What does change: job handling after host failure

Once a host becomes UNREACH or UNAVAIL, LSF evaluates the job’s rerunnable flag.

✔️ If the job is rerunnable (bsub -r or #BSUB -r)

LSF may automatically:

• requeue the job, or

• restart it on another host, or

• kill it and requeue it (depending on configuration)

This avoids leaving the job stuck in UNKNOWN.

Influencing parameters:

• JOB_RERUNNABLE (queue or job level)

• REQUEUE_EXIT_VALUES

• RESUME_COND / STOP_COND

• REMOVE_HUNG_JOBS_FOR (if host unreachable)

Behaviour summary:

❌ If the job is not rerunnable

LSF cannot requeue it automatically.

The job will:

• enter UNKNOWN

• remain UNKNOWN until the host returns

• optionally be killed by hung‑job cleanup

This is why rerunnable jobs are strongly recommended for:

• short jobs

• stateless jobs

• cloud or elastic clusters

• preemptible nodes

🧠 Important nuance: rerunnable ≠ checkpointable

A rerunnable job is killed and restarted from scratch. It does not preserve state unless you implement your own checkpointing.

🧯 Interaction with hung‑job cleanup

If REMOVE_HUNG_JOBS_FOR=host_unreach is enabled:

• rerunnable jobs → killed and requeued

• non‑rerunnable jobs → killed and marked EXIT

This is one of the biggest behavioural differences.

🧭 End‑to‑end flow comparison

🟦 Non‑rerunnable job

1. Host becomes UNREACH/UNAVAIL

2. Job → UNKNOWN

3. LSF waits indefinitely

4. Optional: hung‑job cleanup kills job

🟩 Rerunnable job

1. Host becomes UNREACH/UNAVAIL

2. Job → UNKNOWN

3. LSF evaluates rerunnable flag

4. Job is requeued automatically (if allowed)

5. Job restarts on a new host