Summary
Add a scheduling/reservation mode in which the top-priority pending job (the
next job scheduled to run) reserves resources across ALL of its alternative
resource combinations simultaneously, rather than committing its reservation
to a single combination. The job dispatches on whichever combination becomes
fully available first, and the reservations held on all the other (non-chosen)
combinations are released immediately upon dispatch.
Current behavior
Many jobs are submitted with alternative resource requirements, e.g. a RES_REQ
containing multiple OR'd rusage strings:
rusage[<combo A>] || rusage[<combo B>] || rusage[<combo C>] || ...
Each combination represents an equally acceptable placement. With
RESOURCE_RESERVE enabled, when the top pending job cannot start immediately,
LSF accumulates a reservation against ONE selected combination. If the
resources for that particular combination are held by long-running or exclusive
jobs, the top job waits for that specific combination to free - even though a
different, equally valid combination frees up much sooner. The job effectively
becomes "locked" to a slow-to-free path, inflating its wait time while capacity
it could have used elsewhere sits idle or is consumed by lower-priority jobs.
Proposed enhancement
Introduce an opt-in reservation mode (e.g. a new queue-level parameter such as
RESERVE_ALL_ALTERNATIVES=Y, or an APS/scheduling policy flag) with the following
behavior for the head-of-queue job (or the top N eligible pending jobs):
Reserve resources against EVERY alternative resource combination in the
job's RES_REQ concurrently (a "multi-path" reservation), instead of a single
selected combination.
As soon as ANY one combination becomes fully satisfiable, dispatch the job
on that combination.
Immediately release the reservations held on all other combinations so they
are returned to the scheduler and become available to other jobs.
Bound the feature to the top N pending jobs (configurable) to cap the amount
of simultaneously reserved-but-idle capacity, and continue to honor
MAX_RESERVE_TIME.
Example use case (hardware emulation)
Emulation jobs (Cadence Palladium, Synopsys ZeBu) request a fixed number of
emulator units/clusters that can be placed in many equivalent ways across
racks/groups - so their RES_REQ commonly carries 10+ alternative rusage
combinations. Emulator jobs are exclusive and long-running. Today the top
pending emulation job frequently reserves one group/combination and waits
30-60+ minutes for it to free, while a different group opens up in minutes and
is taken by smaller, lower-priority jobs. Reserving across all combinations
would let the highest-priority job seize the first opening on ANY valid path,
cutting wait time dramatically and improving utilization of scarce, expensive
emulation hardware.
Business value / benefits
Reduces head-of-line blocking and wait time for the highest-priority job.
Prevents "reservation lock-in" to a single slow-to-free combination.
Improves throughput and utilization of scarce, specialized, or exclusive
resources (emulators, GPUs, license- or topology-constrained hosts) where a
job has many equivalent placements but each individual resource frees slowly.
Keeps priority ordering intact - only the top job(s) gain the multi-path
reservation, and reservations auto-release on dispatch.
Suggested safeguards / configuration
Opt-in per queue (default off) to preserve existing behavior.
Continue to allow backfill scheduling on reserved-but-not-yet-consumed
resources up to the reserving job's estimated start time, so utilization is
not lost while multiple paths are held.
Respect MAX_RESERVE_TIME and existing preemption/SLA-guarantee semantics.
Emit reservation/telemetry so admins can see which combination was chosen and
which were released.
