Add Configurable Recall-to-Forward Delay for Multi-Cluster Scheduling

In a Multi-Cluster environment, once a job is forwarded to a remote cluster, it effectively remains there until it is scheduled or completed. As a result, the submission cluster may later have available resources that the job is unable to utilize.

The existing parameters MC_FORWARD_DELAY and MAX_RSCHED_TIME provide partial control over forwarding behavior. However, MC_FORWARD_DELAY is applied only during the initial forwarding decision. After a job is recalled to the submission cluster due to MAX_RSCHED_TIME, it may be forwarded again without spending sufficient time in the submission cluster.

This behavior can lead to inefficient resource utilization, particularly in environments where clusters have different capacities and workloads.

We propose introducing a new configurable parameter that specifies how long a recalled job should remain in the submission cluster before it becomes eligible for forwarding again.

The intention is to allow jobs to cycle between the submission cluster and remote clusters in a controlled manner, rather than remaining primarily in the forwarded cluster. This would provide both the submission cluster and remote clusters with repeated opportunities to schedule the job as resource availability changes over time.

One possible implementation would be a queue-level parameter, such as RECALL_FORWARD_DELAY, which defines the amount of time a recalled job remains in the submission cluster before it can be forwarded again. Unlike MC_FORWARD_DELAY, this parameter would be applied each time a job is recalled by MAX_RSCHED_TIME, enabling the forwarding and recall cycle to repeat until the job is successfully scheduled.

The proposed workflow would be:

1. A job is submitted to the local cluster.

2. The local cluster attempts to schedule the job.

3. After MC_FORWARD_DELAY, the job is forwarded to a remote cluster.

4. If the job remains unscheduled for MAX_RSCHED_TIME, it is recalled to the submission cluster.

5. The job remains in the submission cluster for the duration specified by RECALL_FORWARD_DELAY.

6. If the job is still not scheduled, it becomes eligible for forwarding again.

7. The process repeats until the job is scheduled.

This enhancement would create a more balanced scheduling model between local and remote clusters, improve overall resource utilization, and allow jobs to benefit from newly available resources regardless of which cluster they become available in.