Enhance Processor Affinity Management in Platform LSF To Address All Classical Execution Configurations for HPC Workloads

On behalf of our client, I have investigated the proper way to set the Platform LSF Affinity String ("-R affinity") in order to handle the standard execution configurations of their scientific applications.

By "execution configurations", I mean the combination of the following elements:
- SMT mode (off / 2 / 4 / 8)
- # tasks per node
- # threads per task
- # GPUs involved, either in shared mode (through MPS) or in exclusive mode

I have been able to come up with a valid affinity string for some of those execution configurations, but some execution configurations do not seem to be supported by IBM Platform LSF, according to the final answer I could get from the Platform LSF support to the Service Request I have opened (93180,661,706).

For instance (the following list is not supposed to be comprehensive AT ALL), there doesn't seem to be currently any way in Platform LSF to address the following two configurations:

1) 2 MPI tasks, 10 threads / task, with one task and its associated threads residing on each socket, running on physical cores only (not on logical cores)

2) 10 MPI tasks, 2 threads / task, with all tasks / threads running on physical cores (not on logical cores)

Those execution configurations are rather classical in HPC, and need to be properly handled by the workload scheduler.

More generally speaking, the affinity management in Platform LSF must at the same time:

1) Be properly handled by the workload scheduler without forcing the user to cope with manual operations in his submission script (like setting environment variables, building rankfile...). Things should be mostly transparent to the user, whatever the MPI library used is (PE RTE, Open MPI, Spectrum MPI).

2) Guarantee optimal performance of the workload through automatic setting - so that standard users in a production environment don't need to bother with affinity for their jobs.
Platform LSF should be able to deduce a default optimal processor affinity based on all the parameters of the execution configuration (SMT mode, # tasks, # threads per task...). This would guarantee that a production infrastructure is optimally exploited.

3) Give a high level of control to the user - so that advanced users can achieve any complex processor affinity they might want to experiment - again without having to rely on manual operations (environment variables, rankfile).
The affinity string (or any other keyword) should allow enough customization to specify complex processor affinity.