We are experiencing problem with displaying the correct number of AMD GPUs which are assigned to a LSF job.
Even though we request 1 GPU and LSF sets correctly the ROCR_VISIBLE_DEVICES=0 the command amd-smi displays all of them.
We are able to use only the GPU Flag like this -gpu "num=1:gvendor=amd:mps=no:mode=shared" in shared mode and when we change it to exclusive the LSF system throws an error.
Our Understanding is that there is some kind of a problem with the cgroups since LSF is not able to isolate the given GPU index and allocate it only for one job when requested. Also bhost -l <> doesn't display the correct info as rocm-smi
Example:
[root@rng-dl01-mgmt1 rbadmin_app2]# bhosts -gpu -l rng-dl01-w26a01
HOST: rng-dl01-w26a01
NGPUS NGPUS_SHARED_AVAIL NGPUS_EXCLUSIVE_AVAIL
8 8 8
STATIC ATTRIBUTES
GPU_ID MODEL MTOTAL FACTOR SOCKET VENDOR MIG NVLINK/XGMI
0 AMDInstinctMI355ACOAM 287.9G 0.0 0 AMD N -/N/N/N/N/N/N/N
1 AMDInstinctMI355ACOAM 287.9G 0.0 0 AMD N N/-/N/N/N/N/N/N
2 AMDInstinctMI355ACOAM 287.9G 0.0 0 AMD N N/N/-/N/N/N/N/N
3 AMDInstinctMI355ACOAM 287.9G 0.0 0 AMD N N/N/N/-/N/N/N/N
4 AMDInstinctMI355ACOAM 287.9G 0.0 0 AMD N N/N/N/N/-/N/N/N
5 AMDInstinctMI355ACOAM 287.9G 0.0 0 AMD N N/N/N/N/N/-/N/N
6 AMDInstinctMI355ACOAM 287.9G 0.0 0 AMD N N/N/N/N/N/N/-/N
7 AMDInstinctMI355ACOAM 287.9G 0.0 0 AMD N N/N/N/N/N/N/N/-
DYNAMIC ATTRIBUTES
GPU_ID MODE MUSED MRSV TEMP ECC UT MUT PSTATE STATUS ERROR
0 SHARED 284M 0M 1518782639C 0 0% 0% 0 ok -
1 SHARED 284M 0M 1518782639C 0 0% 0% 0 ok -
2 SHARED 284M 0M 1518782639C 0 0% 0% 0 ok -
3 SHARED 284M 0M 1518782639C 0 0% 0% 0 ok -
4 SHARED 284M 0M 1518782639C 0 0% 0% 0 ok -
5 SHARED 284M 0M 1518782639C 0 0% 0% 0 ok -
6 SHARED 284M 0M 1518782639C 0 0% 0% 0 ok -
7 SHARED 468M 0M 1518782639C 0 0% 0% 0 ok -
GPU JOB INFORMATION
GPU_ID JEXCL RUNJOBIDS SUSPJOBIDS RSVJOBIDS GI_PLACEMENT/SIZE CI_PLACEMENT/SIZE
0 - - - - - -
1 - - - - - -
2 - - - - - -
3 - - - - - -
4 - - - - - -
5 - - - - - -
6 - - - - - -
7 - - - - - -
Even though rocm-smi show this:
============================================ ROCm System Management Interface ============================================
====================================================== Concise Info ======================================================
Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU%
(DID, GUID) (Junction) (Socket) (Mem, Compute, ID)
==========================================================================================================================
0 2 0x75a3, 56721 62.0°C 262.0W NPS1, QPX, 0 94Mhz 2000Mhz 0% auto 1400.0W 0% 0%
1 3 0x75a3, 26000 N/A N/A N/A, N/A, 1 N/A N/A 0% n/a N/A 0% N/A
2 4 0x75a3, 60816 N/A N/A N/A, N/A, 2 N/A N/A 0% n/a N/A 0% N/A
3 5 0x75a3, 21905 N/A N/A N/A, N/A, 3 N/A N/A 0% n/a N/A 0% N/A
4 6 0x75a3, 4508 51.0°C 265.0W NPS1, QPX, 0 94Mhz 2000Mhz 0% auto 1400.0W 0% 0%
5 7 0x75a3, 43421 N/A N/A N/A, N/A, 1 N/A N/A 0% n/a N/A 0% N/A
6 8 0x75a3, 8605 N/A N/A N/A, N/A, 2 N/A N/A 0% n/a N/A 0% N/A
7 9 0x75a3, 39324 N/A N/A N/A, N/A, 3 N/A N/A 0% n/a N/A 0% N/A
8 10 0x75a3, 62905 62.0°C 272.0W NPS1, QPX, 0 94Mhz 2000Mhz 0% auto 1400.0W 0% 0%
9 11 0x75a3, 19896 N/A N/A N/A, N/A, 1 N/A N/A 0% n/a N/A 0% N/A
10 12 0x75a3, 50616 N/A N/A N/A, N/A, 2 N/A N/A 0% n/a N/A 0% N/A
11 13 0x75a3, 32185 N/A N/A N/A, N/A, 3 N/A N/A 0% n/a N/A 0% N/A
12 14 0x75a3, 14772 51.0°C 259.0W NPS1, QPX, 0 94Mhz 2000Mhz 0% auto 1400.0W 0% 0%
13 15 0x75a3, 33205 N/A N/A N/A, N/A, 1 N/A N/A 0% n/a N/A 0% N/A
14 16 0x75a3, 2485 N/A N/A N/A, N/A, 2 N/A N/A 0% n/a N/A 0% N/A
15 17 0x75a3, 45492 N/A N/A N/A, N/A, 3 N/A N/A 0% n/a N/A 0% N/A
16 18 0x75a3, 32240 52.0°C 274.0W NPS1, QPX, 0 94Mhz 2000Mhz 0% auto 1400.0W 0% 0%
17 19 0x75a3, 50673 N/A N/A N/A, N/A, 1 N/A N/A 0% n/a N/A 0% N/A
18 20 0x75a3, 19953 N/A N/A N/A, N/A, 2 N/A N/A 0% n/a N/A 0% N/A
19 21 0x75a3, 62960 N/A N/A N/A, N/A, 3 N/A N/A 0% n/a N/A 0% N/A
20 22 0x75a3, 45565 62.0°C 278.0W NPS1, QPX, 0 94Mhz 2000Mhz 0% auto 1400.0W 0% 0%
21 23 0x75a3, 2556 N/A N/A N/A, N/A, 1 N/A N/A 0% n/a N/A 0% N/A
22 24 0x75a3, 33276 N/A N/A N/A, N/A, 2 N/A N/A 0% n/a N/A 0% N/A
23 25 0x75a3, 14845 N/A N/A N/A, N/A, 3 N/A N/A 0% n/a N/A 0% N/A
24 26 0x75a3, 21976 62.0°C 266.0W NPS1, QPX, 0 95Mhz 2000Mhz 0% auto 1400.0W 0% 0%
25 27 0x75a3, 60889 N/A N/A N/A, N/A, 1 N/A N/A 0% n/a N/A 0% N/A
26 28 0x75a3, 26073 N/A N/A N/A, N/A, 2 N/A N/A 0% n/a N/A 0% N/A
27 29 0x75a3, 56792 N/A N/A N/A, N/A, 3 N/A N/A 0% n/a N/A 0% N/A
28 30 0x75a3, 39381 55.0°C 274.0W NPS1, QPX, 0 95Mhz 2000Mhz 0% auto 1400.0W 0% 0%
29 31 0x75a3, 8660 N/A N/A N/A, N/A, 1 N/A N/A 0% n/a N/A 1% N/A
30 32 0x75a3, 43476 N/A N/A N/A, N/A, 2 N/A N/A 0% n/a N/A 0% N/A
31 33 0x75a3, 4565 N/A N/A N/A, N/A, 3 N/A N/A 0% n/a N/A 0% N/A
==========================================================================================================================
It looks like this capability is handled by a different low level API - hence why LSF is just seeing the physical GPU's
We'll action this as soon as resource is available, probably by the end of June subject to availability of hardware.