RTM: GPU Utilization and GPU Mem Utilization graphs for jobs based on the Host level graphs

Current state challenges:

While there are per job GPU graphs they require DCGM and are restricted to only exclusive process gpu jobs.  DCGM adds complexity and precludes a number of GPUs particularly ones that might be considered gamer cards as they are incompatible.  While Job Exclusive mode is reasonable exclusive process mode makes it incompatible with a large swath of processing.

The new 8 gpu host graphs for utilization and gpu memory utilization are great but when there are 8 different users with jobs on the same node there can be graph lines hidden behind each other and it does require the user to look up the GPU their job was assigned from one page then go to the host page and find that number and pick out the correctly colored line on the graphs.

Request:

The option to generate the per job GPU graphs using the assigned gpus for the job and the host level gpu statistics that are collected.  This eliminates the need for DCGM and the restriction to exclusive process jobs.  It does still require the gpus to be schedule job exclusively but that is less restrictive than the current requirements.  Removing the requirement for DCGM also opens the supported GPUs to any that host level statistics are able to be captured from.

Bonus Ask:

If these graphs that would be effectively the same as the host level graphs just filtered to only the gpus assigned to the job could appear in the Job Graphs tab and under the Graph Type: Job Level Graphs rather than on their own page.  The reason for this ask is being able to view the cpu, memory, gpu, and gpu memory graphs all at the same time is helpful for the end users to visualy identify if there is a particular bottleneck or resource constraint being caused by one of those things.   I understand that the current job level gpu graphs are complex and numerous enough that they need to be on their own page because each metric has it's own graph and then every gpu for the job has one of each of those graphs.  But this ask would be for just 2 graphs no matter how many gpus were requested in the job.