CPU and Memory information

[zhibinuhn]

This is a very useful project and it helps users to receive detailed information when their jobs finish. It will be useful if the email can include cpu and memory efficiency like the walltime. It is not difficult to use "seff" like

Get additional info from seff.

cmd = f"{seff_exe} {job_id}"
rc, stdout, stderr = run_command(cmd)
if rc == 0:
seff_dict = {}
for i in stdout.split("\n"):
x = i.split(": ", 1)
if len(x) == 2:
seff_dict[x[0]] = x[1]
job.cpu_efficiency = seff_dict['CPU Efficiency']
job.cpu_utilized = seff_dict['CPU Utilized']
job.mem_efficiency = seff_dict['Memory Efficiency']
job.mem_utilized = seff_dict['Memory Utilized']

[neilmunday]

Hi @zhibinuhn,

Thank you for the suggestion. As not everyone will have seff installed, I think it might make sense to grab the information from sacct instead.

[neilmunday]

I have created issue #29 to track this.

[neilmunday]

I have implemented the request in the 2.6 branch. If you would like to try it out, you can do as follows:

git clone https://github.com/neilmunday/slurm-mail
git checkout 2.6

From here now copy the contents of the bin and conf.d directories to your Slurm-Mail installation directory.

[zhibinuhn]

Hi Neil, Thanks so much for implementing the new features. I have updated my local version. Best, Zhibin From: Neil Munday ***@***.***> Date: Tuesday, April 12, 2022 at 4:03 PM To: neilmunday/slurm-mail ***@***.***> Cc: zhibinuhn ***@***.***>, Mention ***@***.***> Subject: Re: [neilmunday/slurm-mail] CPU and Memory information (Discussion #28) I have implemented the request in the 2.6 branch. If you would like to try it out, you can do as follows: git clone https://github.com/neilmunday/slurm-mail git checkout 2.6 — Reply to this email directly, view it on GitHub<#28 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFR6AIYOYGEZZQRXTRMR75TVEXJH7ANCNFSM5RVQ4UIQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[neilmunday]

@zhibinuhn no problem. I'd be interested in any feedback (good or bad) as I've only been able to test using a VM set-up.

[zhibinuhn]

If a job is cancelled by the user, I get the following error:

2022/04/18 19:38:03:ERROR: float division by zero
Traceback (most recent call last):
File "/opt/slurm-mail/bin/slurm-send-mail.py", line 682, in
process_spool_file(f, int(fields[0]), fields[1])
File "/opt/slurm-mail/bin/slurm-send-mail.py", line 432, in process_spool_file
job.save()
File "/opt/slurm-mail/bin/slurm-send-mail.py", line 237, in save
float(self.used_cpu_usec) / float(self.__cpu_time_usec)
ZeroDivisionError: float division by zero

Here is the sacct output:
#sacct -j 5176881 -P -n --fields='JobId,User,Group,Partition,Start,End,State,ReqMem,MaxRSS,NCPUS,TotalCPU,NNodes,WorkDir,Elapsed,ExitCode,Comment,Cluster,NodeList,TimeLimit,TimelimitRaw,JobIdRaw,JobName'
5176881|kuichen|pughlab|veryhimem|2022-04-18T00:58:18|2022-04-18T00:58:18|CANCELLED by 14111|180G||30|00:00:00|1|/cluster/projects/pughlab/data/HCC_ctDNA_project/Kui/fastq2bam|00:00:00|0:0||h4huhn|None assigned|4-00:00:00|5760|5176881|T786

[neilmunday]

Apologies, this bug was fixed recently fixed with a commit to the 2.6 branch. Can you try updating your checked out copy please?

cd slurm-mail
git fetch
git pull

Many thanks.

[zhibinuhn]

Thanks, it is working now.

[neilmunday]

Excellent, good to hear - thanks for testing :-)

[zhibinuhn]

I got another error:
Traceback (most recent call last):
File "/opt/slurm-mail/bin/slurm-send-mail.py", line 689, in
process_spool_file(f, int(fields[0]), fields[1])
File "/opt/slurm-mail/bin/slurm-send-mail.py", line 380, in process_spool_file
job.max_rss_str = sacct_dict['MaxRSS']
File "/opt/slurm-mail/bin/slurm-send-mail.py", line 141, in max_rss_str
self.max_rss = get_kbytes_from_str(value)
File "/opt/slurm-mail/bin/slurm-send-mail.py", line 290, in get_kbytes_from_str
kbytes = int(value[:-1])
ValueError: invalid literal for int() with base 10: '27896.50'

MaxRSS could be decimal and function get_kbytes_from_str only handles int.

#sacct -j 5188910 -P -n --fields='JobId,User,Group,Partition,Start,End,State,ReqMem,MaxRSS,NCPUS,TotalCPU,NNodes,WorkDir,Elapsed,ExitCode,Comment,Cluster,NodeList,TimeLimit,TimelimitRaw,JobIdRaw,JobName'
5188910|tonychen|hopegroup|himem|2022-04-22T16:51:23|2022-04-22T16:54:34|COMPLETED|60G||18|00:37.597|1|/cluster/home/tonychen/jobs|00:03:11|0:0||h4huhn|node77|05:00:00|300|5188910|gillespie_STAR_pass2
5188910.batch||||2022-04-22T16:51:23|2022-04-22T16:54:34|COMPLETED||27896.50M|18|00:37.595|1||00:03:11|0:0||h4huhn|node77|||5188910.batch|batch
5188910.extern||||2022-04-22T16:51:23|2022-04-22T16:54:38|COMPLETED||1120K|18|00:00.001|1||00:03:15|0:0||h4huhn|node77|||5188910.extern|extern

[neilmunday]

Thanks the feedback. I have corrected the function in commit ea1679c

Please run git fetch; git pull in your clone of the 2.6 branch to get the fix.

[neilmunday]

Due to the large number of changes to Slurm-Mail for the 2.6 version and due to the new spool file format I have created, I have renamed the 2.6 branch to 3.0.

[neilmunday]

Requested features were published in version 3.0.