Using the job info option to jobsub_submit

Using the --job-info option to jobsub_submit

The --job-info option (or JOBSUB_EXTRA_JOB_INFO environemnt variable) is an easy way to add logging / record keeping to your job submission workflow. This document will give some examples of how it can be used.

A simple submission log

Lets say you wanted to keep a log of all your job submissions for a given project. You could make a simple shell script myproject_log_info:

#!/bin/sh
{
  echo date: $(date)
  echo jobsub_job_id: $1
  shift 
  echo jobsub_command: $*
} >> $HOME/.myproject_job_log

(remember to chmod +x myproject_log_info so it is executable)

This will put a 3 line entry into a logfile in your home directory each time it is called. Now just add --job-info myproject_log_info to your job submissions for this project, and they will all be logged in the log file. Or if you have an existing script framework you are using, you can instead just

export JOBSUB_EXTRA_JOB_INFO=myproject_log_info

before submitting the jobs on that project. Note that you can use --job-info multiple times, and each script will be called for each submission.

[mengel@fifeutilgpvm01 jobsub_lite]$ jobsub_submit -G fermilab --job-info myproject_job_info file:///bin/true
Transferring files to web sandbox...
Copying file:///home/mengel/.cache/jobsub_lite/js_2023_04_18_130702_091aef49-9f96-447a-80c2-242edd5cc690/simple.cmd   [DONE]  after 0s            
Copying file:///home/mengel/.cache/jobsub_lite/js_2023_04_18_130702_091aef49-9f96-447a-80c2-242edd5cc690/true   [DONE]  after 0s                  
Copying file:///home/mengel/.cache/jobsub_lite/js_2023_04_18_130702_091aef49-9f96-447a-80c2-242edd5cc690/simple.sh   [DONE]  after 0s             
Submitting job(s).
1 job(s) submitted to cluster 60893570.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Use job id [email protected] to retrieve output
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


[mengel@fifeutilgpvm01 jobsub_lite]$ jobsub_submit -G fermilab --job-info myproject_job_info file:///bin/false
Transferring files to web sandbox...
Copying file:///home/mengel/.cache/jobsub_lite/js_2023_04_18_130713_30b9170d-ac3a-4829-852f-4aacf08c6868/simple.cmd   [DONE]  after 0s            
Copying file:///home/mengel/.cache/jobsub_lite/js_2023_04_18_130713_30b9170d-ac3a-4829-852f-4aacf08c6868/false   [DONE]  after 0s                 
Copying file:///home/mengel/.cache/jobsub_lite/js_2023_04_18_130713_30b9170d-ac3a-4829-852f-4aacf08c6868/simple.sh   [DONE]  after 0s             
Submitting job(s).
1 job(s) submitted to cluster 60893571.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Use job id [email protected] to retrieve output
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


[mengel@fifeutilgpvm01 jobsub_lite]$ cat $HOME/.myproject_job_log 
date: Tue Apr 18 13:07:06 CDT 2023
jobsub_job_id: [email protected]
jobsub_command: ['/opt/jobsub_lite/bin/jobsub_submit', '-G', 'fermilab', '--job-info', 'myproject_job_info', 'file:///bin/true']
date: Tue Apr 18 13:07:17 CDT 2023
jobsub_job_id: [email protected]
jobsub_command: ['/opt/jobsub_lite/bin/jobsub_submit', '-G', 'fermilab', '--job-info', 'myproject_job_info', 'file:///bin/false']