Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log instances of delays due to resource limitations and their causes #132

Merged
merged 5 commits into from
Apr 30, 2021
Merged

Log instances of delays due to resource limitations and their causes #132

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0fce189
Include job age any time that a plot is held off; Insert the last del…
13thProgression Apr 19, 2021
3e09838
Merge branch 'development' into features/Log_instances_of_delays_and_…
13thProgression Apr 21, 2021
4362b9c
Merge branch 'development' into features/Log_instances_of_delays_and_…
13thProgression Apr 25, 2021
7c391c7
Merge branch 'development' into features/Log_instances_of_delays_and_…
altendky Apr 30, 2021
dc866f8
Merge branch 'development' into features/Log_instances_of_delays_and_…
altendky Apr 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions src/plotman/interactive.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ def curses_main(stdscr):
pressed_key = '' # For debugging

archdir_freebytes = None
aging_reason = None

while True:

Expand All @@ -113,10 +114,16 @@ def curses_main(stdscr):
cfg.directories, cfg.scheduling, cfg.plotting
)
if (started):
if aging_reason is not None:
log.log(aging_reason)
aging_reason = None
log.log(msg)
plotting_status = '<just started job>'
jobs = Job.get_running_jobs(cfg.directories.log, cached_jobs=jobs)
else:
# If a plot is delayed for any reason other than stagger, log it
if msg.find("stagger") < 0:
aging_reason = msg
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So ... when we start a new plot job, we'll log the reason it had been delayed? That could be useful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has helped me with tuning my machines by suggesting what resources are lacking. The only downside that I have encountered is that the Log: count message no longer reflects the total number of jobs launched by plotman since delay messages are also counted.

plotting_status = msg

if archiving_configured:
Expand Down
4 changes: 2 additions & 2 deletions src/plotman/manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ def maybe_start_new_plot(dir_cfg, sched_cfg, plotting_cfg):
if (youngest_job_age < global_stagger):
wait_reason = 'stagger (%ds/%ds)' % (youngest_job_age, global_stagger)
elif len(jobs) >= sched_cfg.global_max_jobs:
wait_reason = 'max jobs (%d)' % sched_cfg.global_max_jobs
wait_reason = 'max jobs (%d) - (%ds/%ds)' % (sched_cfg.global_max_jobs, youngest_job_age, global_stagger)
else:
tmp_to_all_phases = [(d, job.job_phases_for_tmpdir(d, jobs)) for d in dir_cfg.tmp]
eligible = [ (d, phases) for (d, phases) in tmp_to_all_phases
Expand All @@ -86,7 +86,7 @@ def maybe_start_new_plot(dir_cfg, sched_cfg, plotting_cfg):
for (d, phases) in eligible ]

if not eligible:
wait_reason = 'no eligible tempdirs'
wait_reason = 'no eligible tempdirs (%ds/%ds)' % (youngest_job_age, global_stagger)
else:
# Plot to oldest tmpdir.
tmpdir = max(rankable, key=operator.itemgetter(1))[0]
Expand Down