Skip to content

Commit

Permalink
Training monitor enhancements (#691)
Browse files Browse the repository at this point in the history
* Training monitor enhancements
- Cleaned up imports
- Docstrings
- Now update based on time, not epochs
- Added markers for epoch-level losses
- Added best validation loss marker and text
- Reduced minimum possible y-axis value when log scaling
- Marker colors, alpha, sizes and line widths adjusted

* Move training monitor to gui submodule

* add metrics to training monitor title

* add mean time per epoch
* add ETA to finish next 10 epochs
* add plateau patience fraction (when in plateau)
* update dev_requirements to install version of click that does not break black

* Add code coverage

* add coverage for all lines within LossViewer.update_runtime()

Co-authored-by: roomrys <[email protected]>
  • Loading branch information
talmo and roomrys committed Apr 3, 2022
1 parent 7dc55a7 commit ab4f2a3
Show file tree
Hide file tree
Showing 6 changed files with 278 additions and 128 deletions.
3 changes: 2 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,8 @@ jobs:
python-version: 3.7
- name: Install Dependencies
run: |
pip install black==20.8b1
pip install click==8.0.4
pip install black==21.6b0
- name: Run Black
run: |
black --check sleap tests
Expand Down
3 changes: 2 additions & 1 deletion dev_requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,5 @@ twine==3.3.0
PyGithub
jupyterlab
jedi==0.17.2
ipykernel
ipykernel
click==8.0.4
4 changes: 2 additions & 2 deletions sleap/gui/learning/runners.py
Original file line number Diff line number Diff line change
Expand Up @@ -556,7 +556,7 @@ def run_gui_training(
trained_job_paths = dict()

if gui:
from sleap.nn.monitor import LossViewer
from sleap.gui.widgets.monitor import LossViewer
from sleap.gui.widgets.imagedir import QtImageDirectoryWidget

# open training monitor window
Expand Down Expand Up @@ -603,7 +603,7 @@ def run_gui_training(

if gui:
print("Resetting monitor window.")
win.reset(what=str(model_type))
win.reset(what=str(model_type), config=job)
win.setWindowTitle(f"Training Model - {str(model_type)}")
win.set_message(f"Preparing to run training...")
if save_viz:
Expand Down
Loading

0 comments on commit ab4f2a3

Please sign in to comment.