Canceling training run while running freezes app

[yrkim98]

Investiate + fix canceling a training run through the cancel button on the progress bar- this currently freezes the app.

[hughes036]

I am not able to replicate this on Mac.

[hughes036]

Will try on a Windows system over rdp

[yrkim98]

@thao-do I am not able to repro this on a windows instance or my windows desktop at home

[thao-do]

@yrkim98 this happened on my Macbook pro laptop when I tested with Chantelle; and happened again on one of my recent tests but I didn't note any other details down. I'll try it again today or tomorrow and if it's non-reproducible I guess we can close it.

[thao-do]

@yrkim98 I was able to reproduce this on my iMac pro desktop in addition to the Macbook Pro laptop. I have share my virtual env info in my smoke-test report form that @saeliddp is aware of. This potentially happens on Mac/CPU-only devices?

One additional detail (I'll add some screenshot later) is that I noticed in the Terminal window (where I run the virtual env + napari) showed the progress of the current epoch (and the current image that epoch was proceed through) staying the same during the freeze (even overnight) with no other error or updates. This freeze is caused by both clicking the "Cancel" button or "x" to close the progress dialog.