Actually cancel running jobs on worker shutdown to prevent blocking shutdown

[rtibbles]

Summary

• The worker code was only calling cancel on the stored future, but not interacting with the actual task cancellation mechanisms that are required to interrupt threads
• Marks any running jobs as canceling, and lets the existing cancellation handling logic finish up
• In the course of testing this, I ran into a couple of edge cases in our zeroconf handling where things would get cleaned up before certain code executed, so I added some defensive programming checks to handle these without errors

References

Fixes #10249

Reviewer guidance

Start then immediately stop the devserver - observe that it actually finishes up fairly quickly now rather than stalling for a while.

For a more acid test, run the server, start a long running content import task and then shut it down - it should still shut down relatively quickly.

---

Testing checklist

• Contributor has fully tested the PR manually
• If there are any front-end changes, before/after screenshots are included
• Critical user journeys are covered by Gherkin stories
• Critical and brittle code paths are covered by unit tests

PR process

• PR has the correct target branch and milestone
• PR has 'needs review' or 'work-in-progress' label
• If PR is ready for review, a reviewer has been added. (Don't use 'Assignees')
• If this is an important user-facing change, PR or related issue has a 'changelog' label
• If this includes an internal dependency change, a link to the diff is provided

Reviewer checklist

• Automated test coverage is satisfactory
• PR is fully functional
• PR has been tested for accessibility regressions
• External dependency files were updated if necessary (yarn and pip)
• Documentation is updated
• Contributor is in AUTHORS.md

[github-actions]

Build Artifacts

Asset type	Download link
PEX file	kolibri-0.16.0b2.dev0_git.7.g079dc058.pex
Windows Installer (EXE)	kolibri-0.16.0b2.dev0+git.7.g079dc058-unsigned.exe
Debian Package	kolibri_0.16.0b2.dev0+git.7.g079dc058-0ubuntu1_all.deb
Mac Installer (DMG)	kolibri-0.16.0b2.dev0+git.7.g079dc058-0.3.0.dmg
TAR file	kolibri-0.16.0b2.dev0+git.7.g079dc058.tar.gz
WHL file	kolibri-0.16.0b2.dev0+git.7.g079dc058-py2.py3-none-any.whl

[dbnicholson]

I gave this a spin locally and it worked nicely:

DEBUG    2023-08-18 10:18:26,748 JOBCHECKER shut down event received; closing.
INFO     2023-08-18 10:18:26,752 Canceling job id 88bbb5b183b342f7a35728428c3d75e2.
DEBUG    2023-08-18 10:18:26,820 https://studio.learningequality.org:443 "HEAD /content/storage/a/6/a60a1eec38d8a5ca61de29074662a879.png HTTP/1.1" 200 0
DEBUG    2023-08-18 10:18:26,910 https://studio.learningequality.org:443 "GET /content/storage/a/6/a60a1eec38d8a5ca61de29074662a879.png HTTP/1.1" 206 113392
INFO     2023-08-18 10:18:26,930 Canceling import: https://studio.learningequality.org/content/storage/a/6/a60a1eec38d8a5ca61de29074662a879.png
INFO     2023-08-18 10:18:26,933 Canceling import: https://studio.learningequality.org/content/storage/a/6/a60a1eec38d8a5ca61de29074662a879.png
DEBUG    2023-08-18 10:18:27,096 https://studio.learningequality.org:443 "HEAD /content/storage/d/7/d705f01ec547a4cc209f0dd167c2c20a.png HTTP/1.1" 200 0
DEBUG    2023-08-18 10:18:27,177 https://studio.learningequality.org:443 "GET /content/storage/d/7/d705f01ec547a4cc209f0dd167c2c20a.png HTTP/1.1" 206 100840
INFO     2023-08-18 10:18:27,446 Canceling import: https://studio.learningequality.org/content/storage/d/7/d705f01ec547a4cc209f0dd167c2c20a.png
INFO     2023-08-18 10:18:27,447 Canceling import: https://studio.learningequality.org/content/storage/d/7/d705f01ec547a4cc209f0dd167c2c20a.png
INFO     2023-08-18 10:18:27,488 Setting availability to True of 17 LocalFile objects based on passed in checksums
INFO     2023-08-18 10:18:27,509 Setting availability of non-topic ContentNode objects based on LocalFile availability in 1 batches of 10000
INFO     2023-08-18 10:18:27,517 Annotating ContentNode objects with children for 2 levels
INFO     2023-08-18 10:18:27,520 Annotating ContentNode objects with children for level 2
INFO     2023-08-18 10:18:27,524 Annotating ContentNode objects with children for level 1
DEBUG    2023-08-18 10:18:27,526 Recursive topic tree annotation took 0 seconds
INFO     2023-08-18 10:18:27,555 Joining 'ProcessControlPlugin'
INFO     2023-08-18 10:18:28,325 Stopped thread 'ProcessControlPlugin'.
INFO     2023-08-18 10:18:28,326 Bus state: IDLE
INFO     2023-08-18 10:18:28,327 Bus state: EXIT
INFO     2023-08-18 10:18:28,337 Waiting for child threads to terminate...
INFO     2023-08-18 10:18:28,338 Bus state: EXITED

[vkWeb]

These code changes were very much needed! 💯

Checked manually, working alright, green tick from my side ✔️

[bjester]

Thanks @vkWeb! I concur