Fix handling completed job with expired result when work horse dies

[fancyweb]

Since #2039, Job.get_status() doesn't return None but raises InvalidJobOperation when refresh=True.

This change was not handled properly in Worker.monitor_work_horse().

I stumbled upon this with a use-case where a job with result_ttl=0 succeeds but the work horse doesn't exit with 0 (because it's wrapped). Job.get_status() ends up being called and raises an uncaught error.

There's no existing test for the behavior and I wasn't able to produce one.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.46%. Comparing base (2de9491) to head (b82912d).
Report is 106 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2154      +/-   ##
==========================================
- Coverage   93.61%   93.46%   -0.16%     
==========================================
  Files          28       30       +2     
  Lines        3760     4114     +354     
==========================================
+ Hits         3520     3845     +325     
- Misses        240      269      +29     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fancyweb]

Ref DataDog/dd-trace-py#11512

Here's the scenario as I understood it:

1. A job with result_ttl=0 is enqueued
2. This job is executed and completed
3. DataDog's Worker.perform_job() wrapper calls get_result(), raises the new error, and makes the horse process fails
4. rq calls get_result() again and raises the new error too 😅

[selwin]

Mind adding a test for this?

[fancyweb]

@selwin After looking at it again, I was able to add a test demonstrating the issue.

[selwin]

@fancyweb do you mind checking why the tests failed on Python 3.9?