Job stuck in Cancelling state

[hpapazov]

Bug summary

While testing Prefect Worker with prefect-kubernetes 0.5.3, I observed a change in how the package handles job Cancellation. Once you cancel the job via the UI, the job gets stuck in Cancelling state and never transitions to Cancelled state. I tested the same steps with prefect-kubernetes 0.5.0 and I did not observed the above behavior. The job transiting from Cancelling to Cancelled state.

Version info

Prefect packages used when job gets stuck in Cancelling state:
- prefect-client==3.0.4
- prefect-kubernetes==0.5.3


Prefect packages used when job transitions to Cancelled state:
- prefect-client==3.0.4
- prefect-kubernetes==0.5.0

Additional context

To reproduce the issue:

1. Start a job
2. Cancel vi UI
3. Observe the job status remaining "Cancelling" and not transitioning to "Cancelled"

Logs for a job running against prefect-kubernetes==0.5.3 that gets stuck in Cancelling state

Logs for a job running against prefect-kubernetes==0.5.0 that gets transitioned to Cancelled sate

[haddon-korinek-ent]

Also ran into this recently and saw it was deprecated here: #14611

The recommended approach now is to execute your remote flow runs using a Runner instead of calling the engine entrypoint directly.