Task time limit

[scttnlsn]

I'd like to add a time limit to my tasks such that they're terminated with a failure if the runtime of the task exceeds some threshold. It seems like that's currently left to the task to implement (e.g. check the task's own runtime inside of a loop and raise an exception if the threshold was exceeded). Is that correct? I didn't see an alternative to configure this in Procrastinate directly.

For tasks that call into existing functions that may take significant time to complete (e.g. where I don't have direct access to the potentially long-running code) I suppose my only option is to spawn a subprocess (to run the compute intensive work) and then kill it from the parent if the time limit is reached. Would doing so interfere with Procrastinate itself?

[ewjoachim]

If the task is async, that's supposedly easily achievable with asyncio.wait_for.
If the task is sync, it's going to be more complicated. From what I can say, the 2 options are the subprocess (as you mentionned) or a signal handler on SIGALARM (and I have no idea whether this plays nicely with threads, given sync code is executed in threads.)

[ewjoachim]

Note that of the 3 methods specified, none would have you write the loop yourself for thhe timeout:

• async: wait_for has a timeout argument
• sync subprocess: communicate (and thus run) both have a timeout argument
• sync signals: alarm takes a number of seconds too.

Note: the more I research this the less confident I am that signals can be of any help here, due to being executed on the main thread.

[TkTech]

Sync is a process pool isn't it? You can see how I did it here: https://github.com/TkTech/chancy/blob/main/chancy/executors/process.py#L102

Signals send to the main thread of the process, so if you setup the signal in the process pool it works fine. I don't use alarm() and emulate it with a thread because alarm() isn't platform portable.

[ewjoachim]

If you're already doing a subprocess, you don't need a signal, just the main process monitoring the timeout of thz subprocess. Discussion about the signal was to try and explore other solutions than subprocess, but it looks like the subprocess is your best option.

[scttnlsn]

Thanks. In my particular case the work is mostly numeric computation that is not async. A synchronous subprocess with a timeout sounds like the way to go. Thanks!