Running tasks asynchronously

[njutn95]

Hey there, couldn't find anything useful in the source so I thought to ask a question in case I've missed something. Say I've got three tasks scheduled to run every minute that are independent from each other. One task is a few seconds running process where the other two are lasting from 20-30 seconds (due to API wait time) bringing it to almost 60 seconds run-time in total.

I'm curious whether it's somehow possible to configure certain tasks to run asynchronously (through sub-process or something else) and some to continue as one-by-one?

I'm also aware of the fact that sub-process can't survive after main process exited, so a possible implementation might be to wait for all of the sub-processes to exit before terminating the main one.

[kbond]

This is a limitation of the bundle currently. See #7 & #6. v1.1.0 introduced a MessageTask that could help. This task sends messages to your message bus. So if the messages sent are configured as async and you have multiple workers, they could be processed in parallel.

Are you using messenger? Could that help?

[kbond]

I have also been thinking of a "task group" concept that could help here. Each task could be assigned to a group (with no group being the default). You could have each of these 3 tasks in a separate group and have 4 cron entry's (one for the default and one for each group):

* * * * * cd /path-to-your-project && php bin/console schedule:run >> /dev/null 2>&1 # default group
* * * * * cd /path-to-your-project && php bin/console schedule:run --group=group1 >> /dev/null 2>&1
* * * * * cd /path-to-your-project && php bin/console schedule:run --group=group2 >> /dev/null 2>&1
* * * * * cd /path-to-your-project && php bin/console schedule:run --group=group3 >> /dev/null 2>&1

Now, every minute, all 4 cron entry's are executed in parallel.

Keep in mind this feature does not yet exist.

[njutn95]

I'm not using messenger, but I do understand how it would help. I'm currently running those tasks that are running longer as a separate cron entry on the server with a custom lock lookup to ensure two servers are not running it.

Task group is something I was originally looking for (and expected when I realized that it's possible to create two or more task schedulers). It's something that really makes sense, as some tasks may unnecessarily postpone other tasks.

[kbond]

Yeah, messenger would have to be specifically configured with multiple workers for this to work and even then it isn't guaranteed. Task groups is the real solution. I've created #40 and will try to implement when I can. Shouldn't be too difficult.

[njutn95]

I'm going to close this issue in favor of #40, if there's a way to help/brainstorm about it, I'm open for it. I don't have much experience with creating/modifying bundles so I'm not sure if I'll be able to make a PR for it. Might try it in the future if you don't tackle it by then.

[AlexeyKosov]

Symfony\Component\Process\Process has a start() method that runs a process asynchronously. Why is it not possible to run all processes together concurrently and wait() for all of them to finish? It would make the most sense.