One main queue vs multi mini queues?

[ragokan]

Hello, every week I do start a job to check premium for users in my backend server. There is like 220K users currently and it is increasing slowly. Because it took so long to check all of them separately, I started using Bull and it is amazing. My main problem with this is which one is better?:

A: Having one main queue that has a for loop of users and check each one by one
B: Having multi mini queues; I do a for loop outside of queue, in a for loop I do add userId to the queue, and in queue I do check.

If I did not clarify my problem, please ask me. Thanks.

[ccollie]

What exactly are you checking ? Perhaps if you explain in more detail we can provide more insight.

[ragokan]

I want the most non-blocking performance. Before Bull, in the process of checking premium, my server had performance issues while checking, especially while doing synchronous operations.

[ccollie]

Could you be more specific ? It's still not clear what checking premium means. Not knowing your problem set, it seems like you can optimize the iteration by storing results of previous runs, depending on what premium means.

For example, if it means greater than x in purchases and 3 month of membership, those can be maintained as separate indexes (indices ?) so you don't need a full scan.

In that case though, it's a data-modeling and not a bull issue.

[manast]

@ragokan also if you want to go for the Pro version we support groups, which act as micro-queues and you could have one per user id for example: https://docs.bullmq.io/bullmq-pro/groups

[ragokan]

I think what I want looks like groups @manast, let me check it.

@ccollie I probably didn't tell my problem in a good way. The problem is basically like this; You have a job. This job has a for loop of items.length. Items.length is 220k. Is there any difference between doing it in a one queue and 220k sub queues?