You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm thinking of extending the parallelDedupQueue to work as a normal queue. Basically, each actor would have one of the queues for reading and would be adding requests to other queues depending on the hash of the uniqueKey(like the dedup one does).
Just need to figure out the details of how the handleTasks would be spawned:
The crawler must not finish on an empty queue. It can only finish when all queues are done. We can have an interval that checks that once a minute or so and if yes, it shuts down the crawler.
We don't want the crawler that has an empty queue to stay in a busy loop checking for the next request. Need to have some delay there. Need to check the exact implementation in the SDK.
The text was updated successfully, but these errors were encountered:
I'm thinking of extending the parallelDedupQueue to work as a normal queue. Basically, each actor would have one of the queues for reading and would be adding requests to other queues depending on the hash of the uniqueKey(like the dedup one does).
Just need to figure out the details of how the handleTasks would be spawned:
The text was updated successfully, but these errors were encountered: