Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create parallel request queue #4

Open
metalwarrior665 opened this issue Dec 10, 2020 · 0 comments
Open

Create parallel request queue #4

metalwarrior665 opened this issue Dec 10, 2020 · 0 comments
Labels
enhancement New feature or request

Comments

@metalwarrior665
Copy link
Contributor

I'm thinking of extending the parallelDedupQueue to work as a normal queue. Basically, each actor would have one of the queues for reading and would be adding requests to other queues depending on the hash of the uniqueKey(like the dedup one does).

Just need to figure out the details of how the handleTasks would be spawned:

  • The crawler must not finish on an empty queue. It can only finish when all queues are done. We can have an interval that checks that once a minute or so and if yes, it shuts down the crawler.
  • We don't want the crawler that has an empty queue to stay in a busy loop checking for the next request. Need to have some delay there. Need to check the exact implementation in the SDK.
@metalwarrior665 metalwarrior665 added the enhancement New feature or request label Dec 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant