Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[node-pod] Add Queuing #618

Closed
kenslachtajr opened this issue Dec 7, 2022 · 0 comments
Closed

[node-pod] Add Queuing #618

kenslachtajr opened this issue Dec 7, 2022 · 0 comments
Assignees

Comments

@kenslachtajr
Copy link

Description

(these requirements come from meeting with Dustin 12/7/22) We will have a POST endpoint which will accept something that will add the job to BullMQ with the package name. With BullMQ we will automatically listen for changes in the database and run the jobs in FIFO ordering. Jobs should run in worker threads.

Note: there is a possibility that Redis client does not work well with BullMQ, but this will be determined in further research.

Notes

General notes from specifications:

As applications grow, the need for asynchronous workers become needed. Newer developers are less familiar with this pattern. As such, we want to include this as part of the starter kit as an opt-in feature so it can be easily used by developers.


Requirement Description
Select Queue Tech There are several queue solutions on the market that serve different purposes. Select a queue that makes the most sense for the project and add it.
Infrastructure Setup Regardless of the queue selected, a message queue will need to be utilized. For push queues, this is typically done with a formal message queue tech such as ElasticMQ, RabbitMQ, etc. For pull queues, this is typically done with Redis or equivalent in-memory storage. All of these require some level of local setup. Add this setup to the docker compose with setup instructions.
Add job generator endpoint Add a POST /example-job endpoint that receives a message and adds it as a job to the queue to be processed.
Add worker/job processor Add a worker that can process the job added to the queue. It should simply print the job message to STDOUT.
Add queue runner Add or modify scripts to run the queue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants