feat: add metrics to track queue sizes and add operations - #AES-361 #3282
Conversation
|
side node, when i build on my laptop i get a unrelated error after running npm install, not sure why this is, should be unrelated. |
| Metrics.observe(TASK_QUEUE_SIZE, this.#pq.size) | ||
| Metrics.observe(TASK_QUEUE_SIZE_PENDING, this.#pq.pending) |
There was a problem hiding this comment.
These might be less useful because there will be many TaskQueue instances that we won't be able to differentiate since they're unnamed.
Looks like they're instantiated through NamedTaskQueue, so the metrics there should be sufficient.
| Metrics.count(LOAD_S3_QUEUE_ADD, 1) | ||
| Metrics.observe(LOAD_S3_QUEUE_SIZE, this.#loadingLimit.size) |
There was a problem hiding this comment.
This is a get call, should we move these lines to the put call? I don't think the S3 store is used anymore, but adding these doesn't hurt.
There was a problem hiding this comment.
well - the way i read this, for whatever reason the .get of the s3 storage is adding a leveldb get call into the queue. so for monitoring the queue, i want it here.
The put doesn't add to the queue, it just does the put synchronously it seems
Description
From the datadog analysis, it looks like the gitcoin out of memory errors are related to queue growth specifically with the p-queue library.
This adds metrics to all instances of PQueue to track the size and add operations.
How Has This Been Tested?
Describe the tests that you ran to verify your changes. Provide instructions for reproduction.
PR checklist
Before submitting this PR, please make sure:
References:
Please list relevant documentation (e.g. tech specs, articles, related work etc.) relevant to this change, and note if the documentation has been updated.