-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mapLimit with n/sec limit #1314
Comments
tl;dr you/async would need some sort of smoothing function to do this. If we were to do this we would probably implement If anyone is interested in working on this let us know :) |
OP: maybe you've already solved this.....I've used another NPM package to implement rate limiting using leakybucket or token bucket models. https://www.npmjs.com/package/limiter This has worked well for me in the past. To create a rate limited mapper function, just do something like this. Your mapper function is directly used with async. var RateLimiter = require("limiter").RateLimiter;
var limiter = new RateLimiter(2, "second");
var async = require('async');
function rateLimitedMapper(item, callback)
{
limiter.removeTokens(1, function(err, remainingRequests) {
console.log("working on item: " + item);
callback(null, item.length);
});
}
async.map(input, rateLimitedMapper, function(err, results) {
// do something with results
}); |
That seems like a lib we should mention in our readme. Seems to do what we wanted to build. |
Yep, it's using a token bucket under the hood which is the same approach I was taking. I'd be quite fine deferring rate limitting to external libraries :) |
Yeah, honestly I wouldn't be thrilled with having to support something like that within Async. There are so many gotchas when doing rate limiting. |
Changing the tags on this to signify that we need a place in the docs for other libraries that extend Async. |
Another way to do this using Promise.all and promise-throttle: |
Is possible to limit parallel execution to a number/sec?
This is necessary for those services, like Google Sheet API Limits and Quotas (see https://developers.google.com/analytics/devguides/config/mgmt/v3/limits-quotas) , that limit the number of queries per second (QPS) per IP.
Thanks!
The text was updated successfully, but these errors were encountered: