Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mapLimit with n/sec limit #1314

Closed
mvalletta opened this issue Oct 27, 2016 · 8 comments
Closed

mapLimit with n/sec limit #1314

mvalletta opened this issue Oct 27, 2016 · 8 comments

Comments

@mvalletta
Copy link

Is possible to limit parallel execution to a number/sec?
This is necessary for those services, like Google Sheet API Limits and Quotas (see https://developers.google.com/analytics/devguides/config/mgmt/v3/limits-quotas) , that limit the number of queries per second (QPS) per IP.
Thanks!

@aearly
Copy link
Collaborator

aearly commented Oct 27, 2016

Similar features have been asked about before. ( #1113 #1082 #1020 #942 #37 ) There is demand for some sort of "throttled queue" functionality, but managing that kind of construct properly is a hard problem.

This is the kind of thing people use dedicated message queue libraries and systems for.

@aearly aearly added the feature label Oct 27, 2016
@megawac
Copy link
Collaborator

megawac commented Oct 27, 2016

tl;dr you/async would need some sort of smoothing function to do this. If we were to do this we would probably implement an exponential smoothing function a token bucket.

If anyone is interested in working on this let us know :)

@JohnLelii-CMCST
Copy link

JohnLelii-CMCST commented Nov 29, 2016

OP: maybe you've already solved this.....I've used another NPM package to implement rate limiting using leakybucket or token bucket models. https://www.npmjs.com/package/limiter This has worked well for me in the past. To create a rate limited mapper function, just do something like this. Your mapper function is directly used with async.

var RateLimiter = require("limiter").RateLimiter;
var limiter = new RateLimiter(2, "second");
var async = require('async');

function rateLimitedMapper(item, callback)
{
  limiter.removeTokens(1, function(err, remainingRequests) {
    console.log("working on item: " + item);
    callback(null, item.length);
  });
}

async.map(input, rateLimitedMapper, function(err, results) {
  // do something with results
 });

@aearly
Copy link
Collaborator

aearly commented Nov 29, 2016

That seems like a lib we should mention in our readme. Seems to do what we wanted to build.

@megawac
Copy link
Collaborator

megawac commented Nov 30, 2016

Yep, it's using a token bucket under the hood which is the same approach I was taking. I'd be quite fine deferring rate limitting to external libraries :)

@aearly
Copy link
Collaborator

aearly commented Nov 30, 2016

Yeah, honestly I wouldn't be thrilled with having to support something like that within Async. There are so many gotchas when doing rate limiting.

@aearly
Copy link
Collaborator

aearly commented Apr 1, 2017

Changing the tags on this to signify that we need a place in the docs for other libraries that extend Async.

@manoharreddyporeddy
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants