-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
client/core: rate fetcher tweaks #2512
Conversation
defaultRequestTimeout = time.Second * 20 | ||
defaultRequestTimeout = time.Second * 10 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had set this to 20 seconds a while back, but I've come to realize that I had a misconfiguration on my system that brought some glibc bug to the surface that was causing hella long DNS lookup times in some cases.
client/core/exchangeratefetcher.go
Outdated
} | ||
|
||
reader := io.LimitReader(resp.Body, 1<<20) | ||
return json.NewDecoder(reader).Decode(thing) | ||
} | ||
|
||
func tokenSymbol(symbol string) string { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could go in bip-id.go
. It'll probably be useful elsewhere.
// month for the all tickers request, and 3600 * N requests for N assets. | ||
// So any more than 25000 / 3600 = 6.9 assets, and we can expect to run into | ||
// rate limits. But the bandwidth of the full tickers request is kinda | ||
// ridiculous too. Solution needed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could keep track of how many requests are made, and if it gets to point where it can only make it through the rest of the month if only the all assets request is used, then switch to the all assets request. For most people it won't make it to that point because they don't keep dexc on all the time.
2f0c575
to
fd50c48
Compare
Run rate fetcher all the time, regardless of whether we have sources. If we don't have sources, its a no-op every 12 minutes which is not worth the complication involved with starting and stopping the fetch loop.
Use individual timeouts for requests, not a single timeout for all requests to complete.
Adds some documentation around request limits and our request rates.