HTTP rate limiting

[ewdurbin]

We currently have a couple services that ostensibly have HTTP rate limiting configured, but it is not clear that it is functioning as expected. Namely bugs/roundup which is a pretty consistent noisy service due to crawlers/traffic.

Let's assess the current state of rate limiting for services managed by our salt infrastructure and being served by our haproxy load balancers (it should be all of them?) to determine the functionality of rate limiting and if there's some way we could consolidate that rate limiting into the haproxy layer rather than having it be service by service.

Tasks

Preview Give feedback

1. Assess missing, if any, loadbalanced services
2. Investigage doing rate limiting with haproxy

[JacobCoffee]

I think this is the list of items not behind (most/all are expected I think?)

• backups
• consul
• cdn-logs
• mail
• Mailman3
• Pythontest

[JacobCoffee]

could the lb health checks be not helping here?

coffee@bugs:~$ tail -1000 /var/log/nginx/roundup-cpython.access.log | awk '{print $1}' | sort | uniq -c | sort -n | tail
...
15 ...
29 ...
31 ...
53 ...
**82 10.132.111.89**
**83 10.132.109.52**