Improvement on rack-attack configuration

[weiweishi]

We have noticed occasionally ERA will give 429 error message for regular web access as well as OAI harvesting after we introduced rake-attack in #954.

We would like to improve the default configuration:

• Add logging to blocked access to analyze what is legitimate access, and what is malicious spamming access. This can be done as such:

ActiveSupport::Notifications.subscribe('rack.attack') do |name, start, finish, request_id, req|
  if req.env["rack.attack.match_type"] == :throttle
    request_headers = { "CF-RAY" => req.env["HTTP_CF_RAY"],
                        "X-Amzn-Trace-Id" => req.env["HTTP_X_AMZN_TRACE_ID"] }

    Rails.logger.info "[Rack::Attack][Blocked]" <<
                      "remote_ip: \"#{req.remote_ip}\"," <<
                      "path: \"#{req.path}\", " <<
                      "headers: #{request_headers.inspect}"
  end
end

• We can selectively choose the routes we'd like to protect. This can include the home path "/", search "/search", authentication "/auth/saml", anything else we should consider?
• We should safelist IPs include LAC's and staff IP ranges? Those should be set as something that can be managed by Ansible/env vars, so changes don't require codebase change.

resources:
[https://blog.bigbinary.com/2018/05/15/how-to-mitigate-ddos-using-rack-attack.html]
[https://github.com/kickstarter/rack-attack]

[ConnorSheremeta]

I'm not sure its worth safelisting any IPs, most example configurations have an unrealistically attainable throttling limit (limit of 40 hits to a route per minute per IP for example.) Also, if its possible to be throttled wouldn't we like to find out about it first hand so that the configuration could be modified? What are your thoughts @weiweishi ? Also, should throttling be captured in Rollbar?

[pgwillia]

Does anyone know how to use ActiveSupport::Notifications? Apparently that's the natural language of rack-attack logging and could be useful Rails knowledge more generally.

[weiweishi]

Decision is made that we will not safelisting our staff/student assistant IPs. We will add the configuration to safelisting single IP addresses in the configuration file, so that we could do testing with LAC's OAI harvesting when we find a time to coordinate such testing with them.

[weiweishi]

And no, @pgwillia have not heard of ActiveSupport::Notifications. Looks like something very useful to implement for this and can leverage it for other types of application support notifications as well

[ConnorSheremeta]

Did some bench marking regarding lookup times of whitelisted IP's using different structures to hold those ips: array, hash, and btree with a depth of 5. Doing 100,000 requests yeilds (in seconds):

                            1 ip          20ips          80ips         500ips      10,000ips
              array     0.009489       0.034668       0.087295       0.479266       8.882781
              btree     0.035576       0.062198       0.087248       0.116727       0.164749
              hash      0.006784       0.005959       0.005863       0.005547       0.006009

At 80 whitelisted ips it would be more efficient to use a btree over an array but it looks like hash table is by far the most efficient regardless of the number of ips

[mbarnett]

re: hash table

I'm not surprised it beats the array at higher IPs, but early on that's very surprising. I'm going to poke at it a bit just to see if I get similar results, because if so there's no reason not to just use a hashtable approach across the board?