hotfix(mashape-analytics) improve ALF buffer under load

[thibaultcha]

As reported by #750, the buffer is vulnerable to a heavy load because of
its sending_queue of batches pending for sending which might grow too big and eventually crash the Lua VM. It also does not
handle the new 207 HTTP status code returned by the collector in case of
invalid ALFs.

Changes:

• handle 207 HTTP status code by discarding the batch. Some ALFs in it
  will have been saved, and the invalid one(s) should not be retried.
• implement a maximum size (in MB) for the sending_queue, as suggested
  by Lua VM crashed, reason: not enough memory #750. Originally, I was about to implement such a size limitation
  by "number of batches pending", but it would not be intuitive for
  users to know what fits best their use case, because ALF sizes varies
  from one API to another, and one endpoint to another. By defining it
  in MB it is easier for users to chose a value. The default value of
  10MB has been chosen after performing some benchmarking, and should
  handle from 300 to 500 req/s depending on the ALFs sizes.
  When the sending_queue has reached its limit, the current ALFs in
  the buffer will be discarded.
• implement a retry policy. Instead of insisting on retrying to send batches
  when the collector cannot be reached, a delay is computed and
  exponentially increases on each failure to connect to the collector.
  That delay is shared by all workers. This avoids to load the collector
  when it is having difficulties and saves up bandwidth on Kong's side.
  As soon as the collector can be reached again, the delay is reset.
  Currently, the minimum retry delay is 1s and the maximum is 60s. Those
  values cannot be configured.
• no more line jumps in logs printing responses from the collector.

This still needs testing and add a few more tests.

[cb372]

Is this change from 1MB to 500MB correct? Seems awfully large. This is the maximum payload size that the socket server will accept in a single request, right? Given that max_queue_size defaults to 10MB, and the queue holds multiple batches, I would expect this value to be < 10MB.

[thibaultcha]

This value is correct. This constant will be renamed, and its actual purpose is to mirror a constant present in the Galileo collector. It is the maximum payload size accepted by an HTTP request by the Galileo collector. It used to be 1MB, and is now 500. I believe this value should mirror the Galileo restrictions in the case someone really knows what he is doing and wish to send extremely large batches of ALFs (possibly containing request bodies that are very large). This value is hard coded and cannot be bypassed. Any ALF beyond that limit will be dropped as it is too big by itself, and if a batch reaches that limit, it will be put in the sending_queue directly, wether it has reached the configured batch_size or not.

batch_size (n of ALFs) and sending_queue_size (Mbs) are the value one should tweak to enforce a low memory footprint. I could see a third value, being the batch_size but in terms of bytes too, being useful. In case one's ALFs are much much bigger than he intended to. It would make the plugin more complicated to understand and configure (it already does with the sending_queue_size so I am not sure about that yet.

[SGrondin]

Hi @cb372 yes it's very large, but it isn't meant as a way to reject very large batches, it's mostly just a failsafe in case a client gets stuck and forgets to close the tcp socket or something like that. The collector handles it just fine even with extremely large batches. If it reaches a size that big without being a bad socket, it's often due to huge bodies and we don't want to drop those.