Dropping lots of messages

[csakoda]

I am testing a simple setup of

logstash-forwarder -> logstash -> kinesis -> lambda -> ES

I can get individual messages to flow through without issue.

However when logstash-forwarder goes to batch lines (say submit 10 log lines at once), I only see some or none of the lines in ES. I found that disabling aggregation helped some (when True no batched messages were appearing in ES at all)

I'm not seeing any errors in the logstash output, or in my lambda function. Anyone have insights?

[samcday]

A couple of things to try:

• Crank up the logging to verbose by passing --verbose to Logstash on startup.
• Are you triggering your Lambda directly off the Kinesis stream? This plugin uses KPL which defaults to writing aggregated records (combining multiple logs into a single Kinesis record using some protobuf serialization), I'm not sure if Lambda knows how to deserialize those aggregated records or if you're expected to do it yourself in your function code. You can try passing aggregation_enabled => false to the Logstash kinesis plugin.

[csakoda]

Thanks much.

aggregation_enabled => false does get some records through but not all.

I'll see what verbose logstash has to say. Seems like teaching the lambda script about aggregated logs might be worthwhile.

[samcday]

You can also check out the stats that KPL is producing for you. Look in the Custom Metrics namespace of CloudWatch. You can check if the number of records that it delivered matches up with how many you were expecting.

[csakoda]

Nothing notable in the logstash verbose output.

Looking at the logs from lambda itself, I see a number of Lambda failed with error Error: socket hang up errors corresponding to the number of missing rows.

Will look at the metrics next.

[csakoda]

Looks like this is something related to the lambda I'm trying to execute (https://github.com/awslabs/amazon-elasticsearch-lambda-samples/blob/master/src/kinesis_lambda_es.js). I switched to the ES bulk API and things started behaving as desired.

Don't believe this is an issue with your project.

Thanks again for the help.

[samcday]

Hey @csakoda - did you end up sorting things out?

[csakoda]

Yup, just pushed a few million events over night to test it out :)

Thanks again.

[samcday]

Good to hear!!