firehose event gets truncated/appended garbage characters

[di1214]

I am playing with this plugin with packetbeat, some of the messages send to firehose gets truncated or appended garbage characters. Which fails deliver to ES. Not sure why it is happening

firehose raw data: eyJAdGltZXN0YW1wIjoiMjAxOC0wNS0yMVQyMzoyODo0MC4wMDBaIiwiQG1ldGFkYXRhIjp7ImJlYXQiOiJwYWNrZXRiZWF0IiwidHlwZSI6ImRvYyIsInZlcnNpb24iOiI2LjIuNCJ9LCJmaW5hbCI6ZmFsc2UsImJlYXQiOnsibmFtZSI6ImlwLTEtMTEtMy0yMDIiLCJob3N0bmFtZSI6ImlwLTEtMTEtMy0yMDIiLCJ2ZXJzaW9uIjoiNi4yLjQifSwic291cmNlIjp7ImlwdjYiOiJmZTgwOjo0NDo3YmZmOmZlZDg6YTIyZSIsInBvcnQiOjU0Niwic3RhdHMiOnsibmV0X3BhY2tldHNfdG90YWwiOjEsIm5ldF9ieXRlc190b3RhbCI6MTE2fX0sImxhc3RfdGltZSI6IjIwMTgtMDUtMjFUMjM6Mjg6MDkuMTc5WiIsInN0YXJ0X3RpbWUiOiIyMDE4LTA1LTIxVDIzOjI4OjA5LjE3OVoiLCJ0eXBlIjoiZmxvdyIsImZsb3dfaWQiOiJRQUwvLy8vLy8vOEEvLzhnLy84QUFBSCtnQUFBQUFBQUFBQkVlLy8rMktJdS93SUFBQUFBQUFBQUFBQUFBQUVBQWlJQ0l3SSIsImZpZWxkcyI6eyJsb2dfc291cmNlIjoidGF5bG9yLXBhY2tldGJlYXQifSwibWV0YSI6eyJjbG91ZCI6eyJhdmFpbGFiaWxpdHlfem9uZSI6ImV1LXdlc3QtMWIiLCJwcm92aWRlciI6ImVjMiIsImluc3RhbmNlX2lkIjoiaS0wYWRjZmUxMTg1MmViMmQxNiIsIm1hY2hpbmVfdHlwZSI6InQyLm1lZGl1bSIsInJlZ2lvbiI6ImV1LXdlc3QtMSJ9fSwidHJhbnNwb3J0IjoidWRwIiwiZGVzdCI6eyJpcHY2IjoiZmYwMjo6MToyIiwicG9ydCI6NTQ3fX1BQUFGL0FBQUJmdw

json:
{"@timestamp":"2018-05-21T23:28:40.000Z","@metadata":{"beat":"packetbeat","type":"doc","version":"6.2.4"},"final":false,"beat":{"name":"ip-1-11-3-202","hostname":"ip-1-11-3-202","version":"6.2.4"},"source":{"ipv6":"fe80::44:7bff:fed8:a22e","port":546,"stats":{"net_packets_total":1,"net_bytes_total":116}},"last_time":"2018-05-21T23:28:09.179Z","start_time":"2018-05-21T23:28:09.179Z","type":"flow","flow_id":"QAL///////8A//8g//8AAAH+gAAAAAAAAABEe//+2KIu/wIAAAAAAAAAAAAAAAEAAiICIwI","fields":{"log_source":"packetbeat"},"meta":{"cloud":{"availability_zone":"eu-west-1b","provider":"ec2","instance_id":"i-xxxx","machine_type":"t2.medium","region":"eu-west-1"}},"transport":"udp","dest":{"ipv6":"ff02::1:2","port":547}}AAAF/AAABfw

[mumoshu]

@di1214 Hi. I believe I spotted the source of issue.

This is data corruption. awsbeats is passing output from beat json encoder's output to the record sent to Firehose. This is wrong - beat's json encoder isn't designed to be used this way. It should copy the output before further usage.

So, instead of:

return &firehose.Record{Data: serializedEvent}, nil

We should write:

buf := make([]byte, len(serializedEvent))
copy(buf, serializedEvent)
serializedEvent = buf

return &firehose.Record{Data: serializedEvent}, nil

[di1214]

@mumoshu I have tested with your suggestion. It works now! thanks

[s12v]

@mumoshu, awesome catch