aws-cloudtrail-to-logstash

Pulls down cloudtrail logs and outputs to a json file to be ingested by LogStash.

Requires

s3cmd -and- jq

Configuration

1. Edit the script and update variable 'AWS_ACCOUNT_NUMBER' with your account number.
2. Edit the script and update variable 'S3_BUCKET_NAME' with your buckets name.
3. Ensure your AWS credentials are configured (e.g: ~/.aws/credentials)
4. Execute the script (it will run in a constant loop)

Configuration extended.

Included is a traditional init script and monit config (optional), to use the init script...

1. Ensure 'cloud-trail-consumer.sh' is located here: /opt/cloudtrail-consumer
2. Alternatively, edit 'cloudtrail-consumer-init.sh' and change variable 'THE_PATH'
3. Place the init script[cloudtrail-consumer-init.sh] in /etc/init.d
4. Make init script executable: chmod ug+x cloudtrail-consumer-init.sh
5. It can then be started as such: /etc/init.d/cloudtrail-consumer-init.sh start (or stop)
6. The included monit script can be placed in your monits configuration directory and used to startup the cloudtrail consumer on start up.

Ingesting ELB logs

See here: https://github.com/vigeek/aws-elb-logs-to-logstash

Example dashboard (included)

To install, please view readme in kibana-dashboard directory