What is it?

ELK setup to take in flow data and give you searchable dashboards based on any current flow information, as all current flow information being used here delivers destination ports ect at layer 4 the logstash agent will gsub each port to what is defined in RFC for that part. Though both resulsts are still visible in the dashboards.

Reverse DNS has also been enabled for src and dst, if you scroll to the bottom of the dashboard in your brwose you will seee the searchable fields on your left!

These configs are designed to run single instance per service! Do not mix them up unless you have lots of spare CPU and RAM

Installing

This is for Ubuntu 14 only ;)

Lots of pre req command first
The next item is critical as we will be copying configs from git to the server we need to make sure they are right!
ENVIROMENT=Netflow
ENVIROMENT=VPC
Only choose one of the above

wget https://github.com/mbakerbp/elk-flowdata/archive/master.zip
unzip master.zip
cd elk-flowdata-master/
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update
sudo apt-get -y install oracle-java8-installer
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.6.0.deb
sudo dpkg -i elasticsearch-1.6.0.deb
sudo service elasticsearch restart
sudo update-rc.d elasticsearch defaults 95 10
sudo apt-get install nginx
echo 'deb http://packages.elasticsearch.org/logstash/1.5/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.list
sudo apt-get update
sudo apt-get install logstash
cd $ENVIROMENT
``DIR=`pwd```
rm /etc/nginx/sites-enabled/default
cp etc/nginx/sites-enabled/kibana /etc/nginx/sites-enabled/
cp etc/logstash/conf.d/logstash.conf /etc/logstash/conf.d/logstash.conf
cd /opt/logstash/vendor/geoip/
wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
gunzip GeoIPASNum.dat.gz
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz
/opt/logstash/bin/plugin update
service nginx restart
service elasticsearch restart
service logstash restart
mkdir -p /var/www/kibana
cd /var/www/kibana
wget https://download.elastic.co/kibana/kibana/kibana-3.1.2.tar.gz
tar zxf kibana-3.1.2.tar.gz --strip-components=1
rm kibana-3.1.2.tar.gz
cp $DIR/var/www/kibana/config.js .
cp $DIR/var/www/kibana/app/dashboards/default.json app/dashboards/default.json

All installed!

Sit back and wait for the magic, if you do not see anything come up open up /etc/logstash/conf.d/logstash.conf and goto the bottom and uncomment out the line:

stdout { codec => rubydebug }

Then stop logstash and cd /opt/logstash/bin;./logstash -f /etc/logstash/conf.d/logstash.conf --verbose --debug and look for any possible config problems there.