Skip to content

Latest commit

 

History

History
47 lines (44 loc) · 1.24 KB

LogProcessingEx.md

File metadata and controls

47 lines (44 loc) · 1.24 KB
Raw

Log Processing Exercise

  • Log-in into your Elasticsearch sandbox
  • Make sure elasticsearch is running: 
    curl localhost:9200/_cluster/health?pretty
  • Create configuration file: 
    sudo nano /etc/logstash/conf.d/cloud-init.conf
  • Copy-paste settings we have reviewed during previous slide: 
    input {
  file {
    path => "/var/log/cloud-init.log"
    start_position => "beginning"
    type => "logs"
  }
}

filter {
  grok {
    match=> {
      "message"=>"%{TIMESTAMP_ISO8601:datetime}%{SPACE}%{SPACE}-%{SPACE} (?<module>(?<= - )(.+)(?=\[))(\[)(?<loglevel>(.+)(?=\]))(\]: )%{GREEDYDATA:message}"
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "cloud-init"
  }
}
  • Logstash loads all files in the /etc/logstash/conf.d directory, store no extra files there
  • Restart logstash service and monitor messages: 
    sudo service logstash start && sudo tail -f -n 100 /var/log/logstash/logstash-plain.log &
  • Check the data has been populated into elastic search: 
    curl localhost:9200/cloud-init/_search?pretty=true
  • Expected results is a long list of parsed log events
  • New lines added to the log file will be posted into the index