Logs Processing - Grok Filter

grok filter provides parsing of the unstructured log data into something query-able

Log data:

2017-06-21 22:18:25,276 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)

Predefined match patterns

Custom parse patterns added:

For module: (?<= - )(.+)(?=\[)
For loglevel (?<=\[)(.+)(?=\])

Capture into a field: %{TIMESTAMP_ISO8601:datetime} where datetime is a field name

Full capture pattern:

%{TIMESTAMP_ISO8601:datetime}%{SPACE}%{SPACE}-%{SPACE} (?<module>(?<= - )(.+)(?=\[))(\[)(?<loglevel>(.+)(?=\]))(\]: )%{GREEDYDATA:message}

Transformed data (1st log record):

{
  "datetime": "2017-06-21 22:18:25,276",
  "module": "util.py",
  "loglevel": "DEBUG",
  "message": "2017-06-21 22:18:25,276 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)"
}

Configuration settings:

filter {
  grok {
    match=> {
    "message"=>"%{TIMESTAMP_ISO8601:datetime}%{SPACE}%{SPACE}-%{SPACE} (?<module>(?<= - )(.+)(?=\[))(\[)(?<loglevel>(.+)(?=\]))(\]: )%{GREEDYDATA:message}"
  }
}

Invaluable tools:
- grok constructor
- grok debugger
- regex tester
- Kibana Grok Debugger (as part of X-Pack)