Monitoring nginx requests

Overview

Image: Edit Link

Tech stack

• Logspout
• Logstash
• Statsd exporter

Please read about statsd spec to understand the process

Process

• Nginx proxy is configured to log each HTP request in a format which includes
  • response status code
  • response time

# Example log
10.255.0.42 - - [17/Oct/2017:03:55:03 +0000] "POST /api/data/v1/page/assemble HTTP/1.1" 200 28335 0.077 0.073 ."-" "Go-http-client/1.1"

• Logspout fetches logs from all containers, filters nginx proxy logs and sends to logstash

# Config
syslog+tcp://monitor_logstash:51415?filter.name=*_proxy.*

• Logstash parses each log line to extract values of status code and response time. Using statsd plugin pushes this as statsd format metric to Statsd exporter. Example

nginx.logs.statuscode.200:1|c
nginx.logs.responsetime:12|ms

• Statsd exporter aggregates these per request metrics over configured time period like 10s and exposes this in /metrics HTTP endpoint. It has a mapping which converts metrics from statsd format to prometheus metric format. Example

# Config
- match: nginx.log.statuscode.*
  labels:
    name: "nginx_request_status_count"
    status_code: $1
- match: nginx.log.responsetime
  labels:
    name: "nginx_response_time"

# Metrics
nginx_request_status_count{status_code="200"} 328
nginx_response_time{quantile="0.5"}	0.004
nginx_response_time{quantile="0.9"}	0.069
nginx_response_time{quantile="0.99"}	0.098

• Prometheus periodically scrapes metrics from statsd exporter