Skip to content

Export Prometheus metrics from arbitrary unstructured log data. 「支持更复杂的配置规则」。

License

Notifications You must be signed in to change notification settings

theharveyz/grok_exporter

 
 

Repository files navigation

Build Status Build status Coverage Status

grok_exporter

Export Prometheus metrics from arbitrary unstructured log data.

About Grok

Grok is a tool to parse crappy unstructured log data into something structured and queryable. Grok is heavily used in Logstash to provide log data as input for ElasticSearch.

Grok ships with about 120 predefined patterns for syslog logs, apache and other webserver logs, mysql logs, etc. It is easy to extend Grok with custom patterns.

The grok_exporter aims at porting Grok from the ELK stack to Prometheus monitoring. The goal is to use Grok patterns for extracting Prometheus metrics from arbitrary log files.

How to run the example

Download grok_exporter-$ARCH.zip for your operating system from the releases page, extract the archive, cd grok_exporter-$ARCH, then run

./grok_exporter -config ./example/config.yml

The example log file exim-rejected-RCPT-examples.log contains log messages from the Exim mail server. The configuration in config.yml counts the total number of rejected recipients, partitioned by error message.

The exporter provides the metrics on http://localhost:9144/metrics:

screenshot.png

Configuration

注意,本项目在原有配置格式的基础上,进行了二次修改,为了方便支持同一指标对应不同匹配方式的情况。

原有配置格式如下:

Example configuration:

global:
    config_version: 2
input:
    type: file
    path: ./example/example.log
    readall: true
grok:
    patterns_dir: ./logstash-patterns-core/patterns
metrics:
    - type: counter
      name: grok_example_lines_total
      help: Counter metric example with labels.
      match: '%{DATE} %{TIME} %{USER:user} %{NUMBER}'
      labels:
          user: '{{.user}}'
server:
    port: 9144

CONFIG.md describes the grok_exporter configuration file and shows how to define Grok patterns, Prometheus metrics, and labels.

修改后的格式如下:

global:
    config_version: 2
input:
    type: file
    path: /your-log-path.log
    readall: false
    fail_on_missing_logfile: false
grok:
    patterns_dir: /your-grok-patterns-dir/
    additional_patterns:
    - 'ID XXXX[0-9]{4,}'
metrics:
    - type: gauge
      name: your_metric
      help: '订单信息'
      cumulative: false
      retention: 20s
      matchs:
        - match: '.*XXXXXX.*"%{XXXXX:id}".*'
          value: 1
          labels:
            id: '{{. id}}'
        - match: '.*BBBBBB.*"%{XXXXX:id}".*'
          value: 1
          labels:
            id: '{{. id}}'

新增matchs参数,并将如下参数,放到matchs每一项下,但是含义保持不变:

  • value
  • lables
  • match

修改后,对于同一指标项,可以指定多种匹配规则。

Status

Operating system support:

Grok pattern support:

Prometheus support:

How to build from source

In order to compile grok_exporter from source, you need Go installed and $GOPATH set, and you need the header files for the Oniguruma regular expression library.

Installing the Oniguruma library on OS X

The current version of brew install oniguruma will install Oniguruma 6.1.0. Because of this bug version 6.1.0 will not work with grok_exporter. Use the following to install the stable 5.9.6 version:

brew install fstab/oniguruma/oniguruma-5.9.6

Installing the Oniguruma library on Ubuntu Linux

The current version on Ubuntu is 5.9.6, which is good:

sudo apt-get install libonig-dev

Installing the Oniguruma library from source

Make sure to use version 5.9.6 until grok_exporter supports newer versions:

wget https://github.com/kkos/oniguruma/releases/download/v5.9.6/onig-5.9.6.tar.gz
tar xfz onig-5.9.6.tar.gz
cd onig-5.9.6 && ./configure && make && make install

Installing grok_exporter

With Oniguruma 5.9.6 installed, download and compile grok_exporter as follows:

go get github.com/fstab/grok_exporter
cd $GOPATH/src/github.com/fstab/grok_exporter
git submodule update --init --recursive

The resulting grok_exporter binary will be dynamically linked to the Oniguruma library, i.e. it needs the Oniguruma library to run. The releases are statically linked with Oniguruma, i.e. the releases don't require Oniguruma as a run-time dependency. The releases are built with release.sh.

More Documentation

User documentation is included in the GitHub repository:

  • CONFIG.md: Specification of the config file.
  • BUILTIN.md: Definition of metrics provided out-of-the-box.

Developer notes are available on the GitHub Wiki pages:

External documentation:

Contact

  • For feature requests, bugs reports, etc: Please open a GitHub issue.
  • For bug fixes, contributions, etc: Create a pull request.
  • Questions? Contact me at [email protected].

Related Projects

Google's mtail goes in a similar direction. It uses its own pattern definition language, so it will not work out-of-the-box with existing Grok patterns. However, mtail's RE2 regular expressions are probably more CPU efficient than Grok's Oniguruma patterns. mtail reads logfiles using the fsnotify library, which might be an obstacle on operating systems other than Linux.

License

Licensed under the Apache License, Version 2.0. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

About

Export Prometheus metrics from arbitrary unstructured log data. 「支持更复杂的配置规则」。

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 91.7%
  • Shell 7.3%
  • C 1.0%