in_process_exporter_metrics: Add documentation for process_exporter_m…

…etrics (#1198) * in_process_exporter_metrics: Add documentation for process_exporter_metrics Signed-off-by: Hiroshi Hatake <[email protected]> * Explain the detail of process level of metrics Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * Clearify meanings of the sentence Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * Simplify the description of scrape_interval parameter Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * Separate sentence to review easily Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * Use 2.2 version of fluent-bit on command line example Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * Clearify measings of the sentence for enhancements Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * Simplify the sentence Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * Remove a needless sysfs mount Signed-off-by: Hiroshi Hatake <[email protected]> * Use readonly option on docker commandline Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * Add more descriptions for procfs and macOS is not supported Signed-off-by: Hiroshi Hatake <[email protected]> * Clarify read-only to be enough Signed-off-by: Hiroshi Hatake <[email protected]> * Fix incorrect descriptions of regex parameters Signed-off-by: Hiroshi Hatake <[email protected]> * Clarify the behavior of metrics type of events in pipelines Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * in_process_exporter_metrics: Add descriptions for metrics parameter and their metrics Signed-off-by: Hiroshi Hatake <[email protected]> * Clarify the sentence meaning Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * Explain why this plugin won't work on macOS clearly Co-authored-by: Pat <[email protected]> Signed-off-by: Hiroshi Hatake <[email protected]> * Remove environment variable related arugment for simplicity Signed-off-by: Hiroshi Hatake <[email protected]> --------- Signed-off-by: Hiroshi Hatake <[email protected]> Co-authored-by: Pat <[email protected]>
fluent · Dec 1, 2023 · d474c95 · d474c95
1 parent 2ea4855
commit d474c95
Show file tree

Hide file tree

Showing 2 changed files with 104 additions and 0 deletions.
diff --git a/SUMMARY.md b/SUMMARY.md
@@ -103,6 +103,7 @@
   * [Node Exporter Metrics](pipeline/inputs/node-exporter-metrics.md)
   * [Podman Metrics](pipeline/inputs/podman-metrics.md)
   * [Process Log Based Metrics](pipeline/inputs/process.md)
+  * [Process Exporter Metrics](pipeline/inputs/process-exporter-metrics.md)
   * [Prometheus Scrape Metrics](pipeline/inputs/prometheus-scrape-metrics.md)
   * [Random](pipeline/inputs/random.md)
   * [Serial Interface](pipeline/inputs/serial-interface.md)

diff --git a/pipeline/inputs/process-exporter-metrics.md b/pipeline/inputs/process-exporter-metrics.md
@@ -0,0 +1,103 @@
+---
+description: >-
+  A plugin based on Process Exporter to collect process level of metrics of system
+  metrics
+---
+
+# Process Exporter Metrics
+
+[Prometheus Node Exporter](https://github.com/prometheus/node_exporter) is a popular way to collect system level metrics from operating systems, such as CPU / Disk / Network / Process statistics.
+Fluent Bit 2.2 onwards includes a process exporter plugin that builds off the Prometheus design to collect process level metrics without having to manage two separate processes or agents.
+
+The Process Exporter Metrics plugin implements collecting of the various metrics available from [the 3rd party implementation of Prometheus Process Exporter](https://github.com/ncabatoff/process-exporter) and these will be expanded over time as needed.
+
+**Important note:** All metrics including those collected with this plugin flow through a separate pipeline from logs and current filters do not operate on top of metrics.
+
+This plugin is only supported on Linux based operating systems as it uses the `proc` filesystem to access the relevant metrics.
+
+macOS does not have the `proc` filesystem so this plugin will not work for it.
+
+
+## Configuration
+
+| Key                       | Description                                                                            | Default   |
+| ------------------------- | -------------------------------------------------------------------------------------- | --------- |
+| scrape\_interval          | The rate at which metrics are collected.                                               | 5 seconds |
+| path.procfs               | The mount point used to collect process information and metrics. Read-only is enough   | /proc/    |
+| process\_include\_pattern | regex to determine which names of processes are included in the metrics produced by this plugin | It is applied for all process unless explicitly set. Default is `.+`. |
+| process\_exclude\_pattern | regex to determine which names of processes are excluded in the metrics produced by this plugin | It is not applied unless explicitly set. Default is `NULL`. |
+| metrics                   | To specify which process level of metrics are collected from the host operating system. These metrics depend on `/proc` fs. The actual values of metrics will be read from `/proc` when needed. cpu, io, memory, state, context\_switches, fd, start\_time, thread\_wchan, thread depend on procfs. | `cpu,io,memory,state,context_switches,fd,start_time,thread_wchan,thread` |
+
+## Metrics Available
+
+| Name              | Description                                                                                      |
+| ----------------- | -------------------------------------------------- |
+| cpu               | Exposes CPU statistics from `/proc`.               |
+| io                | Exposes I/O statistics from `/proc`.               |
+| memory            | Exposes memory statistics from `/proc`.            |
+| state             | Exposes process state statistics from `/proc`.     |
+| context\_switches | Exposes context\_switches statistics from `/proc`. |
+| fd                | Exposes file descriptors statistics from `/proc`.  |
+| start\_time       | Exposes start\_time statistics from `/proc`.       |
+| thread\_wchan     | Exposes thread\_wchan from `/proc`.                |
+| thread            | Exposes thread statistics from `/proc`.            |
+
+## Getting Started
+
+### Simple Configuration File
+
+In the following configuration file, the input plugin _process\_exporter\_metrics collects _metrics every 2 seconds and exposes them through our [Prometheus Exporter](../outputs/prometheus-exporter.md) output plugin on HTTP/TCP port 2021.
+
+```
+# Process Exporter Metrics + Prometheus Exporter
+# -------------------------------------------
+# The following example collect host metrics on Linux and expose
+# them through a Prometheus HTTP end-point.
+#
+# After starting the service try it with:
+#
+# $ curl http://127.0.0.1:2021/metrics
+#
+[SERVICE]
+    flush           1
+    log_level       info
+
+[INPUT]
+    name            process_exporter_metrics
+    tag             process_metrics
+    scrape_interval 2
+
+[OUTPUT]
+    name            prometheus_exporter
+    match           process_metrics
+    host            0.0.0.0
+    port            2021
+```
+
+You can see the metrics by using _curl:_
+
+```bash
+curl http://127.0.0.1:2021/metrics
+```
+
+### Container to Collect Host Metrics
+
+When deploying Fluent Bit in a container you will need to specify additional settings to ensure that Fluent Bit has access to the process details.
+The following `docker` command deploys Fluent Bit with a specific mount path for `procfs` and settings enabled to ensure that Fluent Bit can collect from the host. 
+These are then exposed over port 2021.
+
+```
+docker run -ti -v /proc:/host/proc:ro \
+               -p 2021:2021        \
+               fluent/fluent-bit:2.2 \
+               /fluent-bit/bin/fluent-bit \
+                         -i process_exporter_metrics -p path.procfs=/host/proc  \
+                         -o prometheus_exporter \
+                         -f 1
+```
+
+## Enhancement Requests
+
+Development prioritises a subset of the available collectors in the [the 3rd party implementation of Prometheus Process Exporter](https://github.com/ncabatoff/process-exporter), to request others please open a Github issue by using the following template:\
+\
+\- [in_process_exporter_metrics](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_process_exporter_metrics:%20add%20ABC%20collector)