Merge branch 'fluent:master' into master

fluent · Dec 9, 2023 · 80f9c57 · 80f9c57
2 parents 60ebef3 + 76ed2fb
commit 80f9c57
Show file tree

Hide file tree

Showing 97 changed files with 4,069 additions and 381 deletions.
diff --git a/.gitbook.yaml b/.gitbook.yaml
@@ -9,6 +9,7 @@ redirects:
     input/disk:            ./pipeline/inputs/disk-io-metrics.md
     #inputs/docker:          ./pipeline/inputs/
     input/dummy:           ./pipeline/inputs/dummy.md
+    input/elasticsearch:   ./pipeline/inputs/elasticsearch.md
     input/exec:            ./pipeline/inputs/exec.md
     input/fluentbit:       ./pipeline/inputs/fluentbit-metrics.md
     input/forward:         ./pipeline/inputs/forward.md
@@ -20,6 +21,7 @@ redirects:
     input/netif:           ./pipeline/inputs/network-io-metrics.md
     input/proc:            ./pipeline/inputs/process.md
     input/random:          ./pipeline/inputs/random.md
+    input/splunk:          ./pipeline/inputs/splunk.md
     input/serial:          ./pipeline/inputs/serial-interface.md
     #inputs/statsd:          ./pipeline/inputs/
     input/stdin:           ./pipeline/inputs/standard-input.md
@@ -28,6 +30,7 @@ redirects:
     input/tail:            ./pipeline/inputs/tail.md
     input/tcp:             ./pipeline/inputs/tcp.md
     input/thermal:         ./pipeline/inputs/thermal.md
+    input/udp:             ./pipeline/inputs/udp.md
     input/winlog:          ./pipeline/inputs/windows-event-log.md
     input/winevtlog:       ./pipeline/inputs/windows-event-log-winevtlog.md
 
@@ -50,6 +53,7 @@ redirects:
     output/azure_kusto:    ./pipeline/outputs/azure_kusto.md
     output/bigquery:       ./pipeline/outputs/bigquery.md
     output/counter:        ./pipeline/outputs/counter.md
+    output/chronicle:      ./pipeline/outputs/chronicle.md
     output/cloudwatch:     ./pipeline/outputs/cloudwatch.md
     output/datadog:        ./pipeline/outputs/datadog.md
     output/es:             ./pipeline/outputs/elasticsearch.md

diff --git a/.gitbook/assets/FluentBitDocumentation-01-01.png b/.gitbook/assets/FluentBitDocumentation-01-01.png
diff --git a/.gitbook/assets/FluentBitV3-02.png b/.gitbook/assets/FluentBitV3-02.png
diff --git a/.gitbook/assets/azure-logs-ingestion-overview.png b/.gitbook/assets/azure-logs-ingestion-overview.png
diff --git a/.gitbook/assets/logo_documentation_2.1.png b/.gitbook/assets/logo_documentation_2.1.png
diff --git a/.gitbook/assets/v2.2 [email protected] b/.gitbook/assets/v2.2 [email protected]
diff --git a/.github/workflows/cron-stale.yaml b/.github/workflows/cron-stale.yaml
@@ -11,7 +11,7 @@ jobs:
       issues: write
       pull-requests: write
     steps:
-      - uses: actions/stale@v7
+      - uses: actions/stale@v8
         with:
           repo-token: ${{ secrets.GITHUB_TOKEN }}
           stale-issue-message: 'This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the `exempt-stale` label.'

diff --git a/.github/workflows/pr-lint.yaml b/.github/workflows/pr-lint.yaml
@@ -8,16 +8,16 @@ jobs:
     runs-on: ubuntu-latest
     name: PR - Actionlint
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - run: |
           echo "::add-matcher::.github/actionlint-matcher.json"
           bash <(curl https://raw.githubusercontent.com/rhysd/actionlint/main/scripts/download-actionlint.bash)
           ./actionlint -color -shellcheck=
         shell: bash
 
-  docslint-pr:
+  shellcheck-pr:
     runs-on: ubuntu-latest
-    name: PR - Markdownlint
+    name: PR - Shellcheck
     steps:
-      - name: Run markdownlint
-        uses: actionshub/[email protected]
+      - uses: actions/checkout@v4
+      - uses: ludeeus/action-shellcheck@master
diff --git a/README.md b/README.md
@@ -1,27 +1,27 @@
 ---
-description: High Performance Log and Metrics Processor
+description: High Performance Telemetry Agent for Logs, Metrics and Traces
 ---
 
-# Fluent Bit v2.0 Documentation
+# Fluent Bit v2.2 Documentation
 
-![](.gitbook/assets/logo\_documentation\_2.0.png)
+<figure><img src=".gitbook/assets/v2.2 big@2x.png" alt=""><figcaption></figcaption></figure>
 
-[Fluent Bit](http://fluentbit.io) is a Fast and Lightweight Logs and Metrics Processor and Forwarder for Linux, OSX, Windows and BSD family operating systems. It has been made with a strong focus on performance to allow the collection of events from different sources without complexity.![](https://static.scarf.sh/a.png?x-pxid=71f0e011-761f-4c6f-9a89-38817887faae)
+[Fluent Bit](http://fluentbit.io) is a Fast and Lightweight **Telemetry Agent** for Logs, Metrics, and Traces for Linux, macOS, Windows, and BSD family operating systems. It has been made with a strong focus on performance to allow the collection and processing of telemetry data from different sources without complexity.![](https://static.scarf.sh/a.png?x-pxid=71f0e011-761f-4c6f-9a89-38817887faae)
 
 ## Features
 
-* High Performance
+* High Performance: High throughput with low resources consumption
 * Data Parsing
   * Convert your unstructured messages using our parsers: [JSON](pipeline/parsers/json.md), [Regex](pipeline/parsers/regular-expression.md), [LTSV](pipeline/parsers/ltsv.md) and [Logfmt](pipeline/parsers/logfmt.md)
-* Metrics Collection (Prometheus compatible)
+* Metrics Support: Prometheus and OpenTelemetry compatible
 * Reliability and Data Integrity
   * [Backpressure](administration/backpressure.md) Handling
   * [Data Buffering](administration/buffering-and-storage.md) in memory and file system
 * Networking
   * Security: built-in TLS/SSL support
   * Asynchronous I/O
-* Pluggable Architecture and [Extensibility](development/library\_api.md): Inputs, Filters and Outputs
-  * More than 80 built-in plugins available
+* Pluggable Architecture and [Extensibility](development/library_api.md): Inputs, Filters and Outputs
+  * More than 100 built-in plugins are available
   * Extensibility
     * Write any input, filter or output plugin in C language
     * WASM: [WASM Filter Plugins](development/wasm-filter-plugins.md) or [WASM Input Plugins](development/wasm-input-plugins.md)
@@ -35,4 +35,6 @@ description: High Performance Log and Metrics Processor
 
 ## Fluent Bit, Fluentd and CNCF
 
-[Fluent Bit](http://fluentbit.io) is a [CNCF](https://cncf.io) sub-project under the umbrella of [Fluentd](http://fluentd.org), it's licensed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0). This project was originally created by [Treasure Data](https://www.treasuredata.com) and is currently a **vendor neutral** and community driven project.
+[Fluent Bit](http://fluentbit.io) is a [CNCF](https://cncf.io) **graduated** sub-project under the umbrella of [Fluentd](http://fluentd.org). Fluent Bit is licensed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
+
+Fluent Bit was originally created by [Eduardo Silva](https://www.linkedin.com/in/edsiper/). As a CNCF-hosted project, it is a fully **vendor-neutral** and community-driven project.
diff --git a/SUMMARY.md b/SUMMARY.md
@@ -1,6 +1,6 @@
 # Table of contents
 
-* [Fluent Bit v2.0 Documentation](README.md)
+* [Fluent Bit v2.2 Documentation](README.md)
 
 ## About
 
@@ -59,16 +59,17 @@
     * [Configuration File](administration/configuring-fluent-bit/yaml/configuration-file.md)
   * [Unit Sizes](administration/configuring-fluent-bit/unit-sizes.md)
   * [Multiline Parsing](administration/configuring-fluent-bit/multiline-parsing.md)
-* [Security](administration/security.md)
+* [Transport Security](administration/transport-security.md)
 * [Buffering & Storage](administration/buffering-and-storage.md)
 * [Backpressure](administration/backpressure.md)
 * [Scheduling and Retries](administration/scheduling-and-retries.md)
 * [Networking](administration/networking.md)
 * [Memory Management](administration/memory-management.md)
 * [Monitoring](administration/monitoring.md)
 * [HTTP Proxy](administration/http-proxy.md)
+* [Hot Reload](administration/hot-reload.md)
 * [Troubleshooting](administration/troubleshooting.md)
-  
+
 ## Local Testing
 
 * [Validating your Data and Structure](local-testing/validating-your-data-and-structure.md)
@@ -84,30 +85,37 @@
   * [Docker Log Based Metrics](pipeline/inputs/docker-metrics.md)
   * [Docker Events](pipeline/inputs/docker-events.md)
   * [Dummy](pipeline/inputs/dummy.md)
+  * [Elasticsearch](pipeline/inputs/elasticsearch.md)
   * [Exec](pipeline/inputs/exec.md)
   * [Exec Wasi](pipeline/inputs/exec-wasi.md)
   * [Fluent Bit Metrics](pipeline/inputs/fluentbit-metrics.md)
   * [Forward](pipeline/inputs/forward.md)
   * [Head](pipeline/inputs/head.md)
   * [HTTP](pipeline/inputs/http.md)
   * [Health](pipeline/inputs/health.md)
+  * [Kafka](pipeline/inputs/kafka.md)
   * [Kernel Logs](pipeline/inputs/kernel-logs.md)
+  * [Kubernetes Events](pipeline/inputs/kubernetes-events.md)
   * [Memory Metrics](pipeline/inputs/memory-metrics.md)
   * [MQTT](pipeline/inputs/mqtt.md)
   * [Network I/O Log Based Metrics](pipeline/inputs/network-io-metrics.md)
   * [NGINX Exporter Metrics](pipeline/inputs/nginx.md)
   * [Node Exporter Metrics](pipeline/inputs/node-exporter-metrics.md)
+  * [Podman Metrics](pipeline/inputs/podman-metrics.md)
   * [Process Log Based Metrics](pipeline/inputs/process.md)
+  * [Process Exporter Metrics](pipeline/inputs/process-exporter-metrics.md)
   * [Prometheus Scrape Metrics](pipeline/inputs/prometheus-scrape-metrics.md)
   * [Random](pipeline/inputs/random.md)
   * [Serial Interface](pipeline/inputs/serial-interface.md)
+  * [Splunk](pipeline/inputs/splunk.md)
   * [Standard Input](pipeline/inputs/standard-input.md)
   * [StatsD](pipeline/inputs/statsd.md)
   * [Syslog](pipeline/inputs/syslog.md)
   * [Systemd](pipeline/inputs/systemd.md)
   * [Tail](pipeline/inputs/tail.md)
   * [TCP](pipeline/inputs/tcp.md)
   * [Thermal](pipeline/inputs/thermal.md)
+  * [UDP](pipeline/inputs/udp.md)
   * [OpenTelemetry](pipeline/inputs/opentelemetry.md)
   * [Windows Event Log](pipeline/inputs/windows-event-log.md)
   * [Windows Event Log (winevtlog)](pipeline/inputs/windows-event-log-winevtlog.md)
@@ -127,6 +135,7 @@
   * [GeoIP2 Filter](pipeline/filters/geoip2-filter.md)
   * [Grep](pipeline/filters/grep.md)
   * [Kubernetes](pipeline/filters/kubernetes.md)
+  * [Log to Metrics](pipeline/filters/log_to_metrics.md)
   * [Lua](pipeline/filters/lua.md)
   * [Parser](pipeline/filters/parser.md)
   * [Record Modifier](pipeline/filters/record-modifier.md)
@@ -136,7 +145,9 @@
   * [Nightfall](pipeline/filters/nightfall.md)
   * [Rewrite Tag](pipeline/filters/rewrite-tag.md)
   * [Standard Output](pipeline/filters/standard-output.md)
+  * [Sysinfo](pipeline/filters/sysinfo.md)
   * [Throttle](pipeline/filters/throttle.md)
+  * [Type Converter](pipeline/filters/type-converter.md)
   * [Tensorflow](pipeline/filters/tensorflow.md)
   * [Wasm](pipeline/filters/wasm.md)
 * [Outputs](pipeline/outputs/README.md)
@@ -147,13 +158,15 @@
   * [Azure Blob](pipeline/outputs/azure\_blob.md)
   * [Azure Data Explorer](pipeline/outputs/azure\_kusto.md)
   * [Azure Log Analytics](pipeline/outputs/azure.md)
+  * [Azure Logs Ingestion API](pipeline/outputs/azure_logs_ingestion.md)
   * [Counter](pipeline/outputs/counter.md)
   * [Datadog](pipeline/outputs/datadog.md)
   * [Elasticsearch](pipeline/outputs/elasticsearch.md)
   * [File](pipeline/outputs/file.md)
   * [FlowCounter](pipeline/outputs/flowcounter.md)
   * [Forward](pipeline/outputs/forward.md)
   * [GELF](pipeline/outputs/gelf.md)
+  * [Google Chronicle](pipeline/outputs/chronicle.md)
   * [Google Cloud BigQuery](pipeline/outputs/bigquery.md)
   * [HTTP](pipeline/outputs/http.md)
   * [InfluxDB](pipeline/outputs/influxdb.md)
@@ -165,6 +178,7 @@
   * [New Relic](pipeline/outputs/new-relic.md)
   * [NULL](pipeline/outputs/null.md)
   * [Observe](pipeline/outputs/observe.md)
+  * [Oracle Log Analytics](pipeline/outputs/oci-logging-analytics.md)
   * [OpenSearch](pipeline/outputs/opensearch.md)
   * [OpenTelemetry](pipeline/outputs/opentelemetry.md)
   * [PostgreSQL](pipeline/outputs/postgresql.md)
@@ -178,6 +192,7 @@
   * [Syslog](pipeline/outputs/syslog.md)
   * [TCP & TLS](pipeline/outputs/tcp-and-tls.md)
   * [Treasure Data](pipeline/outputs/treasure-data.md)
+  * [Vivo Exporter](pipeline/outputs/vivo-exporter.md)
   * [WebSocket](pipeline/outputs/websocket.md)
 
 ## Stream Processing

diff --git a/about/fluentd-and-fluent-bit.md b/about/fluentd-and-fluent-bit.md
@@ -1,33 +1,33 @@
 ---
-description: The Production Grade Ecosystem
+description: The Production Grade Telemetry Ecosystem
 ---
 
 # Fluentd & Fluent Bit
 
-Logging and data processing in general can be complex, and at scale a bit more, that's why [Fluentd](https://www.fluentd.org) was born. Fluentd has become more than a simple tool, it has grown into a fullscale ecosystem that contains SDKs for different languages and sub-projects like [Fluent Bit](https://fluentbit.io).
+Telemetry data processing in general can be complex, and at scale a bit more, that's why [Fluentd](https://www.fluentd.org) was born. Fluentd has become more than a simple tool, it has grown into a fullscale ecosystem that contains SDKs for different languages and sub-projects like [Fluent Bit](https://fluentbit.io).
 
 On this page, we will describe the relationship between the [Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) open source projects, as a summary we can say both are:
 
 * Licensed under the terms of Apache License v2.0
-* Hosted projects by the [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io)
-* Production Grade solutions: deployed **thousands** of times every single day, **millions** per **month**.
-* Community driven projects
-* Widely Adopted by the Industry: trusted by all major companies like AWS, Microsoft, Google Cloud and hundred of others.
-* Originally created by [Treasure Data](https://www.treasuredata.com). 
+* **Graduated** Hosted projects by the [Cloud Native Computing Foundation (CNCF)](https://cncf.io)
+* Production Grade solutions: deployed **million** of times every single day.
+* **Vendor neutral** and community driven projects
+* Widely Adopted by the Industry: trusted by all major companies like AWS, Microsoft, Google Cloud and hundreds of others.
 
 Both projects share a lot of similarities, [Fluent Bit](https://fluentbit.io) is fully designed and built on top of the best ideas of [Fluentd](https://www.fluentd.org) architecture and general design. Choosing which one to use depends on the end-user needs.
 
-The following table describes a comparison in different areas of the projects:
-
-|  | Fluentd | Fluent Bit |
-| :--- | :--- | :--- |
-| Scope | Containers / Servers | Embedded Linux / Containers / Servers |
-| Language | C & Ruby | C |
-| Memory | ~40MB | ~650KB |
-| Performance | High Performance | High Performance |
-| Dependencies | Built as a Ruby Gem, it requires a certain number of gems. | Zero dependencies, unless some special plugin requires them. |
-| Plugins | More than 1000 plugins available | Around 70 plugins available |
-| License | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) |
-
-Both [Fluentd](https://www.fluentd.org) and [Fluent Bit](https://fluentbit.io) can work as Aggregators or Forwarders, they both can complement each other or use them as standalone solutions.
-
+The following table describes a comparison of different areas of the projects:
+
+|              | Fluentd                                                           | Fluent Bit                                                        |
+| ------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------- |
+| Scope        | Containers / Servers                                              | Embedded Linux / Containers / Servers                             |
+| Language     | C & Ruby                                                          | C                                                                 |
+| Memory       |  > 60MB                                                           | \~1MB                                                             |
+| Performance  | Medium Performance                                                | High Performance                                                  |
+| Dependencies | Built as a Ruby Gem, it requires a certain number of gems.        | Zero dependencies, unless some special plugin requires them.      |
+| Plugins      | More than 1000 external plugins are available                     | More than 100 built-in plugins are available                      |
+| License      | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) |
+
+Both [Fluentd](https://www.fluentd.org) and [Fluent Bit](https://fluentbit.io) can work as Aggregators or Forwarders, they both can complement each other or use them as standalone solutions.\
+\
+In the recent years, Cloud Providers switched from Fluentd to Fluent Bit for performance and compatibility reasons. Fluent Bit is now considered the **next generation** solution.
diff --git a/about/history.md b/about/history.md
@@ -4,7 +4,7 @@ description: Every project has a story
 
 # A Brief History of Fluent Bit
 
-On 2014, the [Fluentd](https://fluentd.org) team at [Treasure Data](https://www.treasuredata.com) forecasted the need of a lightweight log processor for constraint environments like Embedded Linux and Gateways, the project aimed to be part of the Fluentd Ecosystem and we called it [Fluent Bit](https://fluentbit.io), fully open source and available under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
+On 2014, the [Fluentd](https://www.fluentd.org/) team at [Treasure Data](https://www.treasuredata.com/) was forecasting the need for a lightweight log processor for constraint environments like Embedded Linux and Gateways, the project aimed to be part of the Fluentd Ecosystem; at that moment, Eduardo created [Fluent Bit](https://fluentbit.io/), a new open source solution written from scratch available under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).\
 
-After the project was around for some time, it got some traction in the Embedded market but we also started getting requests for several features from the Cloud community like more inputs, filters, and outputs. Not so long after that, Fluent Bit becomes one of the preferred solutions to solve the logging challenges in Cloud environments.
 
+After the project was around for some time, it got more traction for normal Linux systems, also with the new containerized world, the Cloud Native community asked to extend the project scope to support more sources, filters, and destinations. Not so long after, Fluent Bit became one of the preferred solutions to solve the logging challenges in Cloud environments.
diff --git a/about/license.md b/about/license.md
@@ -4,7 +4,7 @@ description: Strong Commitment to the Openness and Collaboration
 
 # License
 
-[Fluent Bit](http://fluentbit.io), including it core, plugins and tools are distributed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0):
+[Fluent Bit](http://fluentbit.io), including its core, plugins and tools are distributed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0):
 
 ```text
                                  Apache License