Merge branch 'master' into patch-3

Signed-off-by: Tanmaya Panda <[email protected]>

.gitignore

@@ -2,4 +2,4 @@ _book/*
 node_modules/*
 .gitignore
 old/*
-
+.DS_Store

README.md

@@ -2,46 +2,55 @@
 description: High Performance Telemetry Agent for Logs, Metrics and Traces
 ---
 
-# Fluent Bit v3.1 Documentation
+# Fluent Bit v3.2 Documentation
 
-<figure><img src=".gitbook/assets/3.0.png" alt=""><figcaption></figcaption></figure>
+<figure><img src=".gitbook/assets/3.1.png" alt=""><figcaption></figcaption></figure>
 
-[Fluent Bit](http://fluentbit.io) is a fast and lightweight **telemetry agent**
+[Fluent Bit](http://fluentbit.io) is a fast and lightweight telemetry agent
 for logs, metrics, and traces for Linux, macOS, Windows, and BSD family
-operating systems. It has been made with a strong focus on performance to allow
+operating systems. Fluent Bit has been made with a strong focus on performance to allow
 the collection and processing of telemetry data from different sources without
 complexity.![](https://static.scarf.sh/a.png?x-pxid=71f0e011-761f-4c6f-9a89-38817887faae)
 
 ## Features
 
-* High Performance: High throughput with low resources consumption
-* Data Parsing
-  * Convert your unstructured messages using our parsers: [JSON](pipeline/parsers/json.md), [Regex](pipeline/parsers/regular-expression.md), [LTSV](pipeline/parsers/ltsv.md) and [Logfmt](pipeline/parsers/logfmt.md)
-* Metrics Support: Prometheus and OpenTelemetry compatible
-* Reliability and Data Integrity
-  * [Backpressure](administration/backpressure.md) Handling
-  * [Data Buffering](administration/buffering-and-storage.md) in memory and file system
-* Networking
-  * Security: built-in TLS/SSL support
-  * Asynchronous I/O
-* Pluggable Architecture and [Extensibility](development/library_api.md): Inputs, Filters and Outputs
-  * Connect nearly any source to nearly any destination using preexisting plugins
-  * Extensibility
-    * Write any input, filter or output plugin in C language
-    * WASM: [WASM Filter Plugins](development/wasm-filter-plugins.md) or [WASM Input Plugins](development/wasm-input-plugins.md)
-    * Bonus: write [Filters in Lua](pipeline/filters/lua.md) or [Output plugins in Golang](development/golang-output-plugins.md)
-* [Monitoring](administration/monitoring.md): expose internal metrics over HTTP in JSON and [Prometheus](https://prometheus.io/) format
-* [Stream Processing](stream-processing/introduction.md): Perform data selection and transformation using simple SQL queries
-  * Create new streams of data using query results
-  * Aggregation Windows
-  * Data analysis and prediction: Timeseries forecasting
-* Portable: runs on Linux, macOS, Windows and BSD systems
+- High performance: High throughput with low resources consumption
+- Data parsing
+  - Convert your unstructured messages using our parsers:
+    [JSON](pipeline/parsers/json.md),
+    [Regex](pipeline/parsers/regular-expression.md),
+    [LTSV](pipeline/parsers/ltsv.md) and [Logfmt](pipeline/parsers/logfmt.md)
+- Metrics support: Prometheus and OpenTelemetry compatible
+- Reliability and data integrity
+  - [Backpressure](administration/backpressure.md) handling
+  - [Data buffering](administration/buffering-and-storage.md) in memory and file system
+- Networking
+  - Security: Built-in TLS/SSL support
+  - Asynchronous I/O
+- Pluggable architecture and [extensibility](development/library_api.md): Inputs,
+  Filters and Outputs:
+  - Connect nearly any source to nearly any destination using preexisting plugins
+  - Extensibility:
+    - Write input, filter, or output plugins in the C language
+    - WASM: [WASM Filter Plugins](development/wasm-filter-plugins.md) or
+      [WASM Input Plugins](development/wasm-input-plugins.md)
+    - Write [Filters in Lua](pipeline/filters/lua.md) or
+      [Output plugins in Golang](development/golang-output-plugins.md)
+- [Monitoring](administration/monitoring.md): Expose internal metrics over HTTP
+  in JSON and [Prometheus](https://prometheus.io/) format
+- [Stream Processing](stream-processing/introduction.md): Perform data selection
+  and transformation using simple SQL queries
+  - Create new streams of data using query results
+  - Aggregation windows
+  - Data analysis and prediction: Timeseries forecasting
+- Portable: Runs on Linux, macOS, Windows and BSD systems
 
 ## Fluent Bit, Fluentd and CNCF
 
-[Fluent Bit](http://fluentbit.io) is a [CNCF](https://cncf.io) **graduated** sub-project under the umbrella of [Fluentd](http://fluentd.org). Fluent Bit is licensed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
+[Fluent Bit](http://fluentbit.io) is a [CNCF](https://cncf.io) graduated sub-project
+under the umbrella of [Fluentd](http://fluentd.org). Fluent Bit is licensed under
+the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
 
 Fluent Bit was originally created by [Eduardo Silva](https://www.linkedin.com/in/edsiper/)
 and is now sponsored by [Chronosphere](https://chronosphere.io/). As a
-CNCF-hosted project, it is a fully **vendor-neutral** and community-driven
-project.
+CNCF-hosted project, it is a fully vendor-neutral and community-driven project.

SUMMARY.md

@@ -1,6 +1,6 @@
 # Table of contents
 
-* [Fluent Bit v3.1 Documentation](README.md)
+* [Fluent Bit v3.2 Documentation](README.md)
 
 ## About
 
@@ -67,6 +67,7 @@
 * [Networking](administration/networking.md)
 * [Memory Management](administration/memory-management.md)
 * [Monitoring](administration/monitoring.md)
+* [Multithreading](administration/multithreading.md)
 * [HTTP Proxy](administration/http-proxy.md)
 * [Hot Reload](administration/hot-reload.md)
 * [Troubleshooting](administration/troubleshooting.md)
@@ -131,7 +132,9 @@
   * [Decoders](pipeline/parsers/decoders.md)
 * [Processors](pipeline/processors/README.md)
   * [Content Modifier](pipeline/processors/content-modifier.md)
+  * [Labels](pipeline/processors/labels.md)
   * [Metrics Selector](pipeline/processors/metrics-selector.md)
+  * [OpenTelemetry Envelope](pipeline/processors/opentelemetry-envelope.md)
   * [SQL](pipeline/processors/sql.md)
 * [Filters](pipeline/filters/README.md)
   * [AWS Metadata](pipeline/filters/aws-metadata.md)
@@ -184,6 +187,7 @@
   * [NATS](pipeline/outputs/nats.md)
   * [New Relic](pipeline/outputs/new-relic.md)
   * [NULL](pipeline/outputs/null.md)
+  * [OpenObserve](pipeline/inputs/openobserve.md)
   * [Observe](pipeline/outputs/observe.md)
   * [Oracle Log Analytics](pipeline/outputs/oci-logging-analytics.md)
   * [OpenSearch](pipeline/outputs/opensearch.md)

about/fluentd-and-fluent-bit.md

@@ -1,33 +1,45 @@
 ---
-description: The Production Grade Telemetry Ecosystem
+description: The production grade telemetry ecosystem
 ---
 
-# Fluentd & Fluent Bit
+# Fluentd and Fluent Bit
 
-Telemetry data processing in general can be complex, and at scale a bit more, that's why [Fluentd](https://www.fluentd.org) was born. Fluentd has become more than a simple tool, it has grown into a fullscale ecosystem that contains SDKs for different languages and sub-projects like [Fluent Bit](https://fluentbit.io).
+Telemetry data processing can be complex, especially at scale. That's why
+[Fluentd](https://www.fluentd.org) was created. Fluentd is more than a simple tool,
+it's grown into a fullscale ecosystem that contains SDKs for different languages
+and subprojects like [Fluent Bit](https://fluentbit.io).
 
-On this page, we will describe the relationship between the [Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) open source projects, as a summary we can say both are:
+Here, we describe the relationship between the [Fluentd](http://fluentd.org)
+and [Fluent Bit](http://fluentbit.io) open source projects.
 
-* Licensed under the terms of Apache License v2.0
-* **Graduated** Hosted projects by the [Cloud Native Computing Foundation (CNCF)](https://cncf.io)
-* Production Grade solutions: deployed **million** of times every single day.
-* **Vendor neutral** and community driven projects
-* Widely Adopted by the Industry: trusted by all major companies like AWS, Microsoft, Google Cloud and hundreds of others.
+Both projects are:
 
-Both projects share a lot of similarities, [Fluent Bit](https://fluentbit.io) is fully designed and built on top of the best ideas of [Fluentd](https://www.fluentd.org) architecture and general design. Choosing which one to use depends on the end-user needs.
+- Licensed under the terms of Apache License v2.0.
+- Graduated hosted projects by the [Cloud Native Computing Foundation (CNCF)](https://cncf.io).
+- Production grade solutions: Deployed millions of times every single day.
+- Vendor neutral and community driven.
+- Widely adopted by the industry: Trusted by major companies like AWS, Microsoft,
+  Google Cloud, and hundreds of others.
+
+The projects have many similarities: [Fluent Bit](https://fluentbit.io) is
+designed and built on top of the best ideas of [Fluentd](https://www.fluentd.org)
+architecture and general design. Which one you choose depends on your end-users' needs.
 
 The following table describes a comparison of different areas of the projects:
 
-|              | Fluentd                                                           | Fluent Bit                                                        |
-| ------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------- |
-| Scope        | Containers / Servers                                              | Embedded Linux / Containers / Servers                             |
-| Language     | C & Ruby                                                          | C                                                                 |
-| Memory       |  > 60MB                                                           | \~1MB                                                             |
-| Performance  | Medium Performance                                                | High Performance                                                  |
-| Dependencies | Built as a Ruby Gem, it requires a certain number of gems.        | Zero dependencies, unless some special plugin requires them.      |
-| Plugins      | More than 1000 external plugins are available                     | More than 100 built-in plugins are available                      |
+|  Attribute   | Fluentd               | Fluent Bit            |
+| ------------ | --------------------- | --------------------- |
+| Scope        | Containers / Servers  | Embedded Linux / Containers / Servers |
+| Language     | C & Ruby              | C                                     |
+| Memory       | Greater than 60 MB | Approximately 1 MB |
+| Performance  | Medium Performance    | High Performance                      |
+| Dependencies | Built as a Ruby Gem, depends on other gems. | Zero dependencies, unless required by a plugin. |
+| Plugins      | Over 1,000 external plugins available. | Over 100 built-in plugins available. |
 | License      | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) |
 
-Both [Fluentd](https://www.fluentd.org) and [Fluent Bit](https://fluentbit.io) can work as Aggregators or Forwarders, they both can complement each other or use them as standalone solutions.\
-\
-In the recent years, Cloud Providers switched from Fluentd to Fluent Bit for performance and compatibility reasons. Fluent Bit is now considered the **next generation** solution.
+Both [Fluentd](https://www.fluentd.org) and [Fluent Bit](https://fluentbit.io)
+can work as Aggregators or Forwarders, and can complement each other or be used
+as standalone solutions.
+
+In the recent years, cloud providers have switched from Fluentd to Fluent Bit for
+performance and compatibility. Fluent Bit is now considered the next-generation solution.

administration/backpressure.md

@@ -1,5 +1,7 @@
 # Backpressure
 
+<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=63e37cfe-9ce3-4a18-933a-76b9198958c1" />
+
 Under certain scenarios it is possible for logs or data to be ingested or created faster than the ability to flush it to some destinations. One such common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates backpressure leading to high memory consumption in the service.
 
 In order to avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data that an input plugin can ingest, this is done through the configuration parameters **Mem\_Buf\_Limit** and **storage.Max\_Chunks\_Up**.

administration/buffering-and-storage.md

@@ -1,5 +1,7 @@
 # Buffering & Storage
 
+<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=cde12327-09ed-409c-ac02-7c0afa5eff51" />
+
 The end-goal of [Fluent Bit](https://fluentbit.io) is to collect, parse, filter and ship logs to a central place. In this workflow there are many phases and one of the critical pieces is the ability to do _buffering_ : a mechanism to place processed data into a temporary location until is ready to be shipped.
 
 By default when Fluent Bit processes data, it uses Memory as a primary and temporary place to store the records, but there are certain scenarios where it would be ideal to have a persistent buffering mechanism based in the filesystem to provide aggregation and data safety capabilities.

administration/configuring-fluent-bit/classic-mode/configuration-file.md

@@ -4,6 +4,8 @@ description: This page describes the main configuration file used by Fluent Bit
 
 # Configuration File
 
+<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=5e67142e-3887-4b56-b940-18494bcc23a7" />
+
 One of the ways to configure Fluent Bit is using a main configuration file. Fluent Bit allows to use one configuration file which works at a global scope and uses the [Format and Schema](format-schema.md) defined previously.
 
 The main configuration file supports four types of sections:
@@ -137,12 +139,6 @@ The following configuration file example demonstrates how to collect CPU metrics
     Match my*cpu
 ```
 
-## Visualize <a href="config_include_file" id="config_include_file"></a>
-
-You can also visualize Fluent Bit INPUT, FILTER, and OUTPUT configuration via [Calyptia](https://calyptia.com/free-trial)
-
-![](../../../.gitbook/assets/image.png)
-
 ## Include File <a href="config_include_file" id="config_include_file"></a>
 
 To avoid complicated long configuration files is better to split specific parts in different files and call them (include) from one main file.
@@ -167,5 +163,5 @@ Wildcard character (\*) is supported to include multiple files, e.g:
 @INCLUDE input_*.conf
 ```
 
-Note files matching the wildcard character are included unsorted. 
+Note files matching the wildcard character are included unsorted.
 If plugins ordering between files need to be preserved, the files should be included explicitly.

administration/configuring-fluent-bit/classic-mode/variables.md

@@ -1,5 +1,7 @@
 # Variables
 
+<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=1731c7b5-34c6-424f-bfc6-88c2aa71e81f" />
+
 Fluent Bit supports the usage of environment variables in any value associated to a key when using a configuration file.
 
 The variables are case sensitive and can be used in the following format:

administration/configuring-fluent-bit/multiline-parsing.md

@@ -2,6 +2,8 @@
 
 In an ideal world, applications might log their messages within a single line, but in reality applications generate multiple log messages that sometimes belong to the same context. But when is time to process such information it gets really complex. Consider application stack traces which always have multiple log lines.
 
+<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=e19a4c14-a9e4-4163-8f3a-52196eb9a585" />
+
 Starting from Fluent Bit v1.8, we have implemented a unified Multiline core functionality to solve all the user corner cases. In this section, you will learn about the features and configuration options available.
 
 ## Concepts

administration/configuring-fluent-bit/yaml/configuration-file.md

@@ -2,6 +2,8 @@
 description: This page describes the yaml configuration file used by Fluent Bit
 ---
 
+<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=864c6f0e-8977-4838-8772-84416943548e" />
+
 # YAML Configuration File
 
 One of the ways to configure Fluent Bit is using a YAML configuration file that works at a global scope.

administration/memory-management.md

@@ -1,5 +1,7 @@
 # Memory Management
 
+<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=5cc3ce54-e910-4ebf-85f5-f02530b3e11b" />
+
 In certain scenarios it would be ideal to estimate how much memory Fluent Bit could be using, this is very useful for containerized environments where memory limits are a must.
 
 In order to that we will assume that the input plugins have set the **Mem\_Buf\_Limit** option \(you can learn more about it in the [Backpressure](backpressure.md) section\).