GitBook: [1.4-dev] 86 pages and 11 assets modified

README.md

@@ -8,7 +8,7 @@ description: Next generation Log and Stream processor
 
 [Fluent Bit](http://fluentbit.io) is a Fast and Lightweight Log Processor, Stream Processor and Forwarder for Linux, OSX, Windows and BSD family operating systems. It has been made with a strong focus on performance to allow the collection of events from different sources without complexity.
 
-### Features
+## Features
 
 * High Performance
 * Data Parsing
@@ -25,17 +25,15 @@ description: Next generation Log and Stream processor
     * Write any input, filter or output plugin in C language
     * Bonus: write [Filters in Lua](pipeline/filters/lua.md) or [Output plugins in Golang](development/golang-output-plugins.md)
 * [Monitoring](administration/monitoring.md): expose internal metrics over HTTP in JSON and [Prometheus](https://prometheus.io/) format
-* [Stream Processing](stream-processing/untitled.md): Perform data selection and transformation using simple SQL queries
+* [Stream Processing](): Perform data selection and transformation using simple SQL queries
   * Create new streams of data using query results
   * Aggregation Windows
   * Data analysis and prediction: Timeseries forecasting
 * Portable: runs on Linux, MacOS, Windows and BSD systems
 
-### Fluent Bit, Fluentd and CNCF
+## Fluent Bit, Fluentd and CNCF
 
 [Fluent Bit](http://fluentbit.io) is a sub-component of the [Fluentd](http://fluentd.org) project ecosystem, it's licensed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0). This project was created by [Treasure Data](https://www.treasuredata.com) and is its current primary sponsor.
 
 Nowadays Fluent Bit get contributions from several companies and individuals and same as [Fluentd](https://www.fluentd.org), it's hosted as a [CNCF](https://cncf.io) subproject.
 
-
-

SUMMARY.md

@@ -44,7 +44,7 @@
 ## Administration
 
 * [Configuring Fluent Bit](administration/configuring-fluent-bit/README.md)
-  * [Files Schema / Structure](administration/configuring-fluent-bit/files-schema-structure/README.md)
+  * [Files Schema / Structure](administration/configuring-fluent-bit/files-schema-structure.md)
   * [Variables](administration/configuring-fluent-bit/variables.md)
   * [Commands](administration/configuring-fluent-bit/commands.md)
   * [Configuration File](administration/configuring-fluent-bit/configuration-file.md)
@@ -65,7 +65,7 @@
   * [Disk I/O Metrics](pipeline/inputs/disk-io-metrics.md)
   * [Dummy](pipeline/inputs/dummy.md)
   * [Exec](pipeline/inputs/exec.md)
-  * [Forward](pipeline/inputs/forward.md)
+  * [Forward](pipeline/outputs/forward.md)
   * [Head](pipeline/inputs/head.md)
   * [Health](pipeline/inputs/health.md)
   * [Kernel Logs](pipeline/inputs/kernel-logs.md)
@@ -96,7 +96,7 @@
   * [Parser](pipeline/filters/parser.md)
   * [Record Modifier](pipeline/filters/record-modifier.md)
   * [Rewrite Tag](pipeline/filters/rewrite-tag.md)
-  * [Standard Output](pipeline/filters/standard-output.md)
+  * [Standard Output](pipeline/outputs/standard-output.md)
   * [Throttle](pipeline/filters/throttle.md)
   * [Nest](pipeline/filters/nest.md)
   * [Modify](pipeline/filters/modify.md)
@@ -124,7 +124,7 @@
 
 ## Stream Processing
 
-* [Introduction](stream-processing/README.md)
+* [Introduction](stream-processing/stream-processing.md)
 * [Overview](stream-processing/overview.md)
 * [Changelog](stream-processing/changelog.md)
 * [Getting Started](stream-processing/getting-started/README.md)
@@ -137,3 +137,4 @@
 * [C Library API](development/library_api.md)
 * [Ingest Records Manually](development/ingest-records-manually.md)
 * [Golang Output Plugins](development/golang-output-plugins.md)
+

about/history.md

@@ -4,7 +4,7 @@ description: Every project has a story
 
 # A Brief History of Fluent Bit
 
-On 2014,  the [Fluentd](https://fluentd.org) team at [Treasure Data](https://www.treasuredata.com) forecasted the need of a lightweight log processor for constraint environments like Embedded Linux and Gateways, the project aimed to be part of the Fluentd Ecosystem and we called it [Fluent Bit](https://fluentbit.io), fully open source and available under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
+On 2014, the [Fluentd](https://fluentd.org) team at [Treasure Data](https://www.treasuredata.com) forecasted the need of a lightweight log processor for constraint environments like Embedded Linux and Gateways, the project aimed to be part of the Fluentd Ecosystem and we called it [Fluent Bit](https://fluentbit.io), fully open source and available under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
 
 After the project was around for some time, it got some traction in the Embedded market but we also started getting requests for several features from the Cloud community like more inputs, filters, and outputs. Not so long after that, Fluent Bit becomes one of the preferred solutions to solve the logging challenges in Cloud environments.
 

about/what-is-fluent-bit.md

@@ -1,6 +1,6 @@
 # What is Fluent Bit ?
 
-​[Fluent Bit](http://fluentbit.io/) is an open source and multi-platform log processor tool which aims to be a generic Swiss knife for log processing  and distribution.
+​[Fluent Bit](http://fluentbit.io/) is an open source and multi-platform log processor tool which aims to be a generic Swiss knife for log processing and distribution.
 
 Nowadays the number of sources of information in our environments is ever increasing. Handling data collection at scale is complex, and collecting and aggregating diverse data requires a specialized tool that can deal with:
 
@@ -10,7 +10,5 @@ Nowadays the number of sources of information in our environments is ever increa
 * Security
 * Multiple destinations
 
-[Fluent Bit](https://fluentbit.io) has been designed with performance and low resources consumption in mind. 
-
-
+[Fluent Bit](https://fluentbit.io) has been designed with performance and low resources consumption in mind.
 

administration/backpressure.md

@@ -34,5 +34,5 @@ After some seconds if the scheduler was able to flush the initial 700KB of data
 
 Each plugin is independent and not all of them implements the **pause** and **resume** callbacks. As said, these callbacks are just a notification mechanism for the plugin.
 
-The plugin who implements and keep a good state is the [Tail Input](../input/tail.md) plugin. When the **pause** callback is triggered, it stop their collectors and stop appending data. Upon **resume**, it re-enable the collectors.
+The plugin who implements and keep a good state is the [Tail Input](https://github.com/fluent/fluent-bit-docs/tree/b78cfe98123e74e165f2b6669229da009258f34e/input/tail.md) plugin. When the **pause** callback is triggered, it stop their collectors and stop appending data. Upon **resume**, it re-enable the collectors.
 

administration/buffering-and-storage.md

@@ -1,32 +1,32 @@
-# Fluent Bit and Buffering
+# Buffering & Storage
 
 The end-goal of [Fluent Bit](https://fluentbit.io) is to collect, parse, filter and ship logs to a central place. In this workflow there are many phases and one of the critical pieces is the ability to do _buffering_ : a mechanism to place processed data into a temporal location until is ready to be shipped.
 
-By default when Fluent Bit process data, it uses Memory as a primary and temporal place to store the record logs, but there are certain scenarios where would be ideal to have a persistent buffering mechanism based in the filesystem  to provide aggregation and data safety capabilities.
+By default when Fluent Bit process data, it uses Memory as a primary and temporal place to store the record logs, but there are certain scenarios where would be ideal to have a persistent buffering mechanism based in the filesystem to provide aggregation and data safety capabilities.
 
 Starting with Fluent Bit v1.0, we introduced a new _storage layer_ that can either work in memory or in the file system. Input plugins can be configured to use one or the other upon demand at start time.
 
 ## Configuration
 
 The storage layer configuration takes place in two areas:
 
-- Service Section
-- Input Section
+* Service Section
+* Input Section
 
 The known Service section configure a global environment for the storage layer, and then in the Input sections defines which mechanism to use.
 
 ### Service Section Configuration
 
-| Key                       | Description                                                  | Default |
-| ------------------------- | ------------------------------------------------------------ | ------- |
-| storage.path              | Set an optional location in the file system to store streams and chunks of data. If this parameter is not set, Input plugins can only use in-memory buffering. |         |
-| storage.sync              | Configure the synchronization mode used to store the data into the file system. It can take the values _normal_ or _full_. | normal  |
-| storage.checksum          | Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. | Off     |
-| storage.backlog.mem_limit | If _storage.path_ is set, Fluent Bit will look for data chunks that were not delivered and are still in the storage layer, these are called _backlog_ data. This option configure a hint of maximum value of memory to use when processing these records. | 5M      |
+| Key | Description | Default |
+| :--- | :--- | :--- |
+| storage.path | Set an optional location in the file system to store streams and chunks of data. If this parameter is not set, Input plugins can only use in-memory buffering. |  |
+| storage.sync | Configure the synchronization mode used to store the data into the file system. It can take the values _normal_ or _full_. | normal |
+| storage.checksum | Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. | Off |
+| storage.backlog.mem\_limit | If _storage.path_ is set, Fluent Bit will look for data chunks that were not delivered and are still in the storage layer, these are called _backlog_ data. This option configure a hint of maximum value of memory to use when processing these records. | 5M |
 
 a Service section will look like this:
 
-```
+```text
 [SERVICE]
     flush                     1
     log_Level                 info
@@ -36,31 +36,31 @@ a Service section will look like this:
     storage.backlog.mem_limit 5M
 ```
 
-that configuration configure an optional buffering mechanism where it root for data is _/var/log/flb-storage/_, it will use _normal_ synchronization mode, without checksum and up to a maximum of 5MB of memory when processing backlog data. 
+that configuration configure an optional buffering mechanism where it root for data is _/var/log/flb-storage/_, it will use _normal_ synchronization mode, without checksum and up to a maximum of 5MB of memory when processing backlog data.
 
 ### Input Section Configuration
 
 Optionally, any Input plugin can configure their storage preference, the following table describe the options available:
 
-| Key          | Description                                                  | Default |
-| ------------ | ------------------------------------------------------------ | ------- |
-| storage.type | Specify the buffering mechanism to use. It can be _memory_ or _filesystem_. | memory  |
+| Key | Description | Default |
+| :--- | :--- | :--- |
+| storage.type | Specify the buffering mechanism to use. It can be _memory_ or _filesystem_. | memory |
 
-The following example configure a service that offers filesystem buffering capabilities and two Input plugins  being the first based in memory and the second with the filesystem:
+The following example configure a service that offers filesystem buffering capabilities and two Input plugins being the first based in memory and the second with the filesystem:
 
-```
+```text
 [SERVICE]
     flush                     1
     log_Level                 info
     storage.path              /var/log/flb-storage/
     storage.sync              normal
     storage.checksum          off
     storage.backlog.mem_limit 5M
-    
+
 [INPUT]
     name          cpu
     storage.type  filesystem
-    
+
 [INPUT]
     name          mem
     storage.type  memory

administration/configuring-fluent-bit/commands.md

@@ -1,4 +1,4 @@
-# Configuration Commands
+# Commands
 
 Configuration files must be flexible enough for any deployment need, but they must keep a clean and readable format.
 
@@ -9,7 +9,7 @@ Fluent Bit _Commands_ extends a configuration file with specific built-in featur
 | [@INCLUDE](commands.md#cmd_include) | @INCLUDE FILE | Include a configuration file |
 | [@SET](commands.md#cmd_set) | @SET KEY=VAL | Set a configuration variable |
 
-## @INCLUDE Command {#cmd_include}
+## @INCLUDE Command <a id="cmd_include"></a>
 
 Configuring a logging pipeline might lead to an extensive configuration file. In order to maintain a human-readable configuration, it's suggested to split the configuration in multiple files.
 
@@ -60,7 +60,7 @@ Note that despites the order of inclusion, Fluent Bit will **ALWAYS** respect th
 * Filters
 * Outputs
 
-## @SET Command {#cmd_set}
+## @SET Command <a id="cmd_set"></a>
 
 Fluent Bit supports [configuration variables](variables.md), one way to expose this variables to Fluent Bit is through setting a Shell environment variable, the other is through the _@SET_ command.
 

administration/configuring-fluent-bit/configuration-file.md

@@ -6,16 +6,16 @@ Fluent Bit allows to use one configuration file which works at a global scope an
 
 The configuration file supports four types of sections:
 
-* [Service](file.md#config_section)
-* [Input](file.md#config_input)
-* [Filter](file.md#config_filter)
-* [Output](file.md#config_output)
+* [Service](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/file.md#config_section)
+* [Input](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/file.md#config_input)
+* [Filter](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/file.md#config_filter)
+* [Output](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/file.md#config_output)
 
 In addition there is an additional feature to include external files:
 
-* [Include File](file.md#config_include_file)
+* [Include File](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/file.md#config_include_file)
 
-## Service {#config_section}
+## Service <a id="config_section"></a>
 
 The _Service_ section defines global properties of the service, the keys available as of this version are described in the following table:
 
@@ -26,12 +26,12 @@ The _Service_ section defines global properties of the service, the keys availab
 | Log\_File | Absolute path for an optional log file. |  |
 | Log\_Level | Set the logging verbosity level. Allowed values are: error, info, debug and trace. Values are accumulative, e.g: if 'debug' is set, it will include error, info and debug. Note that _trace_ mode is only available if Fluent Bit was built with the _WITH\_TRACE_ option enabled. | info |
 | Parsers\_File | Path for a _parsers_ configuration file. Multiple Parsers\_File entries can be used. |  |
-| Plugins\_File | Path for a _plugins_ configuration file. A _plugins_ configuration file allows to define paths for external plugins, for an example [see here](https://github.com/fluent/fluent-bit/blob/master/conf/plugins.conf). ||
-| Streams\_File | Path for the Stream Processor configuration file. For details about the format of SP configuration file [see here](stream_processor.md). ||
+| Plugins\_File | Path for a _plugins_ configuration file. A _plugins_ configuration file allows to define paths for external plugins, for an example [see here](https://github.com/fluent/fluent-bit/blob/master/conf/plugins.conf). |  |
+| Streams\_File | Path for the Stream Processor configuration file. For details about the format of SP configuration file [see here](https://github.com/fluent/fluent-bit-docs/tree/5f926fd1330690179b8c1edab90d672699599ec7/administration/configuring-fluent-bit/stream_processor.md). |  |
 | HTTP\_Server | Enable built-in HTTP Server | Off |
 | HTTP\_Listen | Set listening interface for HTTP Server when it's enabled | 0.0.0.0 |
 | HTTP\_Port | Set TCP Port for the HTTP Server | 2020 |
-| Coro_Stack_Size | Set the coroutines stack size in bytes. The value must be greater than the page size of the running system. Don't set too small value (say 4096), or coroutine threads can overrun the stack buffer. | 24576 |
+| Coro\_Stack\_Size | Set the coroutines stack size in bytes. The value must be greater than the page size of the running system. Don't set too small value \(say 4096\), or coroutine threads can overrun the stack buffer. | 24576 |
 
 ### Example
 
@@ -44,7 +44,7 @@ The following is an example of a _SERVICE_ section:
     Log_Level       debug
 ```
 
-## Input {#config_input}
+## Input <a id="config_input"></a>
 
 An _INPUT_ section defines a source \(related to an input plugin\), here we will describe the base configuration for each _INPUT_ section. Note that each input plugin may add it own configuration keys:
 
@@ -65,17 +65,17 @@ The following is an example of an _INPUT_ section:
     Tag  my_cpu
 ```
 
-## Filter {#config_filter}
+## Filter <a id="config_filter"></a>
 
 A _FILTER_ section defines a filter \(related to an filter plugin\), here we will describe the base configuration for each _FILTER_ section. Note that each filter plugin may add it own configuration keys:
 
 | Key | Description |  |
 | :--- | :--- | :--- |
 | Name | Name of the filter plugin. |  |
 | Match | A pattern to match against the tags of incoming records. It's case sensitive and support the star \(\*\) character as a wildcard. |  |
-| Match_Regex | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regex syntax. |  |
+| Match\_Regex | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regex syntax. |  |
 
-The _Name_ is mandatory and it let Fluent Bit know which filter plugin should be loaded. The _Match_ or _Match_Regex_ is mandatory for all plugins. If both are specified, _Match_Regex_ takes precedence.
+The _Name_ is mandatory and it let Fluent Bit know which filter plugin should be loaded. The _Match_ or _Match\_Regex_ is mandatory for all plugins. If both are specified, _Match\_Regex_ takes precedence.
 
 ### Example
 
@@ -87,15 +87,15 @@ The following is an example of an _FILTER_ section:
     Match *
 ```
 
-## Output {#config_output}
+## Output <a id="config_output"></a>
 
 The _OUTPUT_ section specify a destination that certain records should follow after a Tag match. The configuration support the following keys:
 
-| Key | Description |
-| :--- | :--- |
-| Name | Name of the output plugin. |
+| Key | Description |  |
+| :--- | :--- | :--- |
+| Name | Name of the output plugin. |  |
 | Match | A pattern to match against the tags of incoming records. It's case sensitive and support the star \(\*\) character as a wildcard. |  |
-| Match_Regex | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regex syntax. |  |
+| Match\_Regex | A regular expression to match against the tags of incoming records. Use this option if you want to use the full regex syntax. |  |
 
 ### Example
 
@@ -126,7 +126,7 @@ The following configuration file example demonstrates how to collect CPU metrics
     Match my*cpu
 ```
 
-## Include File {#config_include_file}
+## Include File <a id="config_include_file"></a>
 
 To avoid complicated long configuration files is better to split specific parts in different files and call them \(include\) from one main file.
 
@@ -149,3 +149,4 @@ Wildcard character \(\*\) is supported to include multiple files, e.g:
 ```text
 @INCLUDE input_*.conf
 ```
+