Merge branch 'fluent:master' into master

fluent · Sep 18, 2023 · 076f17d · 076f17d
2 parents 2d28c80 + d9de2b2
commit 076f17d
Show file tree

Hide file tree

Showing 15 changed files with 188 additions and 17 deletions.
diff --git a/.gitbook.yaml b/.gitbook.yaml
@@ -30,6 +30,7 @@ redirects:
     input/tail:            ./pipeline/inputs/tail.md
     input/tcp:             ./pipeline/inputs/tcp.md
     input/thermal:         ./pipeline/inputs/thermal.md
+    input/udp:             ./pipeline/inputs/udp.md
     input/winlog:          ./pipeline/inputs/windows-event-log.md
     input/winevtlog:       ./pipeline/inputs/windows-event-log-winevtlog.md
 

diff --git a/SUMMARY.md b/SUMMARY.md
@@ -114,6 +114,7 @@
   * [Tail](pipeline/inputs/tail.md)
   * [TCP](pipeline/inputs/tcp.md)
   * [Thermal](pipeline/inputs/thermal.md)
+  * [UDP](pipeline/inputs/udp.md)
   * [OpenTelemetry](pipeline/inputs/opentelemetry.md)
   * [Windows Event Log](pipeline/inputs/windows-event-log.md)
   * [Windows Event Log (winevtlog)](pipeline/inputs/windows-event-log-winevtlog.md)

diff --git a/administration/configuring-fluent-bit/classic-mode/configuration-file.md b/administration/configuring-fluent-bit/classic-mode/configuration-file.md
@@ -138,7 +138,7 @@ The following configuration file example demonstrates how to collect CPU metrics
 
 ## Visualize <a href="config_include_file" id="config_include_file"></a>
 
-You can also visualize Fluent Bit INPUT, FILTER, and OUTPUT configuration via [https://cloud.calyptia.com](https://cloud.calyptia.com/visualizer)
+You can also visualize Fluent Bit INPUT, FILTER, and OUTPUT configuration via [Calyptia](https://calyptia.com/free-trial)
 
 ![](../../../.gitbook/assets/image.png)
 

diff --git a/administration/monitoring.md b/administration/monitoring.md
@@ -322,18 +322,18 @@ If (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 seconds is TRUE, t
 If (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 seconds is FALSE, then it's healthy.
 
 
-## Calyptia Cloud
+## Calyptia
 
-[Calyptia Cloud](https://cloud.calyptia.com) is a hosted service that allows you to monitor your Fluent Bit agents including data flow, metrics and configurations.
+[Calyptia](https://calyptia.com/free-trial) is a hosted service that allows you to monitor your Fluent Bit agents including data flow, metrics and configurations.
 
 ![](../.gitbook/assets/image-19-.png)
 
 ### Get Started with Calyptia Cloud
 
 Register your Fluent Bit agent will take **less than one minute**, steps:
 
-* Go to [cloud.calyptia.com](https://cloud.calyptia.com) and sign-in
-* On the left menu click on [Settings](https://cloud.calyptia.com/settings) and generate/copy your API key
+* Go to the calyptia core console and sign-in
+* On the left menu click on settings and generate/copy your API key
 
 In your Fluent Bit configuration file, append the following configuration section:
 

diff --git a/installation/docker.md b/installation/docker.md
@@ -17,6 +17,8 @@ The following table describes the Linux container tags that are available on Doc
 
 | Tag(s)       | Manifest Architectures    | Description                                                    |
 | ------------ | ------------------------- | -------------------------------------------------------------- |
+| 2.1.9-debug | x86_64, arm64v8, arm32v7 | Debug images |
+| 2.1.9 | x86_64, arm64v8, arm32v7 | Release [v2.1.9](https://fluentbit.io/announcements/v2.1.9/) |
 | 2.1.8-debug | x86_64, arm64v8, arm32v7 | Debug images |
 | 2.1.8 | x86_64, arm64v8, arm32v7 | Release [v2.1.8](https://fluentbit.io/announcements/v2.1.8/) |
 | 2.1.7-debug | x86_64, arm64v8, arm32v7 | Debug images |

diff --git a/installation/kubernetes.md b/installation/kubernetes.md
@@ -151,7 +151,7 @@ spec:
 
 ### Configure Fluent Bit
 
-Assuming the basic volume configuration described above, you can apply the following config to start logging. You can visualize this configuration [here](https://link.calyptia.com/gzc)
+Assuming the basic volume configuration described above, you can apply the following config to start logging. You can visualize this configuration [here (Sign-up required)](https://calyptia.com/free-trial)
 
 ```yaml
 fluent-bit.conf: |

diff --git a/installation/sources/build-and-install.md b/installation/sources/build-and-install.md
@@ -149,6 +149,7 @@ The _input plugins_ provides certain features to gather information from a speci
 | [FLB\_IN\_TAIL](../../pipeline/inputs/tail.md) | Enable Tail \(follow files\) input plugin | On |
 | [FLB\_IN\_TCP](../../pipeline/inputs/tcp.md) | Enable TCP input plugin | On |
 | [FLB\_IN\_THERMAL](../../pipeline/inputs/thermal.md) | Enable system temperature\(s\) input plugin | On |
+| [FLB\_IN\_UDP](../../pipeline/inputs/udp.md) | Enable UDP input plugin | On |
 | [FLB\_IN\_WINLOG](../../pipeline/inputs/windows-event-log.md) | Enable Windows Event Log input plugin \(Windows Only\) | On |
 | [FLB\_IN\_WINEVTLOG](../../pipeline/inputs/windows-event-log-winevtlog.md) | Enable Windows Event Log input plugin using winevt.h API \(Windows Only\) | On |
 

diff --git a/installation/windows.md b/installation/windows.md
@@ -83,15 +83,15 @@ The latest stable version is 2.1.4. Each version is available on the Github rele
 
 | INSTALLERS                                                                                       | SHA256 CHECKSUMS                                                 |
 | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------- |
-| [fluent-bit-2.1.8-win32.exe](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win32.exe) | [5814f2da845dd27aedf2c805c90b85d8250223e455fe7139be8d307275a99548](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win32.exe.sha256) |
-| [fluent-bit-2.1.8-win32.zip](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win32.zip) | [b45062666de4629c68e6a91176cbe003ad0db8607921ce5e1f89e1f3c6ee9f0d](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win32.zip.sha256) |
-| [fluent-bit-2.1.8-win64.exe](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win64.exe) | [ac982def9a8cff6d302d61090ffde3f965ddf4a796900a66cdaa6e3ae687a621](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win64.exe.sha256) |
-| [fluent-bit-2.1.8-win64.zip](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win64.zip) | [d18e2a0e1f5b649bf0907cfc51ea0e643938a4a4ed70624b5e79abfc0178f383](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win64.zip.sha256) |
+| [fluent-bit-2.1.9-win32.exe](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win32.exe) | [c3c98449a8676d5f3f53ef09647fb71cf7eb63ae8f73260b2ab7394c6a419586](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win32.exe.sha256) |
+| [fluent-bit-2.1.9-win32.zip](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win32.zip) | [1622ed416d1ffca3d3c1721a332fff9e80b8b700db47930d540ca640ecd18f8e](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win32.zip.sha256) |
+| [fluent-bit-2.1.9-win64.exe](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win64.exe) | [30556168954917cd40fa110da88f10bccc59aea7bc1018078522ac0eea90b1b8](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win64.exe.sha256) |
+| [fluent-bit-2.1.9-win64.zip](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win64.zip) | [291660a844d61d225055bfe1da8f10d95a27aa2f6d662db55744d47a716e85ff](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win64.zip.sha256) |
 
 To check the integrity, use `Get-FileHash` cmdlet on PowerShell.
 
 ```powershell
-PS> Get-FileHash fluent-bit-2.1.8-win32.exe
+PS> Get-FileHash fluent-bit-2.1.9-win32.exe
 ```
 
 ## Installing from ZIP archive
@@ -101,7 +101,7 @@ Download a ZIP archive from above. There are installers for 32-bit and 64-bit en
 Then you need to expand the ZIP archive. You can do this by clicking "Extract All" on Explorer, or if you're using PowerShell, you can use `Expand-Archive` cmdlet.
 
 ```powershell
-PS> Expand-Archive fluent-bit-2.1.8-win64.zip
+PS> Expand-Archive fluent-bit-2.1.9-win64.zip
 ```
 
 The ZIP package contains the following set of files.

diff --git a/pipeline/inputs/http.md b/pipeline/inputs/http.md
@@ -28,7 +28,74 @@ The http input plugin allows Fluent Bit to open up an HTTP port that you can the
 
 #### How to set tag
 
-The tag for the HTTP input plugin is set by adding the tag to the end of the request URL. This tag is then used to route the event through the system. For example, in the following curl message below the tag set is `app.log`**. **If you do not set the tag `http.0` is automatically used. If you have multiple HTTP inputs then they will follow a pattern of `http.N` where N is an integer representing the input.
+The tag for the HTTP input plugin is set by adding the tag to the end of the request URL. This tag is then used to route the event through the system.
+For example, in the following curl message below the tag set is `app.log**. **` because the end end path is `/app_log`:
+
+### Curl request
+
+```
+curl -d '{"key1":"value1","key2":"value2"}' -XPOST -H "content-type: application/json" http://localhost:8888/app.log
+```
+
+### Configuration File
+
+```
+[INPUT]
+    name http
+    listen 0.0.0.0
+    port 8888
+
+[OUTPUT]
+    name stdout
+    match app.log
+```
+
+If you do not set the tag `http.0` is automatically used. If you have multiple HTTP inputs then they will follow a pattern of `http.N` where N is an integer representing the input.
+
+### Curl request
+
+```
+curl -d '{"key1":"value1","key2":"value2"}' -XPOST -H "content-type: application/json" http://localhost:8888
+```
+
+### Configuration File
+
+```
+[INPUT]
+    name http
+    listen 0.0.0.0
+    port 8888
+
+[OUTPUT]
+    name  stdout
+    match  http.0
+```
+
+
+#### How to set tag_key
+
+The tag_key configuration option allows to specify the key name that will be used to overwrite a tag. The tag's value will be replaced with the value associated with the specified key. For example, setting tag_key to "custom_tag" and the log event contains a json field with the key "custom_tag" Fluent Bit will use the value of that field as the new tag for routing the event through the system.
+
+### Curl request
+
+```
+curl -d '{"key1":"value1","key2":"value2"}' -XPOST -H "content-type: application/json" http://localhost:8888/app.log
+```
+
+### Configuration File
+
+```
+[INPUT]
+    name http
+    listen 0.0.0.0
+    port 8888
+    tag_key key1
+
+[OUTPUT]
+    name stdout
+    match value1
+```
+
 
 #### How to set multiple custom HTTP header on success
 

diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md
@@ -126,7 +126,7 @@ $ fluent-bit -i tail -p path=/var/log/syslog -o stdout
 
 ### Configuration File
 
-In your main configuration file append the following _Input_ & _Output_ sections. An example visualization can be found [here](https://link.calyptia.com/vg2)
+In your main configuration file append the following _Input_ & _Output_ sections. 
 
 {% tabs %}
 {% tab title="fluent-bit.conf" %}

diff --git a/pipeline/inputs/udp.md b/pipeline/inputs/udp.md
@@ -0,0 +1,89 @@
+# UDP
+
+The **udp** input plugin allows to retrieve structured JSON or raw messages over a UDP network interface (UDP port).
+
+## Configuration Parameters
+
+The plugin supports the following configuration parameters:
+
+| Key          | Description                                                                                                                                                                                                                                                    | Default |
+| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
+| Listen       | Listener network interface.                                                                                                                                                                                                                                    | 0.0.0.0 |
+| Port         | UDP port where listening for connections                                                                                                                                                                                                                       | 5170    |
+| Buffer\_Size | Specify the maximum buffer size in KB to receive a JSON message. If not set, the default size will be the value of _Chunk\_Size_.                                                                                                                              |         |
+| Chunk\_Size  | By default the buffer to store the incoming JSON messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Chunk\_Size_ in KB. If not set, _Chunk\_Size_ is equal to 32 (32KB). | 32      |
+| Format       | Specify the expected payload format. It support the options _json_ and _none_. When using _json_, it expects JSON maps, when is set to _none_, it will split every record using the defined _Separator_ (option below).                                        | json    |
+| Separator    | When the expected _Format_ is set to _none_, Fluent Bit needs a separator string to split the records. By default it uses the breakline character  (LF or 0x10).                                                                                               |         |
+| Source\_Address\_Key| Specify the key where the source address will be injected.                                                                                                                                                                                              |         |
+
+## Getting Started
+
+In order to receive JSON messages over UDP, you can run the plugin from the command line or through the configuration file:
+
+### Command Line
+
+From the command line you can let Fluent Bit listen for _JSON_ messages with the following options:
+
+```bash
+$ fluent-bit -i udp -o stdout
+```
+
+By default the service will listen an all interfaces (0.0.0.0) through UDP port 5170, optionally you can change this directly, e.g:
+
+```bash
+$ fluent-bit -i udp -pport=9090 -o stdout
+```
+
+In the example the JSON messages will only arrive through network interface under 192.168.3.2 address and UDP Port 9090.
+
+### Configuration File
+
+In your main configuration file append the following _Input_ & _Output_ sections:
+
+```python
+[INPUT]
+    Name        udp
+    Listen      0.0.0.0
+    Port        5170
+    Chunk_Size  32
+    Buffer_Size 64
+    Format      json
+
+[OUTPUT]
+    Name        stdout
+    Match       *
+```
+
+## Testing
+
+Once Fluent Bit is running, you can send some messages using the _netcat_:
+
+```bash
+$ echo '{"key 1": 123456789, "key 2": "abcdefg"}' | nc -u 127.0.0.1 5170
+```
+
+In [Fluent Bit](http://fluentbit.io) we should see the following output:
+
+```bash
+$ bin/fluent-bit -i udp -o stdout -f 1
+Fluent Bit v2.x.x
+* Copyright (C) 2015-2022 The Fluent Bit Authors
+* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
+* https://fluentbit.io
+
+[2023/07/21 13:01:03] [ info] [fluent bit] version=2.1.7, commit=2474ccc759, pid=9677
+[2023/07/21 13:01:03] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
+[2023/07/21 13:01:03] [ info] [cmetrics] version=0.6.3
+[2023/07/21 13:01:03] [ info] [ctraces ] version=0.3.1
+[2023/07/21 13:01:03] [ info] [input:udp:udp.0] initializing
+[2023/07/21 13:01:03] [ info] [input:udp:udp.0] storage_strategy='memory' (memory only)
+[2023/07/21 13:01:03] [ info] [output:stdout:stdout.0] worker #0 started
+[2023/07/21 13:01:03] [ info] [sp] stream processor started
+[0] udp.0: [[1689912069.078189000, {}], {"key 1"=>123456789, "key 2"=>"abcdefg"}]
+```
+
+## Performance Considerations
+
+When receiving payloads in JSON format, there are high performance penalties. Parsing JSON is a very expensive task so you could expect your CPU usage increase under high load environments.
+
+To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed.
diff --git a/pipeline/inputs/windows-exporter-metrics.md b/pipeline/inputs/windows-exporter-metrics.md
@@ -24,6 +24,8 @@ The initial release of Windows Exporter Metrics contains a single collector avai
 | we.service.where           | Specify the WHERE clause for retrieving service metrics.  | `NULL`   |
 | we.service.include         | Specify the key value pairs for the include condition for the WHERE clause of service metrics. | `NULL`   |
 | we.service.exclude         | Specify the key value pairs for the exclude condition for the WHERE clause of service metrics. | `NULL`   |
+| we.process.allow\_process\_regex | Specify the regex covering the process metrics to  collect. Collect all by default. | "/.+/"    |
+| we.process.deny\_process\_regex  | Specify the regex for process metrics to prevent collection of/ignore. Allow all by default. | `NULL`    |
 | collector.cpu.scrape\_interval | The rate in seconds at which cpu metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds |
 | collector.net.scrape\_interval | The rate in seconds at which net metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds |
 | collector.logical_disk.scrape\_interval | The rate in seconds at which logical\_disk metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds |
@@ -34,6 +36,9 @@ The initial release of Windows Exporter Metrics contains a single collector avai
 | collector.logon.scrape\_interval    | The rate in seconds at which logon metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds |
 | collector.system.scrape\_interval   | The rate in seconds at which system metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds |
 | collector.service.scrape\_interval   | The rate in seconds at which service metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds |
+| collector.memory.scrape\_interval   | The rate in seconds at which memory metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds |
+| collector.paging_file.scrape\_interval   | The rate in seconds at which paging_file metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds |
+| collector.process.scrape\_interval   | The rate in seconds at which process metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds |
 | metrics | To specify which metrics are collected from the host operating system. | `"cpu,cpu_info,os,net,logical_disk,cs,thermalzone,logon,system,service"` |
 
 ## Collectors available
@@ -53,7 +58,10 @@ The following table describes the available collectors as part of this plugin. A
 | cpu\_info     | Exposes cpu\_info statistics.                                                                    | Windows | v2.0.8  |
 | logon         | Exposes logon statistics.                                                                        | Windows | v2.0.8  |
 | system        | Exposes system statistics.                                                                       | Windows | v2.0.8  |
-| service       | Exposes system statistics.                                                                       | Windows | v2.1.6  |
+| service       | Exposes service statistics.                                                                      | Windows | v2.1.6  |
+| memory        | Exposes memory statistics.                                                                       | Windows | v2.1.9  |
+| paging\_file   | Exposes paging\_file statistics.                                                                | Windows | v2.1.9  |
+| process       | Exposes process statistics.                                                                      | Windows | v2.1.9  |
 
 ## Getting Started
 

diff --git a/pipeline/outputs/elasticsearch.md b/pipeline/outputs/elasticsearch.md
@@ -113,7 +113,7 @@ In your main configuration file append the following _Input_ & _Output_ sections
     Type  my_type
 ```
 
-![example configuration visualization from config.calyptia.com](../../.gitbook/assets/image%20%282%29.png)
+![example configuration visualization from calyptia](../../.gitbook/assets/image%20%282%29.png)
 
 ## About Elasticsearch field names