diff --git a/.gitbook.yaml b/.gitbook.yaml index 07c227704..703fd8f6e 100644 --- a/.gitbook.yaml +++ b/.gitbook.yaml @@ -30,6 +30,7 @@ redirects: input/tail: ./pipeline/inputs/tail.md input/tcp: ./pipeline/inputs/tcp.md input/thermal: ./pipeline/inputs/thermal.md + input/udp: ./pipeline/inputs/udp.md input/winlog: ./pipeline/inputs/windows-event-log.md input/winevtlog: ./pipeline/inputs/windows-event-log-winevtlog.md diff --git a/SUMMARY.md b/SUMMARY.md index 7a0aa9f5d..da19a1bc5 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -114,6 +114,7 @@ * [Tail](pipeline/inputs/tail.md) * [TCP](pipeline/inputs/tcp.md) * [Thermal](pipeline/inputs/thermal.md) + * [UDP](pipeline/inputs/udp.md) * [OpenTelemetry](pipeline/inputs/opentelemetry.md) * [Windows Event Log](pipeline/inputs/windows-event-log.md) * [Windows Event Log (winevtlog)](pipeline/inputs/windows-event-log-winevtlog.md) diff --git a/administration/configuring-fluent-bit/classic-mode/configuration-file.md b/administration/configuring-fluent-bit/classic-mode/configuration-file.md index e8a11aadf..b247fe992 100644 --- a/administration/configuring-fluent-bit/classic-mode/configuration-file.md +++ b/administration/configuring-fluent-bit/classic-mode/configuration-file.md @@ -138,7 +138,7 @@ The following configuration file example demonstrates how to collect CPU metrics ## Visualize -You can also visualize Fluent Bit INPUT, FILTER, and OUTPUT configuration via [https://cloud.calyptia.com](https://cloud.calyptia.com/visualizer) +You can also visualize Fluent Bit INPUT, FILTER, and OUTPUT configuration via [Calyptia](https://calyptia.com/free-trial) ![](../../../.gitbook/assets/image.png) diff --git a/administration/monitoring.md b/administration/monitoring.md index c95501288..a07824da1 100644 --- a/administration/monitoring.md +++ b/administration/monitoring.md @@ -322,9 +322,9 @@ If (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 seconds is TRUE, t If (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 seconds is FALSE, then it's healthy. -## Calyptia Cloud +## Calyptia -[Calyptia Cloud](https://cloud.calyptia.com) is a hosted service that allows you to monitor your Fluent Bit agents including data flow, metrics and configurations. +[Calyptia](https://calyptia.com/free-trial) is a hosted service that allows you to monitor your Fluent Bit agents including data flow, metrics and configurations. ![](../.gitbook/assets/image-19-.png) @@ -332,8 +332,8 @@ If (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 seconds is FALSE, Register your Fluent Bit agent will take **less than one minute**, steps: -* Go to [cloud.calyptia.com](https://cloud.calyptia.com) and sign-in -* On the left menu click on [Settings](https://cloud.calyptia.com/settings) and generate/copy your API key +* Go to the calyptia core console and sign-in +* On the left menu click on settings and generate/copy your API key In your Fluent Bit configuration file, append the following configuration section: diff --git a/installation/docker.md b/installation/docker.md index ef91c0af3..0e85e8613 100644 --- a/installation/docker.md +++ b/installation/docker.md @@ -17,6 +17,8 @@ The following table describes the Linux container tags that are available on Doc | Tag(s) | Manifest Architectures | Description | | ------------ | ------------------------- | -------------------------------------------------------------- | +| 2.1.9-debug | x86_64, arm64v8, arm32v7 | Debug images | +| 2.1.9 | x86_64, arm64v8, arm32v7 | Release [v2.1.9](https://fluentbit.io/announcements/v2.1.9/) | | 2.1.8-debug | x86_64, arm64v8, arm32v7 | Debug images | | 2.1.8 | x86_64, arm64v8, arm32v7 | Release [v2.1.8](https://fluentbit.io/announcements/v2.1.8/) | | 2.1.7-debug | x86_64, arm64v8, arm32v7 | Debug images | diff --git a/installation/kubernetes.md b/installation/kubernetes.md index f2b547893..da44eae16 100644 --- a/installation/kubernetes.md +++ b/installation/kubernetes.md @@ -151,7 +151,7 @@ spec: ### Configure Fluent Bit -Assuming the basic volume configuration described above, you can apply the following config to start logging. You can visualize this configuration [here](https://link.calyptia.com/gzc) +Assuming the basic volume configuration described above, you can apply the following config to start logging. You can visualize this configuration [here (Sign-up required)](https://calyptia.com/free-trial) ```yaml fluent-bit.conf: | diff --git a/installation/sources/build-and-install.md b/installation/sources/build-and-install.md index 376c8c522..17ffa83f3 100644 --- a/installation/sources/build-and-install.md +++ b/installation/sources/build-and-install.md @@ -149,6 +149,7 @@ The _input plugins_ provides certain features to gather information from a speci | [FLB\_IN\_TAIL](../../pipeline/inputs/tail.md) | Enable Tail \(follow files\) input plugin | On | | [FLB\_IN\_TCP](../../pipeline/inputs/tcp.md) | Enable TCP input plugin | On | | [FLB\_IN\_THERMAL](../../pipeline/inputs/thermal.md) | Enable system temperature\(s\) input plugin | On | +| [FLB\_IN\_UDP](../../pipeline/inputs/udp.md) | Enable UDP input plugin | On | | [FLB\_IN\_WINLOG](../../pipeline/inputs/windows-event-log.md) | Enable Windows Event Log input plugin \(Windows Only\) | On | | [FLB\_IN\_WINEVTLOG](../../pipeline/inputs/windows-event-log-winevtlog.md) | Enable Windows Event Log input plugin using winevt.h API \(Windows Only\) | On | diff --git a/installation/windows.md b/installation/windows.md index b269a2ceb..98ebeee23 100644 --- a/installation/windows.md +++ b/installation/windows.md @@ -83,15 +83,15 @@ The latest stable version is 2.1.4. Each version is available on the Github rele | INSTALLERS | SHA256 CHECKSUMS | | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------- | -| [fluent-bit-2.1.8-win32.exe](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win32.exe) | [5814f2da845dd27aedf2c805c90b85d8250223e455fe7139be8d307275a99548](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win32.exe.sha256) | -| [fluent-bit-2.1.8-win32.zip](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win32.zip) | [b45062666de4629c68e6a91176cbe003ad0db8607921ce5e1f89e1f3c6ee9f0d](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win32.zip.sha256) | -| [fluent-bit-2.1.8-win64.exe](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win64.exe) | [ac982def9a8cff6d302d61090ffde3f965ddf4a796900a66cdaa6e3ae687a621](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win64.exe.sha256) | -| [fluent-bit-2.1.8-win64.zip](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win64.zip) | [d18e2a0e1f5b649bf0907cfc51ea0e643938a4a4ed70624b5e79abfc0178f383](https://releases.fluentbit.io/2.1/fluent-bit-2.1.8-win64.zip.sha256) | +| [fluent-bit-2.1.9-win32.exe](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win32.exe) | [c3c98449a8676d5f3f53ef09647fb71cf7eb63ae8f73260b2ab7394c6a419586](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win32.exe.sha256) | +| [fluent-bit-2.1.9-win32.zip](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win32.zip) | [1622ed416d1ffca3d3c1721a332fff9e80b8b700db47930d540ca640ecd18f8e](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win32.zip.sha256) | +| [fluent-bit-2.1.9-win64.exe](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win64.exe) | [30556168954917cd40fa110da88f10bccc59aea7bc1018078522ac0eea90b1b8](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win64.exe.sha256) | +| [fluent-bit-2.1.9-win64.zip](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win64.zip) | [291660a844d61d225055bfe1da8f10d95a27aa2f6d662db55744d47a716e85ff](https://releases.fluentbit.io/2.1/fluent-bit-2.1.9-win64.zip.sha256) | To check the integrity, use `Get-FileHash` cmdlet on PowerShell. ```powershell -PS> Get-FileHash fluent-bit-2.1.8-win32.exe +PS> Get-FileHash fluent-bit-2.1.9-win32.exe ``` ## Installing from ZIP archive @@ -101,7 +101,7 @@ Download a ZIP archive from above. There are installers for 32-bit and 64-bit en Then you need to expand the ZIP archive. You can do this by clicking "Extract All" on Explorer, or if you're using PowerShell, you can use `Expand-Archive` cmdlet. ```powershell -PS> Expand-Archive fluent-bit-2.1.8-win64.zip +PS> Expand-Archive fluent-bit-2.1.9-win64.zip ``` The ZIP package contains the following set of files. diff --git a/pipeline/inputs/http.md b/pipeline/inputs/http.md index 10f842c01..aead92d9d 100644 --- a/pipeline/inputs/http.md +++ b/pipeline/inputs/http.md @@ -28,7 +28,74 @@ The http input plugin allows Fluent Bit to open up an HTTP port that you can the #### How to set tag -The tag for the HTTP input plugin is set by adding the tag to the end of the request URL. This tag is then used to route the event through the system. For example, in the following curl message below the tag set is `app.log`**. **If you do not set the tag `http.0` is automatically used. If you have multiple HTTP inputs then they will follow a pattern of `http.N` where N is an integer representing the input. +The tag for the HTTP input plugin is set by adding the tag to the end of the request URL. This tag is then used to route the event through the system. +For example, in the following curl message below the tag set is `app.log**. **` because the end end path is `/app_log`: + +### Curl request + +``` +curl -d '{"key1":"value1","key2":"value2"}' -XPOST -H "content-type: application/json" http://localhost:8888/app.log +``` + +### Configuration File + +``` +[INPUT] + name http + listen 0.0.0.0 + port 8888 + +[OUTPUT] + name stdout + match app.log +``` + +If you do not set the tag `http.0` is automatically used. If you have multiple HTTP inputs then they will follow a pattern of `http.N` where N is an integer representing the input. + +### Curl request + +``` +curl -d '{"key1":"value1","key2":"value2"}' -XPOST -H "content-type: application/json" http://localhost:8888 +``` + +### Configuration File + +``` +[INPUT] + name http + listen 0.0.0.0 + port 8888 + +[OUTPUT] + name stdout + match http.0 +``` + + +#### How to set tag_key + +The tag_key configuration option allows to specify the key name that will be used to overwrite a tag. The tag's value will be replaced with the value associated with the specified key. For example, setting tag_key to "custom_tag" and the log event contains a json field with the key "custom_tag" Fluent Bit will use the value of that field as the new tag for routing the event through the system. + +### Curl request + +``` +curl -d '{"key1":"value1","key2":"value2"}' -XPOST -H "content-type: application/json" http://localhost:8888/app.log +``` + +### Configuration File + +``` +[INPUT] + name http + listen 0.0.0.0 + port 8888 + tag_key key1 + +[OUTPUT] + name stdout + match value1 +``` + #### How to set multiple custom HTTP header on success diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 7f6d91943..eb84b8267 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -126,7 +126,7 @@ $ fluent-bit -i tail -p path=/var/log/syslog -o stdout ### Configuration File -In your main configuration file append the following _Input_ & _Output_ sections. An example visualization can be found [here](https://link.calyptia.com/vg2) +In your main configuration file append the following _Input_ & _Output_ sections. {% tabs %} {% tab title="fluent-bit.conf" %} diff --git a/pipeline/inputs/udp.md b/pipeline/inputs/udp.md new file mode 100644 index 000000000..5a4e12d5f --- /dev/null +++ b/pipeline/inputs/udp.md @@ -0,0 +1,89 @@ +# UDP + +The **udp** input plugin allows to retrieve structured JSON or raw messages over a UDP network interface (UDP port). + +## Configuration Parameters + +The plugin supports the following configuration parameters: + +| Key | Description | Default | +| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | +| Listen | Listener network interface. | 0.0.0.0 | +| Port | UDP port where listening for connections | 5170 | +| Buffer\_Size | Specify the maximum buffer size in KB to receive a JSON message. If not set, the default size will be the value of _Chunk\_Size_. | | +| Chunk\_Size | By default the buffer to store the incoming JSON messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Chunk\_Size_ in KB. If not set, _Chunk\_Size_ is equal to 32 (32KB). | 32 | +| Format | Specify the expected payload format. It support the options _json_ and _none_. When using _json_, it expects JSON maps, when is set to _none_, it will split every record using the defined _Separator_ (option below). | json | +| Separator | When the expected _Format_ is set to _none_, Fluent Bit needs a separator string to split the records. By default it uses the breakline character (LF or 0x10). | | +| Source\_Address\_Key| Specify the key where the source address will be injected. | | + +## Getting Started + +In order to receive JSON messages over UDP, you can run the plugin from the command line or through the configuration file: + +### Command Line + +From the command line you can let Fluent Bit listen for _JSON_ messages with the following options: + +```bash +$ fluent-bit -i udp -o stdout +``` + +By default the service will listen an all interfaces (0.0.0.0) through UDP port 5170, optionally you can change this directly, e.g: + +```bash +$ fluent-bit -i udp -pport=9090 -o stdout +``` + +In the example the JSON messages will only arrive through network interface under 192.168.3.2 address and UDP Port 9090. + +### Configuration File + +In your main configuration file append the following _Input_ & _Output_ sections: + +```python +[INPUT] + Name udp + Listen 0.0.0.0 + Port 5170 + Chunk_Size 32 + Buffer_Size 64 + Format json + +[OUTPUT] + Name stdout + Match * +``` + +## Testing + +Once Fluent Bit is running, you can send some messages using the _netcat_: + +```bash +$ echo '{"key 1": 123456789, "key 2": "abcdefg"}' | nc -u 127.0.0.1 5170 +``` + +In [Fluent Bit](http://fluentbit.io) we should see the following output: + +```bash +$ bin/fluent-bit -i udp -o stdout -f 1 +Fluent Bit v2.x.x +* Copyright (C) 2015-2022 The Fluent Bit Authors +* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd +* https://fluentbit.io + +[2023/07/21 13:01:03] [ info] [fluent bit] version=2.1.7, commit=2474ccc759, pid=9677 +[2023/07/21 13:01:03] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2023/07/21 13:01:03] [ info] [cmetrics] version=0.6.3 +[2023/07/21 13:01:03] [ info] [ctraces ] version=0.3.1 +[2023/07/21 13:01:03] [ info] [input:udp:udp.0] initializing +[2023/07/21 13:01:03] [ info] [input:udp:udp.0] storage_strategy='memory' (memory only) +[2023/07/21 13:01:03] [ info] [output:stdout:stdout.0] worker #0 started +[2023/07/21 13:01:03] [ info] [sp] stream processor started +[0] udp.0: [[1689912069.078189000, {}], {"key 1"=>123456789, "key 2"=>"abcdefg"}] +``` + +## Performance Considerations + +When receiving payloads in JSON format, there are high performance penalties. Parsing JSON is a very expensive task so you could expect your CPU usage increase under high load environments. + +To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed. diff --git a/pipeline/inputs/windows-exporter-metrics.md b/pipeline/inputs/windows-exporter-metrics.md index 7a9601ab2..61713457e 100644 --- a/pipeline/inputs/windows-exporter-metrics.md +++ b/pipeline/inputs/windows-exporter-metrics.md @@ -24,6 +24,8 @@ The initial release of Windows Exporter Metrics contains a single collector avai | we.service.where | Specify the WHERE clause for retrieving service metrics. | `NULL` | | we.service.include | Specify the key value pairs for the include condition for the WHERE clause of service metrics. | `NULL` | | we.service.exclude | Specify the key value pairs for the exclude condition for the WHERE clause of service metrics. | `NULL` | +| we.process.allow\_process\_regex | Specify the regex covering the process metrics to collect. Collect all by default. | "/.+/" | +| we.process.deny\_process\_regex | Specify the regex for process metrics to prevent collection of/ignore. Allow all by default. | `NULL` | | collector.cpu.scrape\_interval | The rate in seconds at which cpu metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds | | collector.net.scrape\_interval | The rate in seconds at which net metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds | | collector.logical_disk.scrape\_interval | The rate in seconds at which logical\_disk metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds | @@ -34,6 +36,9 @@ The initial release of Windows Exporter Metrics contains a single collector avai | collector.logon.scrape\_interval | The rate in seconds at which logon metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds | | collector.system.scrape\_interval | The rate in seconds at which system metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds | | collector.service.scrape\_interval | The rate in seconds at which service metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds | +| collector.memory.scrape\_interval | The rate in seconds at which memory metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds | +| collector.paging_file.scrape\_interval | The rate in seconds at which paging_file metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds | +| collector.process.scrape\_interval | The rate in seconds at which process metrics are collected from the host operating system. If a value greater than 0 is used then it overrides the global default otherwise the global default is used. | 0 seconds | | metrics | To specify which metrics are collected from the host operating system. | `"cpu,cpu_info,os,net,logical_disk,cs,thermalzone,logon,system,service"` | ## Collectors available @@ -53,7 +58,10 @@ The following table describes the available collectors as part of this plugin. A | cpu\_info | Exposes cpu\_info statistics. | Windows | v2.0.8 | | logon | Exposes logon statistics. | Windows | v2.0.8 | | system | Exposes system statistics. | Windows | v2.0.8 | -| service | Exposes system statistics. | Windows | v2.1.6 | +| service | Exposes service statistics. | Windows | v2.1.6 | +| memory | Exposes memory statistics. | Windows | v2.1.9 | +| paging\_file | Exposes paging\_file statistics. | Windows | v2.1.9 | +| process | Exposes process statistics. | Windows | v2.1.9 | ## Getting Started diff --git a/pipeline/outputs/elasticsearch.md b/pipeline/outputs/elasticsearch.md index 534fa203f..53b94c8a4 100644 --- a/pipeline/outputs/elasticsearch.md +++ b/pipeline/outputs/elasticsearch.md @@ -113,7 +113,7 @@ In your main configuration file append the following _Input_ & _Output_ sections Type my_type ``` -![example configuration visualization from config.calyptia.com](../../.gitbook/assets/image%20%282%29.png) +![example configuration visualization from calyptia](../../.gitbook/assets/image%20%282%29.png) ## About Elasticsearch field names diff --git a/pipeline/outputs/opensearch.md b/pipeline/outputs/opensearch.md index ecfdc739a..b50dd2d60 100644 --- a/pipeline/outputs/opensearch.md +++ b/pipeline/outputs/opensearch.md @@ -110,7 +110,7 @@ In your main configuration file append the following _Input_ & _Output_ sections Type my_type ``` -![example configuration visualization from config.calyptia.com](../../.gitbook/assets/image%20%282%29.png) +![example configuration visualization from calyptia](../../.gitbook/assets/image%20%282%29.png) ## About OpenSearch field names diff --git a/pipeline/outputs/vivo-exporter.md b/pipeline/outputs/vivo-exporter.md index 1f8cd5698..69c00dfcb 100644 --- a/pipeline/outputs/vivo-exporter.md +++ b/pipeline/outputs/vivo-exporter.md @@ -22,6 +22,7 @@ Here is a simple configuration of Vivo Exporter, note that this example is not b [OUTPUT] name vivo_exporter + match * empty_stream_on_read off stream_queue_size 20M 
http_cors_allow_origin * @@ -55,6 +56,7 @@ The example below will generate dummy log events which will be consuming by usin [OUTPUT] name vivo_exporter + match * ```