From e4ed27fde8da676f5c47ae11aab3ec5171e4be8f Mon Sep 17 00:00:00 2001 From: Alexa Kreizinger Date: Mon, 19 Aug 2024 13:53:51 -0700 Subject: [PATCH 1/4] summary: Add multithreading doc to sidebar Signed-off-by: Alexa Kreizinger --- SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/SUMMARY.md b/SUMMARY.md index b23e4d763..34c681c26 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -67,6 +67,7 @@ * [Networking](administration/networking.md) * [Memory Management](administration/memory-management.md) * [Monitoring](administration/monitoring.md) +* [Multithreading](administration/multithreading.md) * [HTTP Proxy](administration/http-proxy.md) * [Hot Reload](administration/hot-reload.md) * [Troubleshooting](administration/troubleshooting.md) From 852cf7902bf77ee5f71747ab4900bcf4a5be6f1a Mon Sep 17 00:00:00 2001 From: Alexa Kreizinger Date: Mon, 19 Aug 2024 16:50:16 -0700 Subject: [PATCH 2/4] administration: multithreading: Write multithreading doc Signed-off-by: Alexa Kreizinger --- administration/multithreading.md | 45 ++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 administration/multithreading.md diff --git a/administration/multithreading.md b/administration/multithreading.md new file mode 100644 index 000000000..3ef0b139a --- /dev/null +++ b/administration/multithreading.md @@ -0,0 +1,45 @@ +--- +description: Learn how to run Fluent Bit in multiple threads for improved scalability. +--- + +# Multithreading + +Fluent Bit has one event loop to handle critical operations, like managing +timers, receiving internal messages, scheduling flushes, and handling retries. +This event loop runs in Fluent Bit's main thread. + +To free up resources in the main thread, you can configure +[inputs](../pipeline/inputs/README.md) and [outputs](../pipeline/outputs/README.md) +to run in their own self-contained threads. However, inputs and outputs implement +multithreading in distinct ways: inputs can run in **threaded** mode, and outputs +can use one or more **workers**. + +## Inputs + +When inputs collect telemetry data, they can either perform this process +inside Fluent Bit's main thread or inside a separate dedicated thread. You can +configure this behavior by enabling or disabling the `threaded` setting. + +All inputs are capable of running in threaded mode, but certain inputs always +run in threaded mode regardless of configuration. These always-threaded inputs are: + +- [Kubernetes Events](../pipeline/inputs/kubernetes-events.md) +- [Node Exporter Metrics](../pipeline/inputs/node-exporter-metrics.md) +- [Process Exporter Metrics](../pipeline/inputs/process-exporter-metrics.md) +- [Windows Exporter Metrics](../pipeline/inputs/windows-exporter-metrics.md) + +Inputs are not internally aware of multithreading. If an input runs in threaded +mode, Fluent Bit manages the logistics of that input's thread. + +## Outputs + +When outputs flush data, they can either perform this operation inside Fluent Bit's +main thread or inside a separate dedicated thread called a _worker_. Each output +can have one or more workers running in parallel, and each worker can handle multiple +concurrent flushes. You can configure this behavior by changing the value of the +`workers` setting. + +All outputs are capable of running in multiple workers, and each output has a +a default value of `0`, `1`, or `2` workers. However, even if an output uses +workers by default, you can safely reduce the number of workers below the default +or disable workers entirely. From ed0684a8374067b3c9edda6fb51e0faed1bd969d Mon Sep 17 00:00:00 2001 From: Alexa Kreizinger Date: Wed, 21 Aug 2024 09:26:35 -0700 Subject: [PATCH 3/4] Update administration/multithreading.md Signed-off-by: Alexa Kreizinger --- administration/multithreading.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/administration/multithreading.md b/administration/multithreading.md index 3ef0b139a..4899a4e57 100644 --- a/administration/multithreading.md +++ b/administration/multithreading.md @@ -14,6 +14,11 @@ to run in their own self-contained threads. However, inputs and outputs implemen multithreading in distinct ways: inputs can run in **threaded** mode, and outputs can use one or more **workers**. +Threading also affects certain processes related to inputs and outputs. For example, +[filters](../pipeline/filters/README.md) always run in the main thread, but +[processors](../pipeline/processors/README.md) run in the self-contained threads of +their respective inputs or outputs, if applicable. + ## Inputs When inputs collect telemetry data, they can either perform this process From cfdf3882e140096f2b19e1c8a179205db038da24 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Lecaros?= Date: Thu, 22 Aug 2024 11:21:30 -0400 Subject: [PATCH 4/4] administration: multithreading: fix typo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: José Lecaros --- administration/multithreading.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/administration/multithreading.md b/administration/multithreading.md index 4899a4e57..57d792d9c 100644 --- a/administration/multithreading.md +++ b/administration/multithreading.md @@ -44,7 +44,7 @@ can have one or more workers running in parallel, and each worker can handle mul concurrent flushes. You can configure this behavior by changing the value of the `workers` setting. -All outputs are capable of running in multiple workers, and each output has a +All outputs are capable of running in multiple workers, and each output has a default value of `0`, `1`, or `2` workers. However, even if an output uses workers by default, you can safely reduce the number of workers below the default or disable workers entirely.