Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multithreading doc #1439

Merged
merged 4 commits into from
Aug 22, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@
* [Networking](administration/networking.md)
* [Memory Management](administration/memory-management.md)
* [Monitoring](administration/monitoring.md)
* [Multithreading](administration/multithreading.md)
* [HTTP Proxy](administration/http-proxy.md)
* [Hot Reload](administration/hot-reload.md)
* [Troubleshooting](administration/troubleshooting.md)
Expand Down
45 changes: 45 additions & 0 deletions administration/multithreading.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
description: Learn how to run Fluent Bit in multiple threads for improved scalability.
---

# Multithreading
patrick-stephens marked this conversation as resolved.
Show resolved Hide resolved

Fluent Bit has one event loop to handle critical operations, like managing
timers, receiving internal messages, scheduling flushes, and handling retries.
This event loop runs in Fluent Bit's main thread.

To free up resources in the main thread, you can configure
[inputs](../pipeline/inputs/README.md) and [outputs](../pipeline/outputs/README.md)
to run in their own self-contained threads. However, inputs and outputs implement
multithreading in distinct ways: inputs can run in **threaded** mode, and outputs
can use one or more **workers**.

## Inputs
alexakreizinger marked this conversation as resolved.
Show resolved Hide resolved

When inputs collect telemetry data, they can either perform this process
inside Fluent Bit's main thread or inside a separate dedicated thread. You can
configure this behavior by enabling or disabling the `threaded` setting.

All inputs are capable of running in threaded mode, but certain inputs always
run in threaded mode regardless of configuration. These always-threaded inputs are:

- [Kubernetes Events](../pipeline/inputs/kubernetes-events.md)
- [Node Exporter Metrics](../pipeline/inputs/node-exporter-metrics.md)
- [Process Exporter Metrics](../pipeline/inputs/process-exporter-metrics.md)
- [Windows Exporter Metrics](../pipeline/inputs/windows-exporter-metrics.md)

Inputs are not internally aware of multithreading. If an input runs in threaded
mode, Fluent Bit manages the logistics of that input's thread.

## Outputs

When outputs flush data, they can either perform this operation inside Fluent Bit's
main thread or inside a separate dedicated thread called a _worker_. Each output
can have one or more workers running in parallel, and each worker can handle multiple
concurrent flushes. You can configure this behavior by changing the value of the
`workers` setting.

All outputs are capable of running in multiple workers, and each output has a
lecaros marked this conversation as resolved.
Show resolved Hide resolved
a default value of `0`, `1`, or `2` workers. However, even if an output uses
workers by default, you can safely reduce the number of workers below the default
or disable workers entirely.