Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed storage.total_limit_size misleading document content #1244

Merged
merged 2 commits into from
Dec 18, 2023
Merged

Fixed storage.total_limit_size misleading document content #1244

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4f34882
Fixed storage.total_limit_size misleading document content
Oct 23, 2023
f207293
Fixed storage.total_limit_size misleading document content
Oct 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions administration/buffering-and-storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ Fluent Bit implements the concept of logical queues: based on its Tag, a Chunk c

It's common to find cases where if we have multiple destinations for a Chunk, one of the destinations might be slower than the other, or maybe one is generating backpressure and not all of them. In this scenario, how do we limit the amount of filesystem Chunks that we are logically queueing?

Starting from Fluent Bit v1.6, we introduced the new configuration property for output plugins called `storage.total_limit_size` which limits the number of Chunks that exist in the filesystem for a certain logical output destination. If one of the destinations reaches the `storage.total_limit_size`, the oldest Chunk from its queue for that logical output destination will be discarded.
Starting from Fluent Bit v1.6, we introduced the new configuration property for output plugins called `storage.total_limit_size` which limits the total size in bytes of chunks that can exist in the filesystem for a certain logical output destination. If one of the destinations reaches the configured `storage.total_limit_size`, the oldest Chunk from its queue for that logical output destination will be discarded to make room for new data.

## Configuration

Expand Down Expand Up @@ -167,7 +167,7 @@ If certain chunks are filesystem _storage.type_ based, it's possible to control

| Key | Description | Default |
| :--- | :--- | :--- |
| storage.total\_limit\_size | Limit the maximum number of Chunks in the filesystem for the current output logical destination. | |
| storage.total\_limit\_size | Limit the maximum disk space size in bytes for buffering chunks in the filesystem for the current output logical destination. | |

The following example create records with CPU usage samples in the filesystem and then they are delivered to Google Stackdriver service limiting the logical queue \(buffering\) to 5M:

Expand All @@ -191,5 +191,5 @@ The following example create records with CPU usage samples in the filesystem an
storage.total_limit_size 5M
```

If for some reason Fluent Bit gets offline because of a network issue, it will continue buffering CPU samples but just keep a maximum of 5M of the newest data.
If for some reason Fluent Bit gets offline because of a network issue, it will continue buffering CPU samples but just keep a maximum of 5MB of the newest data.