From 3e814805c22c26a00824e52266cd58013650284b Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Tue, 22 Oct 2024 12:23:09 -0700 Subject: [PATCH 1/7] fluent: docs: updating key concepts for style Signed-off-by: Lynette Miles --- concepts/key-concepts.md | 135 +++++++++++++++++++++------------------ 1 file changed, 72 insertions(+), 63 deletions(-) diff --git a/concepts/key-concepts.md b/concepts/key-concepts.md index c3b70801a..6cf905676 100644 --- a/concepts/key-concepts.md +++ b/concepts/key-concepts.md @@ -1,25 +1,28 @@ --- -description: >- - There are a few key concepts that are really important to understand how - Fluent Bit operates. +description: Learn these key concepts to understand how Fluent Bit operates. --- -# Key Concepts +# Key concepts -Before diving into [Fluent Bit](https://fluentbit.io) it’s good to get acquainted with some of the key concepts of the service. This document provides a gentle introduction to those concepts and common [Fluent Bit](https://fluentbit.io) terminology. We’ve provided a list below of all the terms we’ll cover, but we recommend reading this document from start to finish to gain a more general understanding of our log and stream processor. +Before diving into [Fluent Bit](https://fluentbit.io) you might want to get acquainted +with some of the key concepts of the service. This document provides an +introduction to those concepts and common [Fluent Bit](https://fluentbit.io) +terminology. Reading this document will help you gain a more general understanding of the +log and stream processor. -* Event or Record -* Filtering -* Tag -* Timestamp -* Match -* Structured Message +- Event or Record +- Filtering +- Tag +- Timestamp +- Match +- Structured Message ## Event or Record -Every incoming piece of data that belongs to a log or a metric that is retrieved by Fluent Bit is considered an Event or a Record. +Every incoming piece of data that belongs to a log or a metric that's retrieved by +Fluent Bit is considered an _Event_ or a _Record_. -As an example consider the following content of a Syslog file: +As an example, consider the following content of a Syslog file: ```text Jan 18 12:52:16 flb systemd[2222]: Starting GNOME Terminal Server @@ -28,30 +31,31 @@ Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server. Jan 18 12:52:16 flb gsd-media-keys[2640]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0) ``` -It contains four lines and all of them represents **four** independent Events. +It contains four lines, representing four independent Events. -Internally an Event is comprised of: +Internally, an Event is comprised of: -* timestamp -* key/value metadata (since v2.1.0) -* payload +- timestamp +- key/value metadata (v2.1.0 and greater) +- payload ### Event format The Fluent Bit wire protocol represents an Event as a 2-element array with a nested array as the first element: -```javascript +```javascript copy [[TIMESTAMP, METADATA], MESSAGE] ``` where -* TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string); -* METADATA is a possibly-empty object containing event metadata; and -* MESSAGE is an object containing the event body. +- _`TIMESTAMP`_ is a timestamp in seconds as an integer or floating point value + (not a string) +- _`METADATA`_ is a possibly empty object containing event metadata +- _`MESSAGE`_ is an object containing the event body -Fluent Bit versions prior to v2.1.0 instead used: +Fluent Bit versions prior to v2.1.0 used: ```javascript [TIMESTAMP, MESSAGE] @@ -62,74 +66,79 @@ streams. ## Filtering -In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering. +You might need to perform modifications on the Event's content. The process to alter, +append to, or drop Events is called [_filtering_](pipeline/filters.md). -There are many use cases when Filtering is required like: +Use filtering to: -* Append specific information to the Event like an IP address or metadata. -* Select a specific piece of the Event content. -* Drop Events that matches certain pattern. +- Append specific information to the Event like an IP address or metadata. +- Select a specific piece of the Event content. +- Drop Events that match a certain pattern. ## Tag -Every Event that gets into Fluent Bit gets assigned a Tag. This tag is an internal string that is used in a later stage by the Router to decide which Filter or Output phase it must go through. +Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string +used in a later stage by the Router to decide which Filter or +[Output](pipeline/outputs.md) phase it must go through. -Most of the tags are assigned manually in the configuration. If a tag is not specified, Fluent Bit will assign the name of the Input plugin instance from where that Event was generated from. +Most tags are assigned manually in the configuration. If a tag isn't specified, +Fluent Bit assigns the name of the [Input](pipeline/inputs.md) plugin instance +where that Event was generated from. {% hint style="info" %} -The only input plugin that **does NOT** assign tags is [Forward](../pipeline/inputs/forward.md) input. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the client. +The [Forward](../pipeline/inputs/forward.md) input plugin doesn't assign tags. This +plugin speaks the Fluentd wire protocol called Forward where every Event already +comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the +client. {% endhint %} -A Tagged record must always have a Matching rule. To learn more about Tags and Matches check the [Routing](data-pipeline/router.md) section. +A tagged record must always have a Matching rule. To learn more about Tags and +Matches, see [Routing](data-pipeline/router.md). ## Timestamp -The Timestamp represents the _time_ when an Event was created. Every Event contains a Timestamp associated. The Timestamp is a numeric fractional integer in the format: +The timestamp represents the time an Event was created. Every Event contains a +timestamp associated. A timestamp always exists, and is set by the Input plugin or +discovered through a data parsing process. + +The timestamp is a numeric fractional integer in the format: ```javascript SECONDS.NANOSECONDS ``` -### Seconds - -It is the number of seconds that have elapsed since the _Unix epoch._ - -### Nanoseconds +where: -Fractional second or one thousand-millionth of a second. - -{% hint style="info" %} -A timestamp always exists, either set by the Input plugin or discovered through a data parsing process. -{% endhint %} +- `_Seconds_` is the number of seconds that have elapsed since the Unix epoch. +- `_Nanoseconds_` is a fractional second or one thousand-millionth of a second. ## Match -Fluent Bit allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule. +Fluent Bit lets you deliver your collected and processed Events to one or multiple +destinations through a routing phase. A _Match_ represents a rule to select Events +where a Tag matches a defined rule. -To learn more about Tags and Matches check the [Routing](data-pipeline/router.md) section. +To learn more about Tags and Matches, see [Routing](concepts/data-pipeline/router.md). -## Structured Messages +## Structured messages -Source events can have or not have a structure. A structure defines a set of _keys_ and _values_ inside the Event message. As an example consider the following two messages: +Source events can have a structure. A structure defines a set of `keys` and `values` +inside the Event message to implement faster operations on data modifications. +Fluent Bit handles every Event message as a structured message. -### No structured message +Consider the following two messages: -```javascript -"Project Fluent Bit created on 1398289291" -``` - -### Structured Message +- No structured message -```javascript -{"project": "Fluent Bit", "created": 1398289291} -``` + ```javascript + "Project Fluent Bit created on 1398289291" + ``` -At a low level both are just an array of bytes, but the Structured message defines _keys_ and _values_, having a structure helps to implement faster operations on data modifications. +- With a structured message -{% hint style="info" %} -Fluent Bit **always** handles every Event message as a structured message. -For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/). - -Consider [MessagePack](https://msgpack.org/) as a binary version of JSON on steroids. -{% endhint %} + ```javascript + {"project": "Fluent Bit", "created": 1398289291} + ``` +For performance reasons, Fluent Bit uses a binary serialization data format called +[MessagePack](https://msgpack.org/). From 4a1f678e75a1d876778f35fc2f9b605cce666b5b Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Tue, 22 Oct 2024 12:33:18 -0700 Subject: [PATCH 2/7] Fluent: docs: correcting links Signed-off-by: Lynette Miles --- concepts/key-concepts.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/concepts/key-concepts.md b/concepts/key-concepts.md index 6cf905676..04ca76110 100644 --- a/concepts/key-concepts.md +++ b/concepts/key-concepts.md @@ -51,9 +51,9 @@ with a nested array as the first element: where - _`TIMESTAMP`_ is a timestamp in seconds as an integer or floating point value - (not a string) -- _`METADATA`_ is a possibly empty object containing event metadata -- _`MESSAGE`_ is an object containing the event body + (not a string). +- _`METADATA`_ is a possibly empty object containing event metadata. +- _`MESSAGE`_ is an object containing the event body. Fluent Bit versions prior to v2.1.0 used: @@ -67,7 +67,7 @@ streams. ## Filtering You might need to perform modifications on the Event's content. The process to alter, -append to, or drop Events is called [_filtering_](pipeline/filters.md). +append to, or drop Events is called [_filtering_](concepts/data-pipeline/filters.md). Use filtering to: @@ -79,11 +79,11 @@ Use filtering to: Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string used in a later stage by the Router to decide which Filter or -[Output](pipeline/outputs.md) phase it must go through. +[Output](concepts/data-pipeline/output.md) phase it must go through. Most tags are assigned manually in the configuration. If a tag isn't specified, -Fluent Bit assigns the name of the [Input](pipeline/inputs.md) plugin instance -where that Event was generated from. +Fluent Bit assigns the name of the [Input](concepts/data-pipeline/input.md) plugin +instance where that Event was generated from. {% hint style="info" %} The [Forward](../pipeline/inputs/forward.md) input plugin doesn't assign tags. This @@ -118,7 +118,7 @@ Fluent Bit lets you deliver your collected and processed Events to one or multip destinations through a routing phase. A _Match_ represents a rule to select Events where a Tag matches a defined rule. -To learn more about Tags and Matches, see [Routing](concepts/data-pipeline/router.md). +To learn more about Tags and Matches, see [Routing](data-pipeline/router.md). ## Structured messages From e33f6c60f7ccbc7ab549d84f9dc856b765784b49 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Tue, 22 Oct 2024 12:41:04 -0700 Subject: [PATCH 3/7] Fluent: docs: still fixing links Signed-off-by: Lynette Miles --- concepts/key-concepts.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/concepts/key-concepts.md b/concepts/key-concepts.md index 04ca76110..723e04a69 100644 --- a/concepts/key-concepts.md +++ b/concepts/key-concepts.md @@ -79,10 +79,10 @@ Use filtering to: Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string used in a later stage by the Router to decide which Filter or -[Output](concepts/data-pipeline/output.md) phase it must go through. +[Output](data-pipeline/output.md) phase it must go through. Most tags are assigned manually in the configuration. If a tag isn't specified, -Fluent Bit assigns the name of the [Input](concepts/data-pipeline/input.md) plugin +Fluent Bit assigns the name of the [Input](data-pipeline/input.md) plugin instance where that Event was generated from. {% hint style="info" %} From 30b6a95914aa70787195ca3730bf7a193c5c8505 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Tue, 22 Oct 2024 12:42:50 -0700 Subject: [PATCH 4/7] fluent: docs: missed one more link Signed-off-by: Lynette Miles --- concepts/key-concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/concepts/key-concepts.md b/concepts/key-concepts.md index 723e04a69..5aa16a1a9 100644 --- a/concepts/key-concepts.md +++ b/concepts/key-concepts.md @@ -67,7 +67,7 @@ streams. ## Filtering You might need to perform modifications on the Event's content. The process to alter, -append to, or drop Events is called [_filtering_](concepts/data-pipeline/filters.md). +append to, or drop Events is called [_filtering_](data-pipeline/filters.md). Use filtering to: From 30d9050c73383b6acf82e4ea9c5d4e00b6f5c9f8 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Tue, 22 Oct 2024 12:44:42 -0700 Subject: [PATCH 5/7] fluent: docs: missed one more link Signed-off-by: Lynette Miles --- concepts/key-concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/concepts/key-concepts.md b/concepts/key-concepts.md index 5aa16a1a9..1dfac280f 100644 --- a/concepts/key-concepts.md +++ b/concepts/key-concepts.md @@ -67,7 +67,7 @@ streams. ## Filtering You might need to perform modifications on the Event's content. The process to alter, -append to, or drop Events is called [_filtering_](data-pipeline/filters.md). +append to, or drop Events is called [_filtering_](data-pipeline/filter.md). Use filtering to: From 63f3489cb9e679e48cb3a0320b8854e0cf9527c0 Mon Sep 17 00:00:00 2001 From: esmerel <6818907+esmerel@users.noreply.github.com> Date: Tue, 22 Oct 2024 15:04:43 -0700 Subject: [PATCH 6/7] Apply suggestions from code review Co-authored-by: Alexa Kreizinger Signed-off-by: esmerel <6818907+esmerel@users.noreply.github.com> --- concepts/key-concepts.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/concepts/key-concepts.md b/concepts/key-concepts.md index 1dfac280f..712c871f7 100644 --- a/concepts/key-concepts.md +++ b/concepts/key-concepts.md @@ -8,7 +8,7 @@ Before diving into [Fluent Bit](https://fluentbit.io) you might want to get acqu with some of the key concepts of the service. This document provides an introduction to those concepts and common [Fluent Bit](https://fluentbit.io) terminology. Reading this document will help you gain a more general understanding of the -log and stream processor. +following topics: - Event or Record - Filtering @@ -31,9 +31,9 @@ Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server. Jan 18 12:52:16 flb gsd-media-keys[2640]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0) ``` -It contains four lines, representing four independent Events. +It contains four lines that represent four independent Events. -Internally, an Event is comprised of: +An Event is comprised of: - timestamp - key/value metadata (v2.1.0 and greater) @@ -41,7 +41,7 @@ Internally, an Event is comprised of: ### Event format -The Fluent Bit wire protocol represents an Event as a 2-element array +The Fluent Bit wire protocol represents an Event as a two-element array with a nested array as the first element: ```javascript copy @@ -52,7 +52,7 @@ where - _`TIMESTAMP`_ is a timestamp in seconds as an integer or floating point value (not a string). -- _`METADATA`_ is a possibly empty object containing event metadata. +- _`METADATA`_ is an object containing event metadata, and might be empty. - _`MESSAGE`_ is an object containing the event body. Fluent Bit versions prior to v2.1.0 used: @@ -66,7 +66,7 @@ streams. ## Filtering -You might need to perform modifications on the Event's content. The process to alter, +You might need to perform modifications on an Event's content. The process to alter, append to, or drop Events is called [_filtering_](data-pipeline/filter.md). Use filtering to: @@ -97,8 +97,8 @@ Matches, see [Routing](data-pipeline/router.md). ## Timestamp -The timestamp represents the time an Event was created. Every Event contains a -timestamp associated. A timestamp always exists, and is set by the Input plugin or +The timestamp represents the time an Event was created. Every Event contains an +associated timestamps. All events have timestamps, and they're set by the input plugin or discovered through a data parsing process. The timestamp is a numeric fractional integer in the format: @@ -114,8 +114,8 @@ where: ## Match -Fluent Bit lets you deliver your collected and processed Events to one or multiple -destinations through a routing phase. A _Match_ represents a rule to select Events +Fluent Bit lets you route your collected and processed Events to one or multiple +destinations. A _Match_ represents a rule to select Events where a Tag matches a defined rule. To learn more about Tags and Matches, see [Routing](data-pipeline/router.md). @@ -124,7 +124,7 @@ To learn more about Tags and Matches, see [Routing](data-pipeline/router.md). Source events can have a structure. A structure defines a set of `keys` and `values` inside the Event message to implement faster operations on data modifications. -Fluent Bit handles every Event message as a structured message. +Fluent Bit treats every Event message as a structured message. Consider the following two messages: From 5b7e1f193cd049cfcdf26c45032e663d01fd614f Mon Sep 17 00:00:00 2001 From: esmerel <6818907+esmerel@users.noreply.github.com> Date: Tue, 22 Oct 2024 15:05:22 -0700 Subject: [PATCH 7/7] Update key-concepts.md Signed-off-by: esmerel <6818907+esmerel@users.noreply.github.com> --- concepts/key-concepts.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/concepts/key-concepts.md b/concepts/key-concepts.md index 712c871f7..fc9841f30 100644 --- a/concepts/key-concepts.md +++ b/concepts/key-concepts.md @@ -109,8 +109,8 @@ SECONDS.NANOSECONDS where: -- `_Seconds_` is the number of seconds that have elapsed since the Unix epoch. -- `_Nanoseconds_` is a fractional second or one thousand-millionth of a second. +- `_SECONDS_` is the number of seconds that have elapsed since the Unix epoch. +- `_NANOSECONDS_` is a fractional second or one thousand-millionth of a second. ## Match