Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fluent: docs: updating key concepts for style #1482

Merged
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 72 additions & 63 deletions concepts/key-concepts.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,28 @@
---
description: >-
There are a few key concepts that are really important to understand how
Fluent Bit operates.
description: Learn these key concepts to understand how Fluent Bit operates.
---

# Key Concepts
# Key concepts

Before diving into [Fluent Bit](https://fluentbit.io) it’s good to get acquainted with some of the key concepts of the service. This document provides a gentle introduction to those concepts and common [Fluent Bit](https://fluentbit.io) terminology. We’ve provided a list below of all the terms we’ll cover, but we recommend reading this document from start to finish to gain a more general understanding of our log and stream processor.
Before diving into [Fluent Bit](https://fluentbit.io) you might want to get acquainted
with some of the key concepts of the service. This document provides an
introduction to those concepts and common [Fluent Bit](https://fluentbit.io)
terminology. Reading this document will help you gain a more general understanding of the
log and stream processor.
esmerel marked this conversation as resolved.
Show resolved Hide resolved

* Event or Record
* Filtering
* Tag
* Timestamp
* Match
* Structured Message
- Event or Record
- Filtering
- Tag
- Timestamp
- Match
- Structured Message

## Event or Record

Every incoming piece of data that belongs to a log or a metric that is retrieved by Fluent Bit is considered an Event or a Record.
Every incoming piece of data that belongs to a log or a metric that's retrieved by
Fluent Bit is considered an _Event_ or a _Record_.
esmerel marked this conversation as resolved.
Show resolved Hide resolved

As an example consider the following content of a Syslog file:
As an example, consider the following content of a Syslog file:

```text
Jan 18 12:52:16 flb systemd[2222]: Starting GNOME Terminal Server
Expand All @@ -28,30 +31,31 @@ Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server.
Jan 18 12:52:16 flb gsd-media-keys[2640]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0)
```

It contains four lines and all of them represents **four** independent Events.
It contains four lines, representing four independent Events.
esmerel marked this conversation as resolved.
Show resolved Hide resolved

Internally an Event is comprised of:
Internally, an Event is comprised of:
esmerel marked this conversation as resolved.
Show resolved Hide resolved

* timestamp
* key/value metadata (since v2.1.0)
* payload
- timestamp
- key/value metadata (v2.1.0 and greater)
- payload

### Event format

The Fluent Bit wire protocol represents an Event as a 2-element array
with a nested array as the first element:
esmerel marked this conversation as resolved.
Show resolved Hide resolved

```javascript
```javascript copy
[[TIMESTAMP, METADATA], MESSAGE]
```

where

* TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string);
* METADATA is a possibly-empty object containing event metadata; and
* MESSAGE is an object containing the event body.
- _`TIMESTAMP`_ is a timestamp in seconds as an integer or floating point value
(not a string).
- _`METADATA`_ is a possibly empty object containing event metadata.
esmerel marked this conversation as resolved.
Show resolved Hide resolved
- _`MESSAGE`_ is an object containing the event body.

Fluent Bit versions prior to v2.1.0 instead used:
Fluent Bit versions prior to v2.1.0 used:

```javascript
[TIMESTAMP, MESSAGE]
Expand All @@ -62,74 +66,79 @@ streams.

## Filtering

In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering.
You might need to perform modifications on the Event's content. The process to alter,
esmerel marked this conversation as resolved.
Show resolved Hide resolved
append to, or drop Events is called [_filtering_](data-pipeline/filter.md).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

append [stuff] to?


There are many use cases when Filtering is required like:
Use filtering to:

* Append specific information to the Event like an IP address or metadata.
* Select a specific piece of the Event content.
* Drop Events that matches certain pattern.
- Append specific information to the Event like an IP address or metadata.
- Select a specific piece of the Event content.
- Drop Events that match a certain pattern.

## Tag

Every Event that gets into Fluent Bit gets assigned a Tag. This tag is an internal string that is used in a later stage by the Router to decide which Filter or Output phase it must go through.
Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string
esmerel marked this conversation as resolved.
Show resolved Hide resolved
used in a later stage by the Router to decide which Filter or
[Output](data-pipeline/output.md) phase it must go through.

Most of the tags are assigned manually in the configuration. If a tag is not specified, Fluent Bit will assign the name of the Input plugin instance from where that Event was generated from.
Most tags are assigned manually in the configuration. If a tag isn't specified,
Fluent Bit assigns the name of the [Input](data-pipeline/input.md) plugin
instance where that Event was generated from.

{% hint style="info" %}
The only input plugin that **does NOT** assign tags is [Forward](../pipeline/inputs/forward.md) input. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the client.
The [Forward](../pipeline/inputs/forward.md) input plugin doesn't assign tags. This
plugin speaks the Fluentd wire protocol called Forward where every Event already
comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the
client.
{% endhint %}

A Tagged record must always have a Matching rule. To learn more about Tags and Matches check the [Routing](data-pipeline/router.md) section.
A tagged record must always have a Matching rule. To learn more about Tags and
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A tagged record must always have a Matching rule. To learn more about Tags and
A tagged record must always have a `Matching` rule. To learn more about Tags and

not sure if a code block is the best approach here but I feel like something is needed because otherwise the capitalization by itself sticks out

Matches, see [Routing](data-pipeline/router.md).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Matches, see [Routing](data-pipeline/router.md).
matches, see [Routing](data-pipeline/router.md).


## Timestamp

The Timestamp represents the _time_ when an Event was created. Every Event contains a Timestamp associated. The Timestamp is a numeric fractional integer in the format:
The timestamp represents the time an Event was created. Every Event contains a
timestamp associated. A timestamp always exists, and is set by the Input plugin or
discovered through a data parsing process.
esmerel marked this conversation as resolved.
Show resolved Hide resolved

The timestamp is a numeric fractional integer in the format:

```javascript
SECONDS.NANOSECONDS
```

### Seconds

It is the number of seconds that have elapsed since the _Unix epoch._

### Nanoseconds
where:

Fractional second or one thousand-millionth of a second.

{% hint style="info" %}
A timestamp always exists, either set by the Input plugin or discovered through a data parsing process.
{% endhint %}
- `_Seconds_` is the number of seconds that have elapsed since the Unix epoch.
- `_Nanoseconds_` is a fractional second or one thousand-millionth of a second.
esmerel marked this conversation as resolved.
Show resolved Hide resolved

## Match

Fluent Bit allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule.
Fluent Bit lets you deliver your collected and processed Events to one or multiple
destinations through a routing phase. A _Match_ represents a rule to select Events
esmerel marked this conversation as resolved.
Show resolved Hide resolved
where a Tag matches a defined rule.

To learn more about Tags and Matches check the [Routing](data-pipeline/router.md) section.
To learn more about Tags and Matches, see [Routing](data-pipeline/router.md).

## Structured Messages
## Structured messages

Source events can have or not have a structure. A structure defines a set of _keys_ and _values_ inside the Event message. As an example consider the following two messages:
Source events can have a structure. A structure defines a set of `keys` and `values`
inside the Event message to implement faster operations on data modifications.
Fluent Bit handles every Event message as a structured message.
esmerel marked this conversation as resolved.
Show resolved Hide resolved

### No structured message
Consider the following two messages:

```javascript
"Project Fluent Bit created on 1398289291"
```

### Structured Message
- No structured message
esmerel marked this conversation as resolved.
Show resolved Hide resolved

```javascript
{"project": "Fluent Bit", "created": 1398289291}
```
```javascript
"Project Fluent Bit created on 1398289291"
```

At a low level both are just an array of bytes, but the Structured message defines _keys_ and _values_, having a structure helps to implement faster operations on data modifications.
- With a structured message
esmerel marked this conversation as resolved.
Show resolved Hide resolved

{% hint style="info" %}
Fluent Bit **always** handles every Event message as a structured message.
For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/).

Consider [MessagePack](https://msgpack.org/) as a binary version of JSON on steroids.
{% endhint %}
```javascript
{"project": "Fluent Bit", "created": 1398289291}
```

For performance reasons, Fluent Bit uses a binary serialization data format called
[MessagePack](https://msgpack.org/).