Skip to content

Commit

Permalink
GitBook: [1.4-dev] 13 pages modified
Browse files Browse the repository at this point in the history
  • Loading branch information
edsiper authored and gitbook-bot committed Feb 2, 2020
1 parent d14d8ea commit 49ba24a
Show file tree
Hide file tree
Showing 13 changed files with 64 additions and 25 deletions.
24 changes: 16 additions & 8 deletions about/fluentd-and-fluent-bit.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,33 @@
---
description: The Production Grade Ecosystem
---

# Fluentd & Fluent Bit

Data collection matters and nowadays the scenarios from where the information can _come from_ are very variable. For hence to be more flexible in certain markets needs, we may need different options. On this page, we will describe the relationship between the [Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) open source projects.
Logging and data processing in general can be complex, and at scale a bit more, that's why [Fluentd](https://www.fluentd.org) was born. But now is more than a simple tool, it's a full ecosystem that contains SDKs for different languages and sub projects like [Fluent Bit](https://fluentbit.io).

[Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) projects are both created and sponsored by [Treasure Data](http://treasuredata.com) and they aim to solve the _collection_, _processing_, and _delivery_ of Logs.
On this page, we will describe the relationship between the [Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) open source projects, as a summary we can say both are:

Both projects share a lot of similarities, Fluent Bit is fully based on the design and experience of Fluentd architecture and general design. Choosing which one to use depends on the final needs, from an architecture perspective we can consider:
* Licensed under the terms of Apache License v2.0
* Hosted projects by the [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io)
* Production Grade solutions: deployed **thousands** of times every single day, **millions** per ****month.
* Community driven projects
* Widely Adopted by the Industry: trusted by all major companies like AWS, Microsoft, Google Cloud and hundred of others.
* Originally created by [Treasure Data](https://www.treasuredata.com).

* Fluentd is a log collector, processor, and aggregator.
* Fluent Bit is a log collector and processor \(it doesn't have strong aggregation features like Fluentd\).
Both projects share a lot of similarities, Fluent Bit is fully designed and built on top of the best ideas of Fluentd architecture and general design. Choosing which one to use depends on the end-user needs.

The following table describes a comparison in different areas of the projects:

| | Fluentd | Fluent Bit |
| :--- | :--- | :--- |
| Scope | Containers / Servers | Containers / Servers |
| Scope | Containers / Servers | Embedded Linux / Containers / Servers |
| Language | C & Ruby | C |
| Memory | ~40MB | ~650KB |
| Performance | High Performance | High Performance |
| Dependencies | Built as a Ruby Gem, it requires a certain number of gems. | Zero dependencies, unless some special plugin requires them. |
| Plugins | More than 650 plugins available | Around 50 plugins available |
| Plugins | More than 1000 plugins available | Around 50 plugins available |
| License | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) |

Consider Fluentd mainly as an Aggregator and Fluent Bit as a Log Forwarder, we can see both projects complement each other providing a full reliable solution.
Both Fluentd and Fluent Bit can work as Aggregators of Forwarders, they both can complement each other or use them as standalone solutions.

4 changes: 4 additions & 0 deletions about/license.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Strong Commitment to the Openness and Collaboration
---

# License

[Fluent Bit](http://fluentbit.io), including it core, plugins and tools are distributed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0):
Expand Down
3 changes: 2 additions & 1 deletion about/what-is-fluent-bit.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
# What is Fluent Bit ?

[Fluent Bit](http://fluentbit.io/) is an open source and multi-platform log processor tool which aims to be a generic Swiss knife for log processing and distribution.
[Fluent Bit](http://fluentbit.io/) is an open source and multi-platform log processor tool which aims to be a generic Swiss knife for logs processing and distribution.

Nowadays the number of sources of information in our environments is ever increasing. Handling data collection at scale is complex, and collecting and aggregating diverse data requires a specialized tool that can deal with:

* Different sources of information
* Different data formats
* Data Reliability
* Security
* Flexible Routing
* Multiple destinations

[Fluent Bit](https://fluentbit.io) has been designed with performance and low resources consumption in mind.
Expand Down
4 changes: 4 additions & 0 deletions concepts/data-pipeline/buffer.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Data processing with reliability
---

# Buffer

When the data or logs are ready to be routed to some destination, by default they are buffered in memory.
Expand Down
4 changes: 4 additions & 0 deletions concepts/data-pipeline/filter.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: 'Modify, Enrich or Drop your records'
---

# Filter

In production environments we want to have full control of the data we are collecting, filtering is an important feature that allows us to **alter** the data before delivering it to some destination.
Expand Down
4 changes: 4 additions & 0 deletions concepts/data-pipeline/input.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: The way to gather data from your sources
---

# Input

[Fluent Bit](http://fluentbit.io) provides different _Input Plugins_ to gather information from different sources, some of them just collect data from log files while others can gather metrics information from the operating system. There are many plugins for different needs.
Expand Down
4 changes: 4 additions & 0 deletions concepts/data-pipeline/output.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: 'Destinations for your data: databases, cloud services and more'
---

# Output

The output interface allows us to define destinations for the data. Common destinations are remote services, local file system or standard interface with others. Outputs are implemented as plugins and there are many available.
Expand Down
4 changes: 4 additions & 0 deletions concepts/data-pipeline/parser.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Convert Unstructured to Structured messages
---

# Parser

Dealing with raw strings is a constant pain; having a structure is highly desired. Ideally we want to set a structure to the incoming data by the Input Plugins as soon as they are collected:
Expand Down
4 changes: 4 additions & 0 deletions concepts/data-pipeline/router.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Create flexible routing rules
---

# Router

Routing is a core feature that allows to **route** your data through Filters and finally to one or multiple destinations.
Expand Down
23 changes: 11 additions & 12 deletions installation/kubernetes.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,25 @@
# Kubernetes

[Fluent Bit](http://fluentbit.io) is a lightweight and extensible **Log Processor** that comes with full support for Kubernetes:
---
description: Kubernetes Production Grade
---

* Read Kubernetes/Docker log files from the file system or through Systemd Journal.
* Enrich logs with Kubernetes metadata.
* Deliver logs to third party storage services like Elasticsearch, InfluxDB, HTTP, etc.
# Kubernetes

![](../.gitbook/assets/fluentbit_kube_logging%20%281%29.png)

Content:
[Fluent Bit](http://fluentbit.io) is a lightweight and extensible **Log Processor** that comes with full support for Kubernetes:

* [Concepts](kubernetes.md#concepts)
* [Installation Steps](kubernetes.md#installation)
* Process Kubernetes containers logs from the file system or Systemd/Journald.
* Enrich logs with Kubernetes Metadata.
* Centralize your logs in third party storage services like Elasticsearch, InfluxDB, HTTP, etc.

## Concepts <a id="concepts"></a>

Before geting started it is important to understand how Fluent Bit will be deployed. Kubernetes manages a cluster of _nodes_, so our log agent tool will need to run on every node to collect logs from every _POD_, hence Fluent Bit is deployed as a DaemonSet \(a POD that runs on every _node_ of the cluster\).
Before getting started it is important to understand how Fluent Bit will be deployed. Kubernetes manages a cluster of _nodes_, so our log agent tool will need to run on every node to collect logs from every _POD_, hence Fluent Bit is deployed as a DaemonSet \(a POD that runs on every _node_ of the cluster\).

When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information \(metadata\):

* POD Name
* POD ID
* Pod Name
* Pod ID
* Container Name
* Container ID
* Labels
Expand Down
5 changes: 2 additions & 3 deletions installation/requirements.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
# Requirements

[Fluent Bit](http://fluentbit.io) uses very low CPU and Memory consumption, it's compatible with most of x86, x86\_64, AArch32 and AArch64 based platforms. In order to build it you need the following components in your system:
[Fluent Bit](http://fluentbit.io) uses very low CPU and Memory consumption, it's compatible with most of x86, x86\_64, AArch32 and AArch64 based platforms. In order to build it you need the following components in your system for the build process:

* Compiler: GCC or clang
* CMake
* Flex \(only if Stream Processor is enabled\)
* Bison \(only if Stream Processor is enabled\)
* Flex & Bison if you enable the Stream Processor

There are not other dependencies besides _libc_ and _pthreads_ in the most basic mode. For certain features that depends on third party components, those are included in the main source code repository.

2 changes: 1 addition & 1 deletion installation/sources/download-source-code.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

For production systems, we strongly suggest that you always get the latest stable release from our web site, you can get the official tarballs \(.tar.gz\) from the following link:

[http://fluentbit.io/download/](http://fluentbit.io/download/)
[https://fluentbit.io/download/](https://fluentbit.io/download/)

## Development

Expand Down
4 changes: 4 additions & 0 deletions installation/upgrade-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ The following article cover the relevant notes for users upgrading from previous

For more details about changes on each release please refer to the [Official Release Notes](https://fluentbit.io/announcements/).

## Fluent Bit v1.4

If you are migrating from Fluent Bit v1.3, there are not breaking changes. Just new exciting features to enjoy :\)

## Fluent Bit v1.3

If you are migrating from Fluent Bit v1.2 to v1.3, there are not breaking changes. If you are upgrading from an older version please review the incremental changes below.
Expand Down

0 comments on commit 49ba24a

Please sign in to comment.