Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blog: Scaling OpenTelemetry Collectors using Ansible #4182

Merged
merged 41 commits into from
Apr 15, 2024
Merged
Changes from 24 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
91d0c77
add blog for otelcol-ansible
ishanjainn Mar 19, 2024
e2bdd24
Merge branch 'main' into otel-ansible
ishanjainn Mar 19, 2024
175ff3a
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
bf58bb0
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
0b75fed
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
0b103f7
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
883cd02
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
63b8bfc
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
ab6aa24
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
3743fa6
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
3de9b49
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
df72dba
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
10e80dc
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
bb0ba96
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
1250d2e
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
c3ce486
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
676a8e7
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
40e03d6
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
1321e7d
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
39723a5
Update content/en/blog/2024/scaling-opentelemetry-collectors.md
ishanjainn Mar 19, 2024
f1f4693
Merge branch 'main' into otel-ansible
ishanjainn Mar 19, 2024
0cedfc3
update blog content
ishanjainn Mar 19, 2024
79f6ff3
remove setup hosts step
ishanjainn Mar 19, 2024
c138901
Update scaling-opentelemetry-collectors.md
theletterf Mar 19, 2024
35fd956
Typo
theletterf Mar 19, 2024
58acec5
Results from /fix:all
opentelemetrybot Mar 19, 2024
1477e4a
add link to grafan for flow
ishanjainn Mar 20, 2024
36a57b3
rephrase the ansible inventory section
ishanjainn Mar 20, 2024
68f3d70
find:fix
ishanjainn Mar 20, 2024
62c9425
Merge branch 'main' into otel-ansible
theletterf Mar 26, 2024
197716c
Results from /fix:all
opentelemetrybot Mar 26, 2024
8873b13
Merge branch 'main' into otel-ansible
ishanjainn Apr 2, 2024
3061cd4
update blog according to suggestions
ishanjainn Apr 10, 2024
b7df8c1
Merge branch 'main' into otel-ansible
chalin Apr 12, 2024
588c459
Results from /fix:all
opentelemetrybot Apr 12, 2024
e029d58
Merge branch 'main' into otel-ansible
chalin Apr 12, 2024
4ef8ce6
Apply suggestions from code review
svrnm Apr 15, 2024
85c3a1c
Merge branch 'main' into otel-ansible
svrnm Apr 15, 2024
bbec3c9
Set date
svrnm Apr 15, 2024
5ad02fb
Results from /fix:dict
opentelemetrybot Apr 15, 2024
206dbc5
Merge branch 'main' into otel-ansible
svrnm Apr 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
221 changes: 221 additions & 0 deletions content/en/blog/2024/scaling-opentelemetry-collectors.md
chalin marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,221 @@
---
title: Manage OpenTelemetry Collectors at scale with Ansible
linkTitle: OTel Collector with Ansible
chalin marked this conversation as resolved.
Show resolved Hide resolved
date: 2024-03-12
chalin marked this conversation as resolved.
Show resolved Hide resolved
author: '[Ishan Jain](https://github.com/ishanjainn) (Grafana)'
cSpell:ignore: ansible Ishan Jain
---

You can scale the deployment of
[OpenTelemetry Collector](/docs/collector/deployment/) across multiple Linux
hosts through [Ansible](https://www.ansible.com/), to function both as
[gateways](/docs/collector/deployment/gateway/) and
[agents](/docs/collector/deployment/agent/) within your observability
architecture. Using the OpenTelemetry Collector in this dual capacity enables a
robust collection and forwarding of metrics, traces, and logs to analysis and
visualization platforms.

We outline a strategy for deploying and managing the OpenTelemetry Collector's
scalable instances throughout your infrastructure using Ansible. In the
following example, we'll use Grafana as the target backend for metrics.

## Prerequisites

To follow this guide, make sure you meet the following requirements:

- Ansible Installed in your system
- Linux hosts along with SSH access to each of these Linux hosts
- Prometheus for gathering metrics
chalin marked this conversation as resolved.
Show resolved Hide resolved

## Install the Grafana Ansible collection

The
[OpenTelemetry Collector role](https://github.com/grafana/grafana-ansible-collection/tree/main/roles/opentelemetry_collector)
is provided through the
[Grafana Ansible collection](https://docs.ansible.com/ansible/latest/collections/grafana/grafana/)
as of the 3.0.0 release.
chalin marked this conversation as resolved.
Show resolved Hide resolved

To install the Grafana Ansible collection, run this command:

```shell
ishanjainn marked this conversation as resolved.
Show resolved Hide resolved
ansible-galaxy collection install grafana.grafana
```

## Create an Ansible inventory file

Next, set up your hosts and assoicated IPs and create an inventory file.

Check warning on line 46 in content/en/blog/2024/scaling-opentelemetry-collectors.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (assoicated)

This example uses eight Linux hosts: two Ubuntu hosts, two CentOS hosts, two
Fedora hosts, and two Debian hosts.

1. Create an Ansible inventory file.

The Ansible inventory, which resides in a file named `inventory`, looks
similar to this:

```ini
ishanjainn marked this conversation as resolved.
Show resolved Hide resolved
10.0.0.1 # hostname = ubuntu-01
10.0.0.2 # hostname = ubuntu-02
10.0.0.3 # hostname = centos-01
10.0.0.4 # hostname = centos-02
10.0.0.5 # hostname = debian-01
10.0.0.6 # hostname = debian-02
10.0.0.7 # hostname = fedora-01
10.0.0.8 # hostname = fedora-02
```

> **Note**: When copying the previous snippet, remove the comments.

2. Create an `ansible.cfg` file within the same directory as `inventory`, with
the following values:

```cfg
ishanjainn marked this conversation as resolved.
Show resolved Hide resolved
[defaults]
inventory = inventory # Path to the inventory file
private_key_file = ~/.ssh/id_rsa # Path to private SSH Key
remote_user=root
```

## Use the OpenTelemetry Collector Ansible role

Next, define an Ansible playbook to apply your chosen or created OpenTelemetry
Collector role across your hosts.

Create a file named `deploy-opentelemetry.yml` in the same directory as your
`ansible.cfg` and `inventory`.
ishanjainn marked this conversation as resolved.
Show resolved Hide resolved

```yaml
- name: Install OpenTelemetry Collector
hosts: all
become: true

tasks:
- name: Install OpenTelemetry Collector
ansible.builtin.include_role:
name: opentelemetry_collectorr
vars:
otel_collector_receivers:
hostmetrics:
collection_interval: 60s
scrapers:
cpu: {}
disk: {}
load: {}
filesystem: {}
memory: {}
network: {}
paging: {}
process:
mute_process_name_error: true
mute_process_exe_error: true
mute_process_io_error: true
processes: {}

otel_collector_processors:
batch:
resourcedetection:
detectors: [env, system]
timeout: 2s
system:
hostname_sources: [os]
transform/add_resource_attributes_as_metric_attributes:
error_mode: ignore
metric_statements:
- context: datapoint
statements:
- set(attributes["deployment.environment"],
resource.attributes["deployment.environment"])
- set(attributes["service.version"],
resource.attributes["service.version"])

otel_collector_exporters:
prometheusremotewrite:
endpoint: https://<prometheus-url>/api/prom/push
headers:
Authorization: 'Basic <base64-encoded-username:password>'

otel_collector_service:
pipelines:
metrics:
receivers: [hostmetrics]
processors:
[
resourcedetection,
transform/add_resource_attributes_as_metric_attributes,
batch,
]
exporters: [prometheusremotewrite]
```

{{% alert title="Note" %}}

Adjust the configuration to match the specific telemetry you
intend to collect and where you plan to forward it. This configuration snippet
is a basic example designed for collecting host metrics and forwarded to
Prometheus.

{{% /alert %}}

The previous configuration would provision the OpenTelemetry Collector to
collect host metrics from the Linux host.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You write: "collect host metrics from the Linux host", which is the host? Was one designated, if so, this should have been identified earlier IMHO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kind of updated but did you mean that I should position this before?


## Running the Ansible playbook

Deploy the OpenTelemetry Collector across your hosts by running
the following command:

```sh
ansible-playbook deploy-opentelemetry.yml
```

## Check your metrics in the backend

After your OpenTelemetry Collector's start sending metrics to Prometheus, follow
ishanjainn marked this conversation as resolved.
Show resolved Hide resolved
these quick steps to visualize them in Grafana:
ishanjainn marked this conversation as resolved.
Show resolved Hide resolved

### Set up Grafana

1. **Install Docker**: Make sure Docker is installed on your system.

2. **Run Grafana Docker Container**: Start a Grafana server with this Docker
ishanjainn marked this conversation as resolved.
Show resolved Hide resolved
command, which fetches the latest Grafana image:

```sh
docker run -d -p 3000:3000 --name=grafana grafana/grafana
```

3. **Access Grafana**: Navigate to `http://localhost:3000` in your web browser.
ishanjainn marked this conversation as resolved.
Show resolved Hide resolved
The default login details are `admin` for both the username and password.
ishanjainn marked this conversation as resolved.
Show resolved Hide resolved

4. **Change Password**: Upon your first login, you will be prompted to set a new
password. Make sure to pick a secure one.
ishanjainn marked this conversation as resolved.
Show resolved Hide resolved

For other installation methods and more detailed instructions, refer to the
official Grafana documentation.

### Add Prometheus as a data source

1. In Grafana, navigate to **Connections** > **Data Sources**.
2. Click **Add data source** and select **Prometheus**.
3. In the settings, enter your Prometheus URL, for example,
`http://<your_prometheus_host>`, along with any other necessary details.
4. Select **Save & Test**.

### Explore your metrics

1. Go to the **Explore** page
2. In the Query editor, select your data source and enter the following query

```PromQL
100 - (avg by (cpu) (irate(system_cpu_time{state="idle"}[5m])) * 100)
```

This query calculates the average percentage of CPU time not spent in the
"idle" state, across each CPU core, over the last 5 minutes.

3. Explore other metrics and create dashboards to gain insights into your
system's performance.

This guide makes it easier for you to set up the OpenTelemetry Collector on
several Linux machines with the help of Ansible and shows you how to see the
data it collects in Grafana.
chalin marked this conversation as resolved.
Show resolved Hide resolved
svrnm marked this conversation as resolved.
Show resolved Hide resolved
Loading