Skip to content

Commit

Permalink
@chalin copyedits
Browse files Browse the repository at this point in the history
  • Loading branch information
chalin committed Aug 19, 2022
1 parent a9cd561 commit 9e24102
Showing 1 changed file with 45 additions and 44 deletions.
89 changes: 45 additions & 44 deletions content/en/blog/2022/instrument-nginx.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,30 @@
---
title: Learn how to instrument nginx with OpenTelemetry
linkTitle: Instrument Nginx
date: 2022-08-10
date: 2022-08-18
author: >-
[Debajit Das](https://github.com/debajitdas), [Kumar
Pratyus](https://github.com/kpratyus) and [Severin
Pratyus](https://github.com/kpratyus), and [Severin
Neumann](https://github.com/svrnm)
draft: true # TODO: remove this line once your post is ready to be published
spelling: >
cSpell:ignore Debajit Kumar Pratyus Severin Neumann webserver xvfz tracestate
cSpell:ignore catalina javaagent jaegertracing protobuf WORKDIR
---

Apache and nginx are the most popular webservers. It's most likely that you are
using one of them in your application. In a [previous blog post] we showed you
how you can use the OpenTelemetry Apache Module to add observability to the
first one. In this blog post, you will learn how you can get the same for the
latter! Read on to learn how you can get observability for nginx!
Apache and nginx are the most popular web servers. It's most likely that you are
using one of them in your application. In a [previous blog post][], you learned
how to use the OpenTelemetry Apache Module to add observability to Apache. In
this blog post, you will learn how you can get observability for nginx!

## Install the nginx module

In the following we are going to use docker to run a nginx server with the
`ngx_http_opentelemetry_module.so` enabled and configured. Of course you can use
the same set of commands used in the `Dockerfile` below to configure a nginx
server on a bare matel machine.
In the following, you are going to use docker to run a nginx server with the
`ngx_http_opentelemetry_module.so` enabled and configured. Of course, you can
use the same set of commands used in the `Dockerfile` below to configure a nginx
server on a bare-metal machine.

Start with an empty directory. Create a file called `Dockerfile` and copy the
Start from an empty directory. Create a file called `Dockerfile` and copy the
following content into it:

```Dockerfile
Expand Down Expand Up @@ -68,20 +70,20 @@ This will enable the OpenTelemetry and apply the following configuration:
`DemoServiceNamespace` and the `service.instance_id` to `DemoInstanceId`
- Report traces as errors, so you will see them in the nginx log

To learn all the settings available, see the [full list of directives].
To learn all the settings available, see the [full list of directives][].

With the `Dockerfile` and nginx config in place, build your docker image and run
the container:

```shell
```console
$ docker build -t nginx-otel --platform linux/amd64 .
$ docker run --platform linux/amd64 --rm -p 8080:80 nginx-otel
...
2022/08/12 09:26:42 [error] 69#69: mod_opentelemetry: ngx_http_opentelemetry_init_worker: Initializing Nginx Worker for process with PID: 69
```

With the container up and running, send requests to nginx, e.g.
`curl localhost:8080`
With the container up and running, send requests to nginx using, for example,
`curl localhost:8080`.

Since the configuration above has `NginxModuleTraceAsError` set to `ON` and you
will see your traces dump to the error log of nginx:
Expand All @@ -105,9 +107,9 @@ Host, client: 172.17.0.1, server: localhost, request: "GET / HTTP/1.1", host: "l
## Viewing spans in Jaeger

At this point the telemetry data generated by nginx is not send to an
OpenTelemetry collector or any other observability backend. We can easily change
that by creating a `docker-compose` file, that starts the nginx server, the
collector and Jaeger:
OpenTelemetry collector or any other observability backend. You can easily
change that by creating a `docker-compose` file, that starts the nginx server,
the collector and Jaeger:

Create a file called `docker-compose.yml` and add the following content:

Expand Down Expand Up @@ -165,35 +167,35 @@ startup.

Get everything up and running:

```shell
```console
$ docker-compose up
```

In another shell create some traffic:
In another shell, create some traffic:

```shell
```console
$ curl localhost:8080
```

In your browser open [http://localhost:16686] and search
In your browser open [localhost:16686][] and search
for traces from `DemoService` and drill into one of them.

![A screenshot of the jaeger trace view, showing a waterfall of spans representing the time consumed by different nginx modules.](/img/instrument-nginx/nginx-spans-in-jaeger.png)
![A screenshot of the Jaeger trace view, showing a waterfall of spans representing the time consumed by different nginx modules.](/img/instrument-nginx/nginx-spans-in-jaeger.png)

You will see one span for each nginx module being executed during the request.
With that you can easily spot issues with certain modules, e.g. a rewrite going
mad.
With that you can easily spot issues with certain modules, for example, a
rewrite going mad.

## Put nginx between two services

Of course we rarely use nginx as a standalone solution! Most of the time it is
Of course, nginx is rarely used as a standalone solution! Most of the time it is
used as a reverse proxy or load balancer in front of another service. And, there
might be a service calling nginx to reach that down stream service.

We want to extend our example by two more services:
Add two more services to the running example:

- A node.js service called `frontend` that sits at the front and calls the nginx
- A java service called `backend` that sits behind the nginx.
- A Node.js service called `frontend` that sits at the front and calls the nginx
- A java service called `backend` that sits behind the nginx

Update the `docker-compose` file to contain those 2 services and to overwrite
the `default.conf` in nginx:
Expand Down Expand Up @@ -249,7 +251,7 @@ server {

Create two empty folders `backend` and `frontend`.

In the frontend folder we create a simple nodejs app:
In the frontend folder, create a simple Node.js app:

```javascript
const opentelemetry = require("@opentelemetry/sdk-node");
Expand Down Expand Up @@ -302,8 +304,8 @@ EXPOSE 8000
CMD [ "node", "app.js" ]
```

For the backend service we are going to use a tomcat with the OpenTelemetry
javaagent installed. For this, create a `Dockerfile` like the following in the
For the backend service, you are going to use Tomcat with the OpenTelemetry
Java agent installed. For this, create a `Dockerfile` like the following in the
`backend` folder

```Dockerfile
Expand All @@ -313,7 +315,7 @@ ENV JAVA_OPTS="-javaagent:javaagent.jar"
CMD ["catalina.sh", "run"]
```

As you can see that `Dockerfile` downloads and adds the OpenTelemetry javaagent
As you can see, the `Dockerfile` downloads and adds the OpenTelemetry Java agent
for you automatically.

You should now have the following files in your top level directory:
Expand All @@ -329,7 +331,7 @@ You should now have the following files in your top level directory:

With everything in place, you can now start the demo environment:

```shell
```console
$ docker-compose up
```

Expand All @@ -342,26 +344,25 @@ Within a few moments you should have five docker containers up and running:
- Backend

Send a few requests to the frontend with `curl localhost:8000` and then check
the Jaeger UI in your browser at
[http://localhost:16686]. You should see traces going
the Jaeger UI in your browser at [localhost:16686]. You should see traces going
from frontend to nginx to backend.

The frontend trace should indicate an error, since nginx is forwarding the
`Page Not Found` from tomcat.
`Page Not Found` from Tomcat.

![A screenshot of the jaeger trace view, showing a waterfall of spans going from the frontend to nginx down to the backend.](/img/instrument-nginx/nginx-spans-in-jaeger.png)

## What's next?

You now should be able to translate the learnings from this blog post into your
own installation of nginx. We are curious to hear about your experience! And, if
you run into any problems, [raise an issue].
You should now be able to apply what you have learned from this blog post to
your own installation of nginx. We would love to hear about your experience! If
you run into any problems, [create an issue][].

[create an issue]:
https://github.com/open-telemetry/opentelemetry-cpp-contrib/issues
[full list of directives]:
https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/otel-webserver-module#configuration-1
[localhost:16686]: http://localhost:16686
[opentelemetry-webserver-sdk-x64-linux]:
https://github.com/open-telemetry/opentelemetry-cpp-contrib/releases/download/webserver%2Fv1.0.0/opentelemetry-webserver-sdk-x64-linux.tgz.zip
[previous blog post]: /blog/2022/instrument-apache-httpd-server/
[raise an issue]:
https://github.com/open-telemetry/opentelemetry-cpp-contrib/issues
[http://localhost:16686]: http://localhost:16686

0 comments on commit 9e24102

Please sign in to comment.