OTLP Exporter for Logs

[kaylareopelle]

Our current OTLP exporter is set up to exclusively send traces. This issue will either adapt current exporters or create a new, temporary exporter to send logs over OTLP. Support for HTTP will likely come first, followed by gRPC.

[github-actions]

👋 This issue has been marked as stale because it has been open with no activity. You can: comment on the issue or remove the stale label to hold stale off for a while, add the keep label to hold stale off permanently, or do nothing. If you do nothing this issue will be closed eventually by the stale bot.

[travisbell]

Hey @kaylareopelle, do you know where this project is at? Just curious when we might get to use a more generalized logging exporter. Thanks.

[kaylareopelle]

Hi @travisbell thanks for checking in! Apologies about the delay here.

There is an unreviewed OTLP logs exporter, along with other code to make Ruby spec-compliant for the Logs signal, available in this PR on my fork: kaylareopelle#6

The current process is to release small parts of this code to the main opentelemetry-ruby repository for review, and then revise the WIP branch as needed. I don't have a good sense of when all those code will be merged into main.

If you'd like to test out the WIP logs implementation, you can follow the instructions in this gist: https://gist.github.com/kaylareopelle/76079747c1a7528449670b8bd1c78893

I also have a demo repository that integrates OpenTelemetry logs into a small Rails application: https://github.com/kaylareopelle/otel_ruby_logs_demo

I'm updating the branches about once a week to stay in sync with the latest changes upstream.

If you decide to give this experimental code a try, please let me know how it goes. You can comment here, open an issue on the demo repo, or add a comment to the gist.

[kaylareopelle]

WIP, unreviewed code for this issue, along with other parts of the logs SDK, can be found in kaylareopelle#6

[cidemaxio]

@kaylareopelle What progress has been made and what else remains to be done? We would like to use this for logging from our ruby code.

[kaylareopelle]

Hi @cidemaxio! Thanks for reaching out. It's great to hear you'd like to use OTel Ruby logs.

Here's the project board for the Logs signal that tracks what remains to be done. I have a draft PR for the OTLP exporter on my fork with a todo list in the description for work to do before opening the PR in the main repo.

There is at least one other PR that needs to be opened for the SDK/API regarding configuration and instrument registries. After both that PR and #1696 are merged, we should be ready to release the logs SDK, API and OTLP exporter gems.

[kaylareopelle]

Hi, @cidemaxio! PR is now opened for the OTLP Log Record Exporter: #1727

Please take a look if you're interested! Additional feedback from the community may help speed up the process to merge/release the gem.

The other PR related to the SDK/API I mentioned in my last comment has been opened too: #1725

[cidemaxio]

Thanks for the update @kaylareopelle! We will look at that PR.

[wsmoak]

@kaylareopelle both PRs seem to have conflicts. Could you bring them up to date with main if you have a chance?

[kaylareopelle]

Thank you, @wsmoak! The merge conflicts have been resolved 😄

[wsmoak]

Thanks @kaylareopelle .

I was able to merge the changes from both PRs to a branch locally and try it out in a simple Rails app based on the example above. I have the collector running in a Docker container according to the quickstart [1], and I can see the logs being sent to the collector.

[1] with the addition of port 4318 in the docker run command

[kaylareopelle]

This is fantastic news. Thanks for testing, @wsmoak!

Where did you find the docker run command? I'm wondering if there's a place we could update the documentation to include your port finding.

It's a bit off-topic for this issue, but if you're interested in trialing a Ruby Logger bridge to automatically emit your Rails logs as OpenTelemetry Logs, you can pull in the branch on this PR: open-telemetry/opentelemetry-ruby-contrib#983

This is intended for the standard Ruby Logger class. If you're using semantic logger, there's a prototype here: reidmorrison/semantic_logger#292

[wsmoak]

The docker run command I mentioned is the one in the otel collector quickstart: https://opentelemetry.io/docs/collector/quick-start/ -> step 3

I had that running, but when I started the Rails app, it complained that it couldn't export logs. Could this error have more details about where it was trying to send them?
E, [2024-10-17T18:47:03.126824 #86414] ERROR -- : OpenTelemetry error: Unable to export 3 log records

I eventually sorted out that the Ruby SDK was speaking HTTP and nothing was listening on 4318 in the collector.

(And I did see it wrapping the Rails logs already, I think I picked up the contrib gem from your otel_ruby_logs_demo Rails example.) Thanks!

[kaylareopelle]

That's great feedback! I'll see what I can do about the logs exporter message.

Thanks for pointing out the command. I opened open-telemetry/opentelemetry.io#5441 to see if we can get that added.

So glad to hear you found the demo! 🎉

[kaylareopelle]

Hi @wsmoak, I updated the OTLP log exporter to log an OpenTelemetry error with more detail whenever it returns a FAILURE status code. Could you give it a try and let me know if that information would have helped debug the connection problem you encountered?

Here's the commit that introduced the logging: abfcd54

[marcelaraujo]

@kaylareopelle can you update your repository with the latest changes from main repo?

[kaylareopelle]

Hi @marcelaraujo! Thank you for the reminder! I'm in the process of updating the branch with the latest changes. There are a few discrepancies I want to take a closer look on and a few more tests I want to run. I'm not certain if the latest push works. I'll add a comment here when it's ready to go.

[marcelaraujo]

Hi @kaylareopelle

Yeap, I'm facing some issues. I forked the OTEL repo and merged your changes into my fork, but it's now failing.

Your bundle is locked to opentelemetry-exporter-otlp (0.1.0) from https://github.com/marcelaraujo/opentelemetry-ruby (at main@77da721,
glob: exporter/otlp/*.gemspec), but that version can no longer be found in that source. That means the author of
opentelemetry-exporter-otlp (0.1.0) has removed it. You'll need to update your bundle to a version other than opentelemetry-exporter-otlp
(0.1.0) that hasn't been removed in order to install.

[kaylareopelle]

Hi @marcelaraujo! The branch is updated and ready for use. 🚀

[marcelaraujo]

Hi @kaylareopelle

I was able to install the deps at the first run, but if I run bundle install again, it fails

root@4766442dcedf:/app# bundle install
Fetching gem metadata from https://rubygems.org/........
Your bundle is locked to opentelemetry-exporter-otlp (0.1.0) from https://github.com/kaylareopelle/opentelemetry-ruby (at log-record-processor3@31ff063, glob: exporter/otlp/*.gemspec), but that version can no longer be found in
that source. That means the author of opentelemetry-exporter-otlp (0.1.0) has removed it. You'll need to update your bundle to a version other than opentelemetry-exporter-otlp (0.1.0) that hasn't been removed in order to
install.

[kaylareopelle]

Hi @marcelaraujo. My apologies, I should've mentioned this in my earlier comment.

To use experimental logs, we no longer need to install opentelemetry-exporter-otlp from the branch source. I had a few problems with versioning when I initially copied the OTLP traces exporter to build the OTLP logs exporter.

If you'd like to export traces, I recommend installing the latest version of opentelemetry-exporter-otlp from RubyGems. Do not install it from my fork.

# OTLP Exporters with logging enhancements
+ gem 'opentelemetry-exporter-otlp'
- gem 'opentelemetry-exporter-otlp',
-   git: 'https://github.com/kaylareopelle/opentelemetry-ruby',
-   branch: 'log-record-processor3',
-   glob: 'exporter/otlp/*.gemspec'

I'll update my gist to stop installing opentelemetry-exporter-otlp from my fork.

If you install the gem from source and still run into this problem, let me know. Something else may be going on.

[kaylareopelle]

@marcelaraujo - Something else was going on with the OTLP exporter. 🤦

The namespaces for the OTLP exporter for traces and the OTLP exporter for logs collided because they both had the same constant. I've updated the OTLP log exporter now to have its own unique namespace: OpenTelemetry::Exporter::OTLP::Logs. This follows the pattern we implemented for metrics. You can find the full discussion here.

This fix is now part of the log-record-processor3 branch on my fork.

---

Update: Though this was the reason why the opentelemetry-exporter-otlp gem couldn't be installed from the log-record-processor3 branch, there are no changes to the opentelemetry-exporter-otlp library in the log-record-processor3 branch, so you do not need to install opentelemetry-exporter-otlp from my fork.