Create Open Telemetry receiver

[AllanOricil]

Create an open telemetry receiver to scrape nebula logs from Salesforce

Use the one made for cloudwatch logs as an example
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver%2Fawscloudwatchreceiver%2FREADME.md

[jongpie]

@AllanOricil completely agree, my plan/hope is to use OpenTelemetry for bidirectional logging

• Issue Add an Apex RestResource class provide logging capabilities to external inbound integrations #635 is an enhancement to provide the ability for external integrations to log data in a Salesforce org
• I am also considering creating a batchable or queueable job that will take Nebula Logger data, convert it to OpenTelemetry's format, and send it to an OpenTelemetry endpoint

These 2 enhancements would let you send OpenTelemetry-formatted logging data to or from a Salesforce org. I don't have a timeline of when this will be added, but it's definitely on the roadmap.

[manwe651]

I am also considering creating a batchable or queueable job that will take Nebula Logger data, convert it to OpenTelemetry's format, and send it to an OpenTelemetry endpoint

This would be absolutely incredible, and I'm very interested in the progress. We have a lot of external processing for our Salesforce instance in AWS, and I'd love to have a unified view of all of our logs.

[jongpie]

@manwe651 thanks for the feedback! There has been a lot of interest lately in adding OpenTelemetry (OTel) support to Nebula Logger, and I don't have everything fully documented/represented here in GitHub at the moment - so to give everyone an update, here's a lengthy summary of my plan & current progress:

In Progress: Initial OpenTelemetry (OTel) Features

Issue #635 is actively being developed, and my current hope is for it to be part of Nebula Logger's Winter '25 release next month-ish. It will add 2 main features related to OpenTelemetry support:

1. The ability to pull/retrieve Nebula Logger data from your org, and return the data in OpenTelemetry's format. This will be helpful for people that want to have some external tool/system easily extract Nebula Logger's data using OTel's format (and then store it in a centralized logging/observability tool, etc.)
   • For the initial release, it will have some basic filtering & pagination capabilities - I'm still trying to finalize exactly what filters will be available, but it will definitely have filters for things like the logging level & timestamp of the log entries
2. The ability for external systems to store their logs in your Salesforce org, using OpenTelemetry's format. External systems will be able to send their OTel-formatted logging data to a custom REST endpoint in your org, and it'll be converted/stored in your org in Nebula Logger's custom objects (alongside your logs for Apex, Flow, and lightning components). This is useful for situations where some external system is used to do processing off-platform, but Salesforce admins/devs/architects would still like to have some insight into what happened in the external system
   • I have a functioning prototype of this, but it's not fully compliant yet with OTel's JSON format, so there's still a bit more work left

Long-Term: Push Nebula Logger Data to an OTel Endpoint + New Notification System

Once #635 is implemented, it should be fairly easy to build a basic batchable/queueable job to send your (Nebula Logger) logging data, in OTel format, to an OTel-compliant endpoint. But my plan is to wait to introduce an official batchable/queueable job for a bit, because I want to tie that feature into another overlapping effort that's also ongoing: adding a new notification system into Nebula Logger.

I'm using issue #593 to track this effort overall - the idea is for Nebula Logger to have built-in features to support being notified about some/all records in Nebula Logger. My goal is for it to provide:

• The ability to declaratively add notification rules for LogEntry__c records. You'll be able to specify what records you want to be notified about (e.g., you want an alert about any LogEntry__c records with LoggingLevel__c = 'ERROR' and Message__c contains 'some string')
  • This will leverage the new FormulaEval namespace in Apex (once it's generally available) so you can use Salesforce's formula syntax (used in validation rules, Flow formulas, etc) to add your criteria for which LogEntry__c you care about. My comment here has a little more context on the timeline for this.
• The ability to send notifications with a variety of tools. It will eventually include the ability to send notifications using...
  1. Email alerts
  2. Slack channel (and the existing Slack plugin would be deprecated/no longer needed)
  3. ➡️ OpenTelemetry endpoint 🥳 You'll be able to specify any OpenTelemetry-compliant endpoint, and any matching log entries (based on the notification rule you setup) will be auto-sent, in OTel format, to your OTel endpoint
     - 🥇 Bonus - the ability to send Nebula Logger data between Salesforce orgs 😎 For example, you could have your sandboxes setup to send important log entries to your production org, making your production org a centralized place to see logs from all of your Salesforce orgs
  4. Other tools like MS Team (lower priority)
  5. Notification plugin to add your own custom Apex classes to run for the notification (lower priority)

So, there should definitely be some OTel-related progress soon, though the new batchable/queueable job in particular may not be ready for a while.

If anyone has feedback/questions about of the above plan, please let me know!

[AllanOricil]

@jongpie why not do all the processing outside of Salesforce, in the loader? For example, Cloudwatch logs are scraped by its receiver as is, we don't need to deploy an additional service in aws just to prepare logs for open telemetry. The receiver takes it as is, then it hand over to the processors to format it to comply with the destination, and after that they are sent out.

[Salesforce] <- receiver (pulls) -> processors (transformations/aggregations) -> exporters -> [Signoz]

You could just create and maintain a receiver, and base transformers for the most important tools out there. There wouldn't be necessary to add a job in salesforce if the receiver "pulls" and hand the transformations over to the processors.

I suggested Signoz, in the above example, because it is a new open source tool that supports open telemetry out of the box. It is like datadog, but it can be self-hosted. It helps a lot to analyze logs.

[jongpie]

@AllanOricil certainly some of that could be built off platform, and there's nothing preventing someone from using that approach - but that is not something that I'm planning to provide as part of Nebula Logger. One of Nebula Logger's advantages is that it provides native observability directly in Salesforce, without any external dependencies. This could one day change, but I don't see a need to go off platform to provide OTel support.

You could just create and maintain a receiver, and base transformers for the most important tools out there.

The word "just" is also doing a lot of work in that sentence - you're proposing building & hosting a service, including providing tool-specific transformations. That is a huge effort, and in my opinion, building anything that's vendor-specific or tool-specific defeats the purpose of trying to leverage OpenTelemetry as a standard.

My plan is for Nebula Logger to have options for accessing logging data in OTel format, using REST/HTTP and JSON formatted according to OTel's logs data model.

• Push OTel data out of Salesforce: Having Nebula Logger push OTel data to an OTel endpoint will be one option
• Pull OTel data out of Salesforce: There will also be a custom endpoint that transforms & returns the data in Nebula Logger's custom objects in OTel's format

This will provide a consistent transformation of Nebula Logger's data model into OTel's data model, regardless of what external tool people are using. For some observability tools, there may be some additional transformations needed, but I'm considering that kind of work to be outside of the scope of Nebula Logger's role.

[AllanOricil]

he word "just" is also doing a lot of work in that sentence - you're proposing building & hosting a service, including providing tool-specific transformations.

The word "just" was refering to the amount of work if you follow my suggestion, which I believe reduces the amount of work because it wouldn't be necessary to add changes to your logger library, only create the 3 components (receiver, processors, exporters) of a collector. Let me try to explain my reasoning for this. Please answer the following questions separately.

A use case, for context.

I have 2 services that belong to the same transaction. One runs in Salesforce, and the other in AWS, which is a very simple lambda function. The transaction starts in Salesforce, goes to this service, and comes back to Salesforce. This transaction creates logs for both services. It is a pain to read and aggregate logs produced by 2 services running on different places, each using its own logger library. To solve this problem, I need to format, aggregate and send those logs to a single place, such as Signoz or DataDog, where I will be able to read, and relate them.

To extract, format, aggregate and export those logs, I decided to use Open Telemetry, and its series of open source components: receiver, processors and exporters. I'm going to setup a EC2 and start an "otel collector" agent. I'm going to configure the collector's manifest to scrape logs from both Salesforce and AWS, as they are, using their respectively "receiver". The logs generated by each service are using different formats, because they are both using different logger libraries. The logs from Salesforce are created by NebulaLogger, and the ones from AWS Winston. In this manifest, I will also configure a log exporter to send those logs to my tool, which is Signoz but it could also be DataDog.

Questions:

For logs from AWS, I don't need to deploy a job that is going to "prepare" and push logs to Signoz. The "otel collector" agent that is running in my EC2 was configured to scrape those logs using a "receiver". For Salesforce, the same will happen. Its "receiver" will scrape logs from Salesforce using the "tooling api". The "processor" will then format those, and finally the "exporter" will send them to the tool I configured, which is Signoz. Such as in the picture below

As explained above, if the whole log extraction, formatting, aggregation, export can happen in the "collector", using each of its components, and both native APIs (tooling and cloudwatch), why do I need an extra API and an extra job running in Salesforce to process those logs? If a problem can be solved in a single place, using a single tool, and native apis, I don't see a reason for having part of the job running in Salesforce. That is why I'm seeing duplicate effort. As soon as someone creates a receiver which uses the native tooling api, and a processor that can process NebulaLogger logs, there is no need for also doing this job inside Salesforce.

Do you still think a job and an extra api is required? If so, can you explain why, and how I would configure it in my otel collector manifest, please?

Additionaly, Salesforce has tons of limits, could you predict the throughput of logs comming out of Salesforce if the "extraction" uses a "push" mechanism? I don't want logs to arrive to my tool with a huge delay, or even arrive in the next day. How much bandwidth from other jobs already running in Salesforce, will this processing and push jobs consume? Maybe orgs already have too many jobs running that adding another one will be just "too much", causing process that must run once a day to be delayed.

I want it to be plug and play. I don't want to manage jobs inside Salesforce to "pre process" my logs, or push them to somewhere. My "processor", running in the otel collector agent, in the EC2, will do all the full pipeline to me. The same way it does for my Cloud Watch logs.

That is a huge effort, and in my opinion, building anything that's vendor-specific or tool-specific defeats the purpose of trying to leverage OpenTelemetry as a standard.

I'm not sure why you understood I said that. I was trying to explain that we "just" need a "processor" that can process "NebulaLogger logs", which produces a specific log format. You would create this "processor" to prepare the logs to be exported to anywhere, like in the image below.

• Push OTel data out of Salesforce: Having Nebula Logger push OTel data to an OTel endpoint will be one option

This shouldn't be an option. As soon as receiver and a processor is created, which will run outside of Salesforce, in a environment with way less processing and mem restrictions, why would a someone opt for the "push" job running inside Salesforce? What are the advantages? I can only see disavantages, and no trade-offs

I also don't see a reason for an "Open Telemetry compliant rest API" to receive logs inside Salesforce. Why would I use Salesforce to read logs? What is the use case?

[jamessimone]

I'm not sure if you meant to attach the stock image from OTel's Collector documentation page twice - I'm also not sure if the image itself is really unpacking much for the audience. Hidden, for example, in your "just ...create the 3 components (receiver, processors, exporters) of a collector" quote are little pieces of complexity like:

• authentication - I've always loved describing authentication as simple
• the actual nuances of building receivers - nothing screams "simple" like 14 subheadings on a page
• the actual nuances of exporting (I'll just use the Cloudwatch exporter as an example) aren't exactly terse even when working with something as simple as JSON from the get-go

So if we are to take the "receiver, processors, and exporters" part of that quote, 2 out of the 3 mentioned steps are actually fairly non-trivial (and that's prior to handling the authentication piece). It doesn't take a lot of investigative work to understand why the OTel registry has dozens of entries for popular languages like C#, Rust, and Java for wrapping common processes - like AWS Lambda invocations - within OTel-compliant logs: because it's much easier and far less complicated to transform data in the host language prior to pushing it to an endpoint (and the same holds true for importing data from Salesforce using an OTel-compliant endpoint, if we can do something as "simple" as assume authentication has been handled).

On the subject of complexity - it's my personal opinion that representing complex tasks as simple serves to obfuscate both the time commitment in terms of authoring functionality, as well as the maintenance time spent on existing code (which, in general, tends to far outweigh the original time costs of any piece of authored code/configuration - there are dozens if not hundreds of studies that have been done about this). In our role as software engineers, accurately and without ego representing complexity is extremely important. I don't think anybody minds when somebody takes their own time (however long that may take) to contribute something they believe is important to open source; conversely, I think many people mind when it's suggested that somebody else "should" do that work in an opinionated way.

I'd caution you against using words like "should" or "shouldn't" (citation) when talking about possibilities. I think we can all agree that because this discussion is occurring within the realm of programming (where most things are possible), there are many ways this functionality could be implemented. The introduction of "should" and "shouldn't" doesn't serve anybody. Luckily, since this is open source, you're always free to fork and/or contribute the functionality you'd like to see in the way you'd like for it to be implemented.

---

As an aside, there are plenty of counter-examples when it comes to answering questions like: "Why would I use Salesforce to read logs?" I've worked for several companies that wanted their Salesforce org to serve as the source of truth for combined logging; I've worked at places where exporting logs to a log aggregator was also common. That's really apropos of nothing, other than to say that if we consider the two of us, it's pretty telling if 50% of the sample size has had experience with those use-cases.