Generic event support

[graphaelli]

Describe the solution you'd like

A simple intake for arbitrary events including a message to be used for annotations and low volume logs. eg document:

{
  "@timestamp": "2019-02-12T18:53:42.281Z",
  "agent": {
    "name": "elastic-foo",
    "version": "1.2.3"
  },
  "processor": {
    "event": "custom",
    "name": "event"
  },
  "labels": {
    "action": "deployment",
  },
  "message": {
    "original": "some message"
  },
  "service": {
    "name": "someservice"
  },
}

Describe alternatives you've considered

• Could abuse error.exception.log but it's very difficult to split these into their own index, plus semantics are all wrong.
• Users can already simply report this kind of information directly to Elasticsearch, but often apm data is stored separately from application data and users without direct access to es but with access to an apm-server would be out of luck.

Additional detail

Just starting a discussion here.

• Are there use cases where agent and service should be omitted?
• Should there be a way to select which metadata is included, eg include_process?
• Is there a good reason to avoid log in favor of message? Log is already in ecs.

[felixbarny]

What are some of the use cases you have in mind? Seems like deployment tracking would be one? Can events have links to a span/transaction/trace?

[graphaelli]

Certainly deployment tracking - version promotion/rollback, autoscaling events, failovers, CI status - but really any type of annotation or notable event. I'm not proposing integration with APM UI here, yet. These would be immediately useful on custom TSVBs . Optional links to our other events (trace.id, etc) certainly makes sense.

[felixbarny]

• Is there a good reason to avoid log in favor of message? Log is already in ecs.

I would really like to avoid the term log here. People associate that with, well, logging and that's not the use case we have in mind for this.

to be used for annotations and low volume logs.

What about calling it annotations? The problem with advocating it for low volume logs is that people with high volume log use-cases will eventually also want to use that.

I can't remember who it was but someone proposed to have a global concept of annotations, spanning different Kibana applications. To me, it seems like your proposal goes in a similar direction.

• Should there be a way to select which metadata is included, eg include_process?

For the intake API, why not just leverage the metadata event which is sent once per Intake API request? That way, there is no need to transmit additional/duplicated metadata on annotation events.

[graphaelli]

someone proposed to have a global concept of annotations

Interesting, I wonder if it makes more sense to avoid a specific apm implementation or try to drive that from here then. I'll try to track down more information.

For the intake API, why not just leverage the metadata event which is sent once per Intake API request? That way, there is no need to transmit additional/duplicated metadata on annotation events.

I explained it poorly. The intention is to make it possible to optionally include/exclude process/service/etc info from metadata in the annotation document persisted to elasticsearch, not to specifically add those into the intake for annotations.

Currently, spans include only a subset of that information because it can be tracked down in the linked transaction. Metrics include a full set of metadata just like transactions - when we get to user defined metrics we'll probably run into the idea I'm trying to explore here - allowing events (annotations in this case) to live outside of a process or even a service. We considered this briefly in elastic/apm-server#1901 (comment) but as @simitt pointed out there, there are other complications it introduces with merging vs overwriting.

To make it concrete, consider this intake:

{"annotation": {"message": "something happened"}}

should that persist a document with all metadata - process, service, system? Should the payload be able to prevent inclusion in the document with something like:

{"annotation": {"message": "something happened"}, "service": null}

or maybe

{"annotation": {"message": "something happened"}, "include_service": false}

[ruflin]

Even though this is as far as I understand only for the intake AKA internal format it would be still beneficial if it would not conflict with ECS, especially referring to message.original vs message.

[graphaelli]

@ruflin To be clear, the intake defines agent->apm-server communication, which we intend to keep reasonably compact. The elasticsearch documents generated will continue to adhere to ECS.

[ruflin]

@graphaelli Yes, I'm aware of that. My point was not necessarly to follow ECS but still not conflict with it. I would hope that if long term also internal communication is in ECS where possible.

If it's more of a burden at the moment, please ignore it.

[watson]

If we allow this format to link to a trace/transaction similar to errors, we could use this to support what OpenTracing calls logs

[graphaelli]

@watson's comment turned into some good offline discussions which I'll recap here.

There are two types of annotations that seem useful:

1. trace-specific, interesting events occurring during a trace
2. high level, specific to a service or set of services but not to an individual trace.

transaction.marks currently serve some of the need for 1. They are k/v pairs where the value is the offset from the beginning of the transaction. They are displayed in the UI timeline like domComplete here:

and useful in aggregations over, for example, percentiles over transaction.marks.navigationTiming.domComplete. The marks aren't a good fit for opentracing logs as they can't carry the same volume of information without introducing high cardinality issues.

This proposal initially targeted 2 directly - that is creating a first class object, alongside transaction/span/error/metric, to capture an event for display on visualizations covering an entire service and per-trace views.

Now it's unclear whether it should be expanded to cover 1 and whether this could satisfy opentracing logs. I'm working through options for this effort and whether they can be combined into a single annotation type.

[mikker]

1. I've wanted to create custom marks like the domComplete one in your example. They could provide useful insight to stuff like dynamic page changes in a webapp for example. I wanted to add them to Turbolinks' page changes.
2. These could be deploys for example. Showing up on the Request per minute graph in the Transaction overview.

Just thinking out loud.

[alfkonee]

Please any update on this ??
Would be really helpful to add these anotations in the APM

[felixbarny]

I think there's a lot of overlap with the proposal for a logs endpoint in the APM Server. Logs can be more than just a simple message: they can also have the form of a structured event.

For .NET we already have a library to construct ECS compatible events with a typesafe API: https://github.com/elastic/ecs-dotnet.
This could be used in combination with standard logging APIs so that users can construct custom ECS-compliant events. With APM/log correlation in place, this can suddenly become a custom event in the APM context.

See also #252 (comment)

[graphaelli]

Agreed on the overlap. Closing this one as logs intake is covered in #252, deployment annotations are already shown, and an annotations API has been added to kibana, covering all of the use cases here.