-
Notifications
You must be signed in to change notification settings - Fork 525
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Logs intake endpoint #3723
Comments
@ruflin Can we consider writing to to |
Yes, that is possible. But I wonder if this might become confusing as these are not really APM logs itself (like from apm-server) but logs from a specific service? Should these also go to |
Is the Should we go with |
Filebeat is and will not be using the new indexing strategy. The new indices will exist in 7.9 and will be used by the Agent and all the processes it runs. What is the timeline of the above? It kind of makes sense to migrate everything at the same time but also data migrations are painful. |
The timeline is not set but certainly not before 7.9. We're still in the information gathering step for this issue and it's good to know
The processes it runs are other beats right?
Agreed, that's a big part of why I want to explore this option. The other part is that this proposes to introduce an entirely new intake and I would like to optimize for what users would expect and provides the most value for them, even if it means managing a non
Two options I had in mind here:
There's nothing
Also, I'm not clear on whether it's compliant with datastreams to have an optional namespace. If required, an absent |
What's the correlation between the dataset, which is part of the index name, and the ECS field I don't want to go too much off-track here but the thin line between |
@felixbarny The fields this is based on are To not go off track, will not comment on logs vs events :-) |
Thanks for all of the details @ruflin .
Can you link to an issue or doc once there is one? We'd be happy to participate in a discussion as well.
The should be at least compatible (fully ECS) if not identical- we enforce the format via intake. @felixbarny please correct me if I missed something in elastic/apm#252
The initial proposal doesn't have special UI needs for consumption, they will be used like any other application logs.
That's possible but it would help to understand why this is relevant to decide if its significant. |
That's correct. But note that users can add their own custom fields to log events (see what I did there?).
Yep, mainly through the Logs App.
It'd be great if the user could customize the |
|
We're changing tack on logging a bit, and this will no longer be required. If that changes, let's reopen. |
To enable agents to ship logs, the APM Server should implement a dedicated logs intake endpoint.
See this PR for a specification of the endpoint: elastic/apm#252
Notes on metadata:
The intake metadata event is not ECS compliant. Therefore, it has to be converted, just as we already do it in the event intake API.
Also, it contains a lot of information that we probably don't want to add to every log event, for example
process.argv
Should the agents make sure to omit those fields or should the server do that?
Index
Which index should the APM Server write to? The Logs App should include that index by default.
There should also be an ILM setup for this index.
De-dotting
When I said the logs lines are all ECS compatible, that's actually not entirely true. The log events use dotted property names as an equivalent for nesting. For example:
log.level
. The main reasons are readability and easier and more performant JSON serialization in the loggers.I've created an issue for a de-dotting filter in beats a while ago: elastic/beats#17021. This filter would be useful to apply within the APM Server logs endpoint processing.
I think that de-dotting is not required to get started. But it does mean that someone who wants to process log events in an ingest node pipeline, for example, will face issues.
The text was updated successfully, but these errors were encountered: