-
Notifications
You must be signed in to change notification settings - Fork 419
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ECS Ingest processor #181
Comments
It just occurred to me that you proposed this on the ECS repo, not Beats. So you're thinking of reference implementations (more or less), correct? They can be used for Beats, but they would be published here independent of Beats implementation details, and useable by anyone with a need for them? I love the idea. I think we should experiment concretely with them on Beats first, before building too much "infrastructure" to support this here, however. WDYT? |
The idea came definitively from the duplicated code we have in the ingest pipelines of Beats but it was meant to be more general. I don't think it's something we should do right now but I need a place to write down the idea and get other thoughts on it. |
Here's some ideas that have been occurring to me while working on the Beats migrations. They're broken up in very small ideas, but we could likely join a few together to create these reference pipelines:
|
Other ideas
|
|
cc @andrewkroh |
|
|
|
|
|
|
|
With ECS we know the exact structure of some fields. Based on this some common processing happens. A few examples:
source.ip
-> geoip processor to enrich with geo informationuser_agent.original
-> user_agent enrichementSince Elasticsearch 6.5 it is possible to have a pipeline that calls an other pipeline: https://www.elastic.co/guide/en/elasticsearch/reference/6.5/pipeline-processor.html We could provide an ECS pipeline that does all this default processing. All that users would have to do is adding it to their ingest pipeline.
Over time we could add more processing in such a pipeline. For example if we have a convention that all
http.request.method
should be upper case the Uppercase Processor could be applied to this field and similar things.The text was updated successfully, but these errors were encountered: