Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor filebeat elasticsearch module pipelines (audit, server) for maintability #30443

Closed
matschaffer opened this issue Feb 17, 2022 · 3 comments

Comments

@matschaffer
Copy link
Contributor

matschaffer commented Feb 17, 2022

Describe the enhancement:

In the rush to get #30018 merged for 8.0.0 we left the pipelines in https://github.com/elastic/beats/tree/main/filebeat/module/elasticsearch somewhat messy.

As we worked on each fileset we learned more about the problem and refined the approach, but didn't go back to implement the new ideas on already-working filesets.

We should refactor at least the 8.0 pipelines to be:

  1. json
  2. dot_expander/*
  3. whatever additional processing is required (hopefully very little)

The deprecation and slowlog processors already follow this model.

Describe a specific use case for the enhancement or feature:

This should help simplify the pipelines and also make for easier maintenance since the ones that are similar (mostly ECS, minimal processing) will look and work similarly.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/stack-monitoring (Stack monitoring)

@botelastic
Copy link

botelastic bot commented Feb 17, 2023

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Feb 17, 2023
@botelastic botelastic bot closed this as completed Aug 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants