Remove all fields other than a defined list

[markwalkom]

This came up in a forum thread - https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275

The idea being that if you want to remove a bunch of fields, that you may not necessarily know the names of (or don't care), having a config option for the remove processor that you could simply define a set of fields to keep, thereyby removing everything else, would be a simple solution.

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[jakelandis]

Sounds like a good enhancement request. However, we probably don't want to actually use reg-ex's here. Either a new keep processor (which is the opposite of remove) or a new field all_except_fields (mutually exclusive with fields) would likely be the best implementation.

[harshlancer]

If this issue is beginner-friendly I would like to start my open source journey with this issue.
Please tell me if I can start working on this...

[danhermann]

@harshlancer, you are welcome to start working on this if you would like. I would suggest adding a new option to the existing remove processor rather than creating a whole new processor to perform this function.

[harshlancer]

Sorry I don't think this issue is for a complete newbie....

[zembrzuski]

Hi @danhermann
Since @harshlancer is not going to work on this issue, I will give it a try on this issue.

[zembrzuski]

Just to keep you guys posted: I am already working on the implementation of this issue, and I expect to raise a PR in 2 or 3 days.
Tks!

[zembrzuski]

Hi guys!
I've created a PR to try to address this issue:
Please let me know if I am on track to get it done :)

[danhermann]

Closed by #83665

[zez3]

Wouldn't make more sense to have a correct parser whatever Integration in the first place and not waste those computational resources on ingesting first wrong and delete after?