Generic filtering Implementation #830
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
monicasarbu
changed the title
monicasarbu
force-pushed
the
generic_filtering
branch
from
6f8fc1c to
dd01b2b
Compare
monicasarbu
changed the title
monicasarbu
force-pushed
the
generic_filtering
branch
2 times, most recently
from
672c482 to
4115a5c
Compare
| keyPart := keyParts[i] | ||
|
|
||
| if _, ok := mapp[keyPart]; ok { | ||
| mapp = mapp[keyPart].(MapStr) |
monicasarbu
force-pushed
the
generic_filtering
branch
5 times, most recently
from
4fa7a64 to
2d3b680
Compare
| filters, err := filter.New(b.Config.Filter) | ||
| if err != nil { | ||
| fmt.Printf("Error Initialising filters: %v\n", err) | ||
| logp.Critical(err.Error()) |
There was a problem hiding this comment.
I noticed we spell "initializing" two different ways in messages in this file. We should stop choose one way and change them all.
monicasarbu
force-pushed
the
generic_filtering
branch
3 times, most recently
from
1b95eca to
9528839
Compare
Closed
monicasarbu
force-pushed
the
generic_filtering
branch
from
be22cfe to
8b6f82f
Compare
| return nil | ||
| } | ||
|
|
||
| func (m MapStr) Clone() MapStr { |
There was a problem hiding this comment.
Maybe replace it with copystructure?
Both actions receive fields as argument that can be of type map or a value. For example: "proc", "proc.cpu.total_p". Each event MUST contain a few fields that cannot be removed. For now, these "mandatory" fields are: ["@timestamp", "beat", type", "count"]. If you configure one of these fields in the drop_fields, it will be ignored. In case there is a field to include that is not available in the event, then just ignore the field. If none of the fields to include are part of the event, then drop the event as it contains no information except the "mandatory" fields. When multiple filtering rules are defined, we start with a copy of the event and apply the filtering rules one by to the event. event -> filter rule 1 -> event 1 -> filter rule 2 -> event 2 ... where event is the initial event, event1 is the event resulted after applying "filter rule1" and it's considered input for the "filter rule 2" and so on.
The event checker verifies if no field has the struct or pointer type and it's called by the publisher just before publishing the event.
Check if the event (of type MapStr) passed to the publisher contains the expected types. If not, then convert the values by calling json.marshall & json.unmarshall.
monicasarbu
force-pushed
the
generic_filtering
branch
from
0529fe8 to
f8ecee6
Compare
|
Closing this PR in favor of #1120. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The proposal for the generic filtering can be found here: #451
In this implementation two actions are implemented:
and both accept
fieldsas argument:- include_fields: fields: ["proc.mem", "proc.cpu", "cmdline"]Each event MUST contain a few fields that cannot be removed. For now, these "mandatory" fields are: ["@timestamp", "beat", type", "count"]. If you configure one of these fields in the
drop_fields, it will be ignored.A field can be a dictionary or a value. Example: `proc.cpu.total_p", "proc", "proc.cpu".
Special cases to consider:
- include_fields: fields: ["xx", "proc.mem", "proc.cpu"]type != processwill be dropped as they don't contain any of the fields:proc.mem,proc.cpu.event -> filter rule 1 -> event 1 -> filter rule 2 -> event 2 ...
- include_fields: fields: ["proc"] - include_fields: fields: ["proc.cpu.total_p", "proc.mem.rss_p", "proc.cmdline"]Testing: