[Filebeat] NetFlow input support for custom field definitions #10945
Conversation
adriansr
added
enhancement
review
Filebeat
|
Pinging @elastic/secops |
| return nil | ||
| } | ||
|
|
||
| var emptyDefinitionErr = errors.New("custom fields definition is empty") |
There was a problem hiding this comment.
error var emptyDefinitionErr should have name of the form errFoo
| @@ -92,7 +94,15 @@ func (d DecoderV9) ReadFieldDefinition(buf *bytes.Buffer) (field fields.Key, len | |||
| return field, length, nil | |||
| } | |||
|
|
|||
| func (d DecoderV9) GetFields() fields.FieldDict { | |||
There was a problem hiding this comment.
exported method DecoderV9.GetFields should have comment or be unexported
| @@ -20,12 +20,20 @@ type Field struct { | |||
|
|
|||
| type FieldDict map[Key]*Field | |||
|
|
|||
| func RegisterFields(dict FieldDict) error { | |||
| func RegisterGlobalFields(dict FieldDict) error { | |||
There was a problem hiding this comment.
exported function RegisterGlobalFields should have comment or be unexported
| @@ -6,7 +6,7 @@ package fields | |||
|
|
|||
| import "fmt" | |||
|
|
|||
| var Fields = FieldDict{} | |||
| var GlobalFields = FieldDict{} | |||
There was a problem hiding this comment.
exported var GlobalFields should have comment or be unexported
Added parsing for Logstash's Custom fields definitions. They allow to extend the NetFlow/IPFIX collector supported fields.
Extend NetFlow collector library to configure custom fields for NetFlow V9 / IPFIX.
Added a new configuration option, `custom_definitions`, allows a list of paths to YAML field definitions in Logstash codec-netflow format.
adriansr
force-pushed
the
feature_netflow_field_defs
branch
from
0d0577c
to
722ec4b
Compare
| } | ||
|
|
||
| // LoadFieldDefinitionsFromFile takes the path to a YAML file in Logstash | ||
| // Netflow or IPFIX custom fields format and converts it to a FieldDict. |
There was a problem hiding this comment.
We'll need a description of the config format in the Filebeat docs.
|
|
||
| Where `id` is the numeric field ID. | ||
|
|
||
| The IPFIX format similar, but grouped by Private Enterprise NUmber (PEN): |
There was a problem hiding this comment.
Nit: I assume "Number" should be the capitalization here? ;-)
| t.FailNow() | ||
| } | ||
|
|
||
| } |
There was a problem hiding this comment.
Perhaps have a test demonstrating the direction of the overwrite, when a key is present in multiple sources? None of the fields in the test above conflict.
There was a problem hiding this comment.
There's a conflict with Key{3, 4}, which is overwritten by c. Is that what you mean?
| @@ -148,7 +154,7 @@ func readDatTests(t testing.TB) *DatTests { | |||
| return &tests | |||
| } | |||
|
|
|||
| func getFlowsFromDat(t testing.TB, name string, datFiles ...string) TestResult { | |||
| func getFlowsFromDat(t testing.TB, name string, testData DatTest) TestResult { | |||
There was a problem hiding this comment.
Nit: This mixed naming in var/funcs named dat vs data is confusing. Both are right, but I think if you pick one or the other across the board, things will be more clear. I think "data" better describes it.
…c#10945) This PR adds support for loading custom (enterprise-specific) fields to the Filebeat NetFlow input. These fields can extend and/or override fields in NetFlow V9 and IPFIX. For compatibility, the feature uses the same field definition YAML format as Logstash's netflow codec plugin. A new configuration option custom_definitions consists of a list of paths to definition files. (cherry picked from commit cd49078)
adriansr
added
v7.0.0
and removed
needs_backport
…m field definitions (#11223) [Filebeat] NetFlow input support for custom field definitions (#10945) This PR adds support for loading custom (enterprise-specific) fields to the Filebeat NetFlow input. These fields can extend and/or override fields in NetFlow V9 and IPFIX. For compatibility, the feature uses the same field definition YAML format as Logstash's netflow codec plugin. A new configuration option custom_definitions consists of a list of paths to definition files. (cherry picked from commit cd49078)
This PR adds support for loading custom (enterprise-specific) fields to the Filebeat NetFlow input.
These fields can extend and/or override fields in NetFlow V9 and IPFIX.
For compatibility, the feature uses the same field definition YAML format as Logstash's netflow codec plugin.
A new configuration option
custom_definitionsconsists of a list of paths to definition files.