Add sampling to netflow and sflow

[jsirianni]

We want to give the user the option of filtering / sampling their netflow / sflow logs. The data can be redundant, therefore it is not necessary to handle every entry.

Default behavior excludes sampling. When enabled, the user has their choice of 0-100% in 10% increments. A sample rate of 0% will filter nothing, and a sample rate of 90% will filter most entries.

In testing, a sample rate of 0.5 produced has as many logs, while a sample rate of 0.0 produced 100% of logs. Testing for short durations is hit and miss but 3 minutes or longer has nice results.

You can test netflow v5 like this
config.yaml (without sample rate)

pipeline:
- type: netflow
  netflow_version: v5
  listen_address: 0.0.0.0:2056
- type: stdout

wc out, 528, 30 seconds
wc out, 4648, 3 minutes

config.yaml (with sample rate)

pipeline:
- type: netflow
  netflow_version: v5
  listen_address: 0.0.0.0:2056
  sampling_enable: true
  sampling_drop_rate: "0.5"
- type: file_output
  path: out.sampled

wc out.sampled, 418, 30 seconds
wc out.sampled, 2163, 3 minutes

Start Stanza

./stanza -c ./config.yaml --plugin_dir ./plugins

Run a loadgen container. 10.99.1.24 is my ip address, you need to determine your ip address. I believe it will not work with localhost / 127.0.0.1

docker run -it --rm networkstatic/nflow-generator -t 10.99.1.23 -p 2056

[BinaryFissionGames]

Few comments about some of the more meta-data related stuff!