ipt_NETFLOW linux 2.6.x-4.x kernel module by <[email protected]> -- 2008-2018.

   High performance NetFlow v5, v9, IPFIX flow data export module for Linux
   kernel. Created to be useful for linux routers in high-throughput networks.
   It should be used as iptables target.


=========================
= Detailed Feature List =
=========================

   * High performance and scalability. For highest performance module could be
     run without conntrack being enabled in kernel. Reported to be able to
     handle 10Gbit traffic with more than 1500000 pps with negligible server
     load (on S5500BC).

   * NetFlow v5, v9, and IPFIX are fully supported.

     Support of v9/IPFIX is adding flexibility to exporting of flow data
     plus greater visibility of traffic, letting export many additional fields
     besides what was possible in v5 era. Such as
     
   * IPv6 option headers, IPv4 options, TCP options, ethernet type, dot1q
     service and customer VLAN ids, MAC addresses, and

   * Full IPv6 support,

   * NAT translations events (from conntrack) using NetFlow Event Logging (NEL).
     This is standardized way for v9/IPFIXr, but module export such events even
     for v5 collectors via specially crafted pseudo-records.

   * Deterministic (systematic count-based), random and hash Flow Sampling.
     With appropriate differences in support of v5, v9, and IPFIX.

   * SNMP agent (for net-snmp) for remote management and monitoring.

   * Options Templates (v9/IPFIX) let export useful statistical,
     configurational, and informational records to collector.
     Such as metering, exporting, sampling stat and reliability stat, sampling 
     configuration, network devices ifName, ifDescr list.

   * Tested to compile and work out of the box on Centos 5, 6, 7, Debian and
   * Ubuntu. Many vanilla Linux kernels since 2.6.18 up to the latest (as of
   * writing is 3.19) are supported and tested.

   * Module load time and run-time (via sysctl) configuration.

   * Flexibility in enabling features via ./configure script. This will let you
     disable features you don't need, which increase compatibility with custom
     kernels and performance.

   * SNMP-index translation rules, let convert meaningless and unstable
     interface indexes (ifIndex) to more meaningful numbering scheme.

   * Easy support for catching mirrored traffic with promisc option. Which is
     also supporting optional MPLS decapsulation and MPLS-aware NetFlow.


============================
= OBTAINING LATEST VERSION =
============================

   $ git clone git://github.com/aabc/ipt-netflow.git ipt-netflow
   $ cd ipt-netflow


================
= INSTALLATION =
================

   Five easy steps.

** 1. Prepare Kernel source

   If you have package system install kernel-devel package, otherwise install
   raw kernel source from http://kernel.org matching _exactly_ version of your
   installed kernel.

   a) What to do for Centos:

      ~# yum install kernel-devel

   b) What to do for Debian and Ubuntu:

      ~# apt-get install module-assistant
      ~# m-a prepare

   c) Otherwise, if you downloaded raw kernel sources don't forget to create
    .config by copying it from your distribution's kernel. Its copy could reside
    in /boot or sometimes in /proc, examples:

      kernel-src-dir/# cp /boot/config-`uname -r` .config
    or
      kernel-src-dir/# zcat /proc/config.gz > .config

    Assuming you unpacked kernel source into `kernel-src-dir/' directory.
    Then run:

      kernel-src-dir/# make oldconfig

    After that you'll need to prepare kernel for modules build:

      kernel-src-dir/# make prepare modules_prepare

   Note: Don't try to `make prepare' in Centos kernel-devel package directory
     (which is usually something like /usr/src/kernels/2.6.32-431.el6.x86_64)
     as this is wrong and meaningless.

** 2. Prepare Iptables

   Before this step it also would be useful to install pkg-config if don't
   already have.

   If you have package system just install iptables-devel (or on Debian, Ubuntu
   and derivatives libxtables-dev if available, otherwise iptables-dev)
   package, otherwise install iptables source matching version of your
   installation from ftp://ftp.netfilter.org/pub/iptables/

   a) What to do for Centos:

      # yum install iptables-devel

   b) What to do for Debian or Ubuntu:

      # apt-get install iptables-dev pkg-config

   c) Otherwise, for raw iptables source build it and make install.

** 3. Prepare net-snmp (optional)

  In case you want to manage or monitor module performance via SNMP you
  may install net-snmp. If you want to skip this step run configure
  with --disable-snmp-agent option.

  a) For Centos:

      # yum install net-snmp net-snmp-devel

  b) For Debian or Ubuntu:

      # apt-get install snmpd libsnmp-dev

  c) Otherwise install net-snmp from www.net-snmp.org

** 4. Now, to actually build the module run:

      ~/ipt-netflow# ./configure
      ~/ipt-netflow# make all install
      ~/ipt-netflow# depmod

   This will install kernel module and iptables specific library.

   Troubleshooting:

     a) Sometimes you will want to add CC=gcc-3 to make command.
     Example: make CC=gcc-3.3

     b) Compile module with actual kernel source compiled.
     I.e. first compile kernel and boot into it, and then compile module.
     If you are using kernel-devel package check that its version matches
     your kernel package.

     c) If you have sources in non-standard places or configure isn't able to
     find something run ./configure --help to see how to specify paths manually.

     d) To run irqtop on Debian 8 you may need to install:

       # apt-get install ruby ruby-dev ncurses-dev
       # gem install curses

     z) If all fails create ticket at
          https://github.com/aabc/ipt-netflow/issues

** 5. After this point you should be able to load module and
     use -j NETFLOW target in your iptables. See next section.


=====================
= Configure Options =
=====================

   Configure script allows to enable or disable optional features:

     --enable-natevents
         enables natevents (NEL) support, (this and option will require
	 conntrack support to be enabled into kernel and conntack
         module (nf_conntrack) loaded before ipt_NETFLOW. Usually this is
         done automatically because of `depmod', but if you don't do `make
         install' you'll need to load nf_conntrack manually.
         Read below for explanation of natevents.

     --enable-sampler
         enables flow sampler. Read below for explanation of its configuration
	 option.

     --enable-sampler=hash
         additionally enables 'hash' sampler.

     --disable-snmp-agent
         disables building net-snmp agent module, which is enabled by default.

     --enable-snmp-rules
         enables SNMP-index conversion rules. Read below for explanation
         of snmp-rules.

     --enable-macaddress
         enables exporting of src and dst MAC addresses for every flow
         in v9/IPFIX. Difference in any of MAC address will be accounted
         as differnt flow. I.e. MAC addresses will be part of flow key.

     --enable-vlan
         enables exporting of dot1q VLAN Ids and Priorities for every flow
         in v9/IPFIX. It supports outer and second dot1q tags if present.

         Any of two previous options will enable exporting of Ethernet Packet
         Type, ethernetType(256).

     --enable-direction
         enables exporting of flowDirection(61) Element for v9/IPFIX.

         Packets captured in PREROUTING and INPUT chains will be accounted as
         ingress flows(0), in OUTPUT and POSTROUTING as egress flows(1), and
         in FORWARD will have flowDirection set to undefined value 255.

     --enable-aggregation
         enables aggregation rules. Read below for explanation of aggregation.

     --disable-dkms
         disable creating dkms.conf and auto-install module into DKMS tree.

     --disable-dkms-install
         only disable auto-install into DKMS, but still create dkms.conf, in
         case you will want to install it manually.

     --enable-physdev
	 Export ingressPhysicalInterface(252) and egressPhysicalInterface(253)
	 (relevant for bridges) in V9 and IPFIX. If your collector does not
	 support these Elements but you still need physdevs then use
	 --enable-physdev-override, in that case physdevs will override normal
	 interface numbers ingressInterface(10) and egressInterface(14).

     --enable-promisc
	 Enables capturing of promiscuous packets into raw/PREROUTING chain.
	 See README.promisc Solution 1 for usage details and example.

     --promisc-mpls
	 Enables MPLS label stack decapsulation for promiscuous packets. (For
	 IPv4 and IPv6 packets only). This also enables MPLS-aware NetFlow (v9
	 and IPFIX), you may wish to specify with --promisc-mpls=n how much MPLS
	 labels you want to be recorded and exported (default is 3, maximum is
	 10, set to 0 to not report anything).


===========
= RUNNING =
===========

1. You can load module directly by insmod like this:

     # insmod ipt_NETFLOW.ko destination=127.0.0.1:2055 debug=1

   Or if properly installed (make install; depmod) by this:

     # modprobe ipt_NETFLOW destination=127.0.0.1:2055

   See, you may add options in insmod/modprobe command line, or add
   them in /etc/modprobe.conf or /etc/modprobe.d/ipt_NETFLOW.conf
   like thus:

     options ipt_NETFLOW destination=127.0.0.1:2055 protocol=9 natevents=1

2. Statistics is in /proc/net/stat/ipt_netflow
   Machine readable statistics is in /proc/net/stat/ipt_netflow_snmp
   To view boring slab statistics: grep ipt_netflow /proc/slabinfo
   Dump of all flows is in /proc/net/stat/ipt_netflow_flows

3. You can view parameters and control them via sysctl, example:

     # sysctl net.netflow
     # sysctl net.netflow.hashsize=32768

   Note: For after-reboot configuration I recommend to store module parameters
   in modprobe configs instead of storing them in /etc/sysctl.conf, as it's
   less clear when init process will apply sysctl.conf, before of after
   module's load.

4. Example of directing all IPv4 traffic into the module:

     # iptables -I FORWARD -j NETFLOW
     # iptables -I INPUT -j NETFLOW
     # iptables -I OUTPUT -j NETFLOW

   Note: It is preferable (because easier to understand) to _insert_
   NETFLOW target at the top of the chain, otherwise not all traffic may
   reach NETFLOW if your iptables configuration is complicated and some
   other rule inadvertently consume the traffic (dropping or acepting before
   NETFLOW is reached). It's always good to test your configuration.
   Use  iptables -L -nvx  to check pkts/bytes counters on the rules.

5. If you want to account IPv6 traffic you should use protocol 9 or 10.
   Example of directing all IPv6 traffic into the module:

     # sysctl net.netflow.protocol=10
     # ip6tables -I FORWARD -j NETFLOW
     # ip6tables -I INPUT -j NETFLOW
     # ip6tables -I OUTPUT -j NETFLOW

   Note: First enable right version of protocol and after that add ip6tables
     rules, otherwise you will get errors in dmesg.

6. If you want to account NAT events (NEL):

     # sysctl net.netflow.natevents=1

   Note that natevents feature is completely independent from traffic accounting
   (it's using so called conntrack events), thus you don't need to set or change
   any iptables rules to use that. You may need to enable kernel config option
   CONFIG_NF_CONNTRACK_EVENTS though (if it isn't already enabled).
   For details on how they are exported for different protocol versions see
   below.

7. For SNMP support you will need to add this command into snmpd.conf to
   enable IPT-NETFLOW-MIB in SNMP agent:

      dlmod netflow /usr/lib/snmp/dlmod/snmp_NETFLOW.so

   Restart snmpd for changes to take effect. Don't forget to properly configure
   access control. Example simplest configuration may looks like (note that this
   is whole /etc/snmp/snmpd.conf):

      rocommunity public 127.0.0.1
      dlmod netflow /usr/lib/snmp/dlmod/snmp_NETFLOW.so

   Note, that this config will also allow _full_ read-only access to the whole
   linux MIB. To install IPT-NETFLOW-MIB locally, copy file IPT-NETFLOW-MIB.my
   into ~/.snmp/mibs/

   * Detailed example of SNMP configuration is there:
   * https://github.com/aabc/ipt-netflow/wiki/Configuring-SNMP-access

   To check that MIB is installed well you may issue:

     $ snmptranslate -m IPT-NETFLOW-MIB -IR -Tp iptNetflowMIB

   This should output IPT-NETFLOW-MIB in tree form.

   To check that snmp agent is working well issue:

     $ snmpwalk -v 1 -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowMIB

   Should output full MIB. If MIB is not installed try:

     $ snmpget -v 1 -c public 127.0.0.1 .1.3.6.1.4.1.37476.9000.10.1.1.1.1.0

   Which should output STRING: "ipt_NETFLOW".

   MIB provides access to very similar statistics that you have in
   /proc/net/stat/ipt_netflow, you can read description of objects in
   text file IPT-NETFLOW-MIB.my

   If you want to access to SNMP stat in machine readable form for your
   scripts there is file /proc/net/stat/ipt_netflow_snmp

   Note: Using of SNMP v2c or v3 is mandatory for most tables, because
   this MIB uses 64-bit counters (Counter64) which is not supported in old
   SNMP v1. You should understand that 32-bit counter will wrap on 10Gbit
   traffic in just 3.4 seconds! So, always pass option `-v2c' or `-v3'
   to net-snmp utils. Or, for example, configure option `defVersion 2c'
   in ~/.snmp/snmp.conf  You can also have `defCommunity public' ov v3
   auth parameters (defSecurityName, defSecurityLevel, defPassphrase)
   set there (man snmp.conf).

   Examples for dumping typical IPT-NETFLOW-MIB objects:

   - Module info (similar to modinfo, SNMPv1 is ok for following two objects):

     $ snmpwalk -v 1 -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowModule

   - Read-write sysctl-like parameters (yes, they are writable via snmpset, you
     may need to configure write access to snmpd, though):

     $ snmpwalk -v 1 -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowSysctl

   - Global performance stat of the module (note -v2c, because rest of the
     objects require SNMP v2c or SNMP v3):

     $ snmpwalk -v2c -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowTotals

   - Per-CPU (metering) and per-socket (exporting) statistics in table format:

     $ snmptable -v2c -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowCpuTable
     $ snmptable -v2c -c public 127.0.0.1 -m IPT-NETFLOW-MIB iptNetflowSockTable


===========
= OPTIONS =
===========

   Options can be passed as parameters to module or changed dynamically
   via  sysctl net.netflow  or  IPT-NETFLOW-MIB::iptNetflowSysctl

   protocol=5
     - what version of NetFlow protocol to use. Default is 5.
       You can choose from 5, 9, or 10 (where 10 is IPFIX). If you plan
       to account IPv6 traffic you should use protocol 9 or 10 (IPFIX),
       because NetFlow v5 isn't compatible with IPv6.

   destination=127.0.0.1:2055
     - where to export netflow, to this ip address. Port is optional, default
       is 2055. You will see this connection in netstat like this:

       udp 0 0 127.0.0.1:32772 127.0.0.1:2055 ESTABLISHED 

   destination=[2001:db8::1]:2055
     - export target using IPv6 address. Brackets are optional, but otherwise
       you should delimit port with 'p' or '#' character.

   destination=127.0.0.1:2055,192.0.0.1:2055
     - mirror flows to two (can be more) addresses, separate addresses
       with comma.

   destination=127.0.0.1:[email protected]
     - bind socket to address (127.0.0.2).

   destination=127.0.0.1:2055%eth0
     - bind socket to interface (eth0). May be useful for multi-homed boxes.

   sampler=deterministic:123
   sampler=random:123
   sampler=hash:123
     - enables Flow Sampling. To disable set to the empty value or to `0'.
       Note, that this is flow sampling (as of RFC 7014), not packet
       sampling (PSAMP).

       There is three sampling modes:

         deterministic:  select each N-th observed flow; in IPFIX this mode
                         is called Systematic count-based Sampling;
         random:         select randomly one out of N flows.
         hash:           select hash-randomly one out of N flows.

       Number after colon is population size N, with valid values 2-16383.
       (This 16383 limit is for compatibility with NetFlow v5.)
         Using 'deterministic' and 'random' sampling will not reduce resource
       usage caused by the module, because flows are sampled late in exporting
       process. This will reduces amount of flows which go to the collector,
       thus, reducing load on the collector.
         On the other hand, using 'hash' sampling will reduce CPU and memory
       load caused by the module, because flows are discarded early in the
       processing chain. They are discarded almost like in random sampler,
       except that pseudo-random value is depend on the Flow Key hash for each
       packet.
         All required NetFlow/IPFIX information to signal use of sampling is
       also sent to the collector. 'Hash' sampling will be presented as 'random'
       sampling to the collector, because of their similarity.
       Note, that Flow Sampling is compatible with NetFlow v5, v9, and IPFIX.

   natevents=1
     - Collect and send NAT translation events as NetFlow Event Logging (NEL)
       for NetFlow v9/IPFIX, or as dummy flows compatible with NetFlow v5.
       Default is 0 (don't send).

       For NetFlow v5 protocol meaning of fields in dummy flows are such:
         Src IP, Src Port  is Pre-nat source address.
         Dst IP, Dst Port  is Post-nat destination address.
           - These two fields made equal to data flows catched in FORWARD chain.
         Nexthop, Src AS  is Post-nat source address for SNAT. Or,
         Nexthop, Dst AS  is Pre-nat destination address for DNAT.
         TCP Flags is SYN+SCK for start event, RST+FIN for stop event.
         Pkt/Traffic size is 0 (zero), so it won't interfere with accounting.

       Natevents are compilation disabled by default, to enable you will need to
       add --enable-natevents option to ./configure script.

       For techical description of NAT Events see:
         http://tools.ietf.org/html/draft-ietf-behave-ipfix-nat-logging-04

   inactive_timeout=15
     - export flow after it's inactive for 15 seconds. Default value is 15.

   active_timeout=1800
     - export flow after it's active for 1800 seconds (30 minutes). Default
       value is 1800.

   refresh-rate=20
     - for NetFlow v9 and IPFIX it's rate how frequently to re-send templates
       (per packets). You probably don't need to change default (which is 20).

   timeout-rate=30
     - for NetFlow v9 and IPFIX it's rate when to re-send old templates (in
       minutes). No need to change it.

   debug=0
     - debug level (none).

   sndbuf=number
     - size of output socket buffer in bytes. I recommend you to put higher
       value if you experience netflow packet drops (can be seen in statistics
       as 'sock: fail' number.)
       Default value is system default.

   hashsize=number
     - Hash table bucket size. Used for performance tuning.
       Abstractly speaking, it should be minimum two times bigger than flows
       you usually have, but not need to.
       Default is system memory dependent small enough value.

   maxflows=2000000
     - Maximum number of flows to account. It's here to prevent DOS attacks.
       After this limit is reached new flows will not be accounted. Default is
       2000000, zero is unlimited.

   aggregation=string..
     - Few aggregation rules (or some say they are rule.)

       Buffer for aggregation string 1024 bytes, and sysctl limit it
         to ~700 bytes, so don't write there a lot.
       Rules worked in definition order for each packet, so don't
         write them a lot again.
       Rules applied to both directions (dst and src).
       Rules tried until first match, but for netmask and port
          aggregations separately.
       Delimit them with commas.

       Rules are of two kinds: for netmask aggregation
          and port aggregation:

       a) Netmask aggregation example: 192.0.0.0/8=16
       Which mean to strip addresses matching subnet 192.0.0.0/8 to /16.

       b) Port aggregation example: 80-89=80
       Which mean to replace ports from 80 to 89 with 80.

       Full example:
          aggregation=192.0.0.0/8=16,10.0.0.0/8=16,80-89=80,3128=80

       Aggregation rules are enabled by default, if you feel you don't need them
       you may add --disable-aggregation to ./configure script.

   snmp-rules=string...
     - Few SNMP-index conversion rules similar to fproble-ulog.

       Quoting man fprobe-ulog:

         "Comma separated list of interface name to SNMP-index conversion
         rules.  Each rule consists of interface base name and SNMP-index
         base separated by colon (e.g. ppp:200). Final SNMP-index is  sum
         of corresponding SNMP-index base and interface number.
         In the above example SNMP-index of interface ppp11 is 211.

         If interface  name  did not fit to any of conversion rules then
         SNMP-index will be taken from kernel."

       This implementation isn't optimized for performance (no rule caching
       or hashing), but should be fast if rules list are short.

       Rules are parsed in order from first to last until first match.

       snmp-rules are compilation disabled by default, to enable you will need
       to add --enable-snmp option to ./configure script.

   scan-min=1
     - Minimal interval between flow export scans. Sometimes could be useful
       to reduce load on exporting CPU by increasing this interval. Value are
       in kernel jiffies units (which is x/HZ seconds).

   promisc=1
     - Enables promisc hack. See README.promisc Solution 1 for details.

   exportcpu=number
     - Lock exporter to single CPU. This may be useful to fine control CPU
       load. Common use case: with smp_affinity and RSS you spread packet
       processing to all CPUs except one, and lock it to the exporter. While
       exporter CPU load generally is not high, for someone it may be not
       desirable to combine it with packet processing on very highly loaded
       routers.

       This option could be changed at runtime with:

         # echo number > /sys/module/ipt_NETFLOW/parameters/exportcpu

   engine_id=number
     - Observation Domain ID (on IPFIX, Source Id on NetFlow v9, or Engine Id
       on NetFlow v5) value to be exported. This may help your collector to
       distinguish between multiple exporters. On Netflow v9 and IPFIX this
       value is 32-bit on NetFlow v5 only 8 low bits are significant.
       Default value is 0.

       This option could be changed at runtime with:

         # echo number > /sys/module/ipt_NETFLOW/parameters/engine_id


====================
= HOW TO READ STAT =
====================

  Statistics is your friend to fine tune and understand netflow module
  performance.

  To see stat in human readable form:
    # cat /proc/net/stat/ipt_netflow

  How to interpret the data:

> ipt_NETFLOW version v1.8-122-gfae9d59-dirty, srcversion 6141961152BE0DFA6A21EF4; aggr mac vlan

  This line helps to identify actual source that your module is build on.
  Please always supply it in all bug reports.

  v1.8-122: 1.8 is release, 122 is commit number after release;
  -gfae9d59: fae9d59 is short git commit id;
  -dirty: if present, meaning that git detected that sources are changed since
      last git commit, you may wish to do `git diff' to view changes;
  srcversion 6141961152BE0DFA6A21EF4: binary version of module, you can
      compare this with data from `modinfo ./ipt_NETFLOW.ko' to identify
      actual binary loaded;
  aggr mac vlan: tags to identify compile time options that are enabled.

> Protocol version 10 (ipfix), refresh-rate 20, timeout-rate 30, (templates 2, active 2). Timeouts: active 5, inactive 15. Maxflows 2000000

  Protocol version currently in use. Refresh-rate and timeout-rate
      for v9 and IPFIX. Total templates generated and currently active.
  Timeout: active X: how much seconds to wait before exporting active flow.
    - same as sysctl net.netflow.active_timeout variable.
  inactive X: how much seconds to wait before exporting inactive flow.
    - same as sysctl net.netflow.inactive_timeout variable.
  Maxflows 2000000: maxflows limit.
    - all flows above maxflows limit must be dropped.
    - you can control maxflows limit by sysctl net.netflow.maxflows variable.

> Promisc hack is disabled (observed 0 packets, discarded 0).

  observed n: To see that promisc hack is really working.

> Natevents disabled, count start 0, stop 0.

    - Natevents mode disabled or enabled, and how much start or stop events
      are reported.

> Flows: active 5187 (peak 83905 reached 0d0h1m ago), mem 283K, worker delay 100/1000 (37 ms, 0 us, 4:0 0 [3]).

  active X: currently active flows in memory cache.
    - for optimum CPU performance it is recommended to set hash table size to
      at least twice of average of this value, or higher.
  peak X reached Y ago: peak value of active flows.
  mem XK: how much kilobytes of memory currently taken by active flows.
    - one active flow taking 56 bytes of memory.
    - there is system limit on cache size too.
  worker delay X/HZ: how frequently exporter scan flows table per second.
  Rest is boring debug info.

> Hash: size 8192 (mem 32K), metric 1.00, [1.00, 1.00, 1.00]. InHash: 1420 pkt, 364 K, InPDU 28, 6716.

  Hash: size X: current hash size/limit.
    - you can control this by sysctl net.netflow.hashsize variable.
    - increasing this value can significantly reduce CPU load.
    - default value is not optimal for performance.
    - optimal value is twice of average of active flows.
  mem XK: how much memory occupied by hash table.
    - hash table is fixed size by nature, taking 4 bytes per entry.
  metric X, [X, X, X]: how optimal is your hash table being used.
    - lesser value mean more optimal hash table use, min is 1.0.
    - last three numbers in squares is moving average (EWMA) of hash table
      access divided by match rate (searches / matches) for 4sec, and 1, 5, and
      15 minutes. Sort of hash table load average. First value is instantaneous.
      You can try to increase hashsize if averages more than 1 (increase
      certainly if >= 2).
  InHash: X pkt, X K: how much traffic accounted for flows in the hash table.
  InPDU X, X: how much traffic in flows preparing to be exported.

> Rate: 202448 bits/sec, 83 packets/sec; 1 min: 668463 bps, 930 pps; 5 min: 329039 bps, 483 pps

  - Module throughput values for 1 second, 1 minute, and 5 minutes.

> cpu#  pps; <search found new [metric], trunc frag alloc maxflows>, traffic: <pkt, bytes>, drop: <pkt, bytes>
> cpu0  123; 980540  10473 180600 [1.03],    0    0    0    0, traffic: 188765, 14 MB, drop: 27863, 1142 K

  cpu#: this is Total and per CPU statistics for:
  pps: packets per second on this CPU. It's useful to debug load imbalance.
  <search found new, trunc frag alloc maxflows>: internal stat for:
  search found new: hash table searched, found, and not found counters.
  [metric]: one minute (ewma) average hash metric per cpu.
  trunc: how much truncated packets are ignored
    - for example if packets don't have valid IP header.
    - it's also accounted in drop packets counter, but not in drop bytes.
  frag: how much fragmented packets have seen.
    - kernel defragments INPUT/OUTPUT chains for us if nf_defrag_ipv[46]
      module is loaded.
    - these packets are not ignored but not reassembled either, so:
    - if there is no enough data in fragment (ex. tcp ports) it is considered
      to be zero.
  alloc: how much cache memory allocations are failed.
    - packets ignored and accounted in traffic drop stat.
    - probably increase system memory if this ever happen.
  maxflows: how much packets ignored on maxflows (maximum active flows reached).
    - packets ignored and accounted in traffic drop stat.
    - you can control maxflows limit by sysctl net.netflow.maxflows variable.

  traffic: <pkt, bytes>: how much traffic is accounted.
  pkt, bytes: sum of packets/megabytes accounted by module.
    - flows that failed to be exported (on socket error) is accounted here too.

  drop: <pkt, bytes>: how much of traffic is not accounted.
  pkt, bytes: sum of packets/kilobytes that are dropped by metering process.
    - reasons these drops are accounted here:
      truncated/fragmented packets,
      packet is for new flow but failed to allocate memory for it,
      packet is for new flow but maxflows is already reached.
    Traffic lost due to socket errors is not accounted here. Look below
      about export and socket errors.

> Export: Rate 0 bytes/s; Total 2 pkts, 0 MB, 18 flows; Errors 0 pkts; Traffic lost 0 pkts, 0 Kbytes, 0 flows.

  Rate X bytes/s: traffic rate generated by exporter itself.
  Total X pkts, X MB: total amount of traffic generated by exporter.
  X flows: how much data flows are exported.
  Errors X pkts: how much packets not sent due to socket errors.
  Traffic lost 0 pkts, 0 Kbytes, 0 flows: how much metered traffic is lost
    due to socket errors.
  Note that `cberr' errors are not accounted here due to their asynchronous
    nature. Read below about `cberr' errors.

> sock0: 10.0.0.2:2055 unconnected (1 attempts).

  If socket is unconnected (for example if module loaded before interfaces is
  up) it shows now much connection attempts was failed. It will try to connect
  until success.

> sock0: 10.0.0.2:2055, sndbuf 106496, filled 0, peak 106848; err: sndbuf reached 928, connect 0, cberr 0, other 0

  sockX: per destination stats for:
  X.X.X.X:Y: destination ip address and port.
    - controlled by sysctl net.netflow.destination variable.
  sndbuf X: how much data socket can hold in buffers.
    - controlled by sysctl net.netflow.sndbuf variable.
    - if you have packet drops due to sndbuf reached (error -11) increase this
      value.
  filled X: how much data in socket buffers right now.
  peak X: peak value of how much data in socket buffers was.
    - you will be interested to keep it below sndbuf value.
  err: how much packets are dropped due to errors.
    - all flows from them will be accounted in drop stat.
  sndbuf reached X: how much packets dropped due to sndbuf being too small
      (error -11).
  connect X: how much connection attempts was failed.
  cberr X: how much connection refused ICMP errors we got from export target.
    - probably you are not launched collector software on destination,
    - or specified wrong destination address.
    - flows lost in this fashion is not possible to account in drop stat.
    - these are ICMP errors, and would look like this in tcpdump:
      05:04:09.281247 IP alice.19440 > bob.2055: UDP, length 120
      05:04:09.281405 IP bob > alice: ICMP bob udp port 2055 unreachable, length 156
  other X: dropped due to other possible errors.

> aggr0: ...
  aggrX: aggregation rulesets.
    - controlled by sysctl net.netflow.aggregation variable.


==========================
= NetFlow considerations =
==========================

  List of all IPFIX Elements http://www.iana.org/assignments/ipfix/ipfix.xhtml

  Flow Keys are Elements that distinguish flows. Quoting RFC: "If a Flow
  Record for a specific Flow Key value already exists, the Flow Record is
  updated; otherwise, a new Flow Record is created."

  In this implementation following Elements are treated as Flow Keys:

     IPv4 source address:        sourceIPv4Address(8),
     IPv6 source address:        sourceIPv6Address(27),
     IPv4 destination address:   destinationIPv4Address(12),
     IPv6 destination address:   destinationIPv6Address(28),
     TCP/UDP source port:        sourceTransportPort(7),
     TCP/UDP destination port:   destinationTransportPort(11),
     input interface:            ingressInterface(10),
     IP protocol:                protocolIdentifier(4),
     IP TOS:                     ipClassOfService(5),
     and address family (IP or IPv6).

  Additional Flow Keys if VLAN exporting is enabled:

     First (outer) dot1q VLAN tag: dot1qVlanId(243) and
                                 dot1qPriority(244) for IPFIX,
                                 or vlanId(243) for NetFlow v9.
     Second (customer) dot1q VLAN tag: dot1qCustomerVlanId(245)
                                 and dot1qCustomerPriority(246).

  Additional Flow Keys if MAC address exporting is enabled:

     Destination MAC address:    destinationMacAddress(80),
     Source MAC address:         sourceMacAddress(56).

  Additional Flow Keys if MPLS-aware NetFlow is enabled:

     Captured MPLS stack is fully treated as flow key (including TTL values),
     which is Elements from mplsTopLabelStackSection(70) to
     mplsLabelStackSection10(79), and, if present, mplsTopLabelTTL(200).


  Other Elements are not Flow Keys. Note that outer interface, which is
  egressInterface(14), is not regarded as Flow Key. Quoting RFC 7012: "For
  Information Elements ... for which the value may change from packet to packet
  within a single Flow, the exported value of an Information Element is by
  default determined by the first packet observed for the corresponding Flow".

  Note that NetFlow and IPFIX modes of operation may have slightly different
  Elements being used and different statistics sent via Options Templates.


=========
= VOILA =
=========