Missing NetFlow Option packets

[SirBreadc]

Relevant telegraf.conf

[global_tags]
  VM = "${HOSTNAME}"

[agent]
  debug = false
  quiet = true
  metric_batch_size = 5000

  metric_buffer_limit = 2000000

  flush_interval = "1s"
 
  flush_jitter = "5s"

  precision = "0s"



  hostname = "${HOST_HOSTNAME}"
  omit_hostname = false

# Netflow v5, Netflow v9 and IPFIX collector
[[inputs.netflow]]

  service_address = "udp4://:2055"
  

  protocol = "ipfix"


  private_enterprise_number_files = ["conf/custom_fields.csv"]

  dump_packets = false

# # Configuration for sending metrics to InfluxDB 2.0
[[outputs.influxdb_v2]]
   
   urls = ["host1","host2","host3"]

   token = "{INFLUXADMINTOKEN}"

   organization = "org"
   namepass = ["netflow"]

   bucket = "${BUCKET_NAME}"
   content_encoding = "gzip"
   timeout = "15s"

    insecure_skip_verify = true

Logs from Telegraf

See attached file

System info

Telegraf 1.30, Centos,CentOS Linux 7

Docker

No response

Steps to reproduce

1. Send Netflow from device.

Expected behavior

Option packets are also writen to output

Actual behavior

Option packets are not written to outputs

Additional info

No response

[SirBreadc]

For My netflowV9 and IPFIX I am sending snmp option packets with interface snmp mib to long/short name mappings. Looks like telegarf is ignoring these option packets. See IOS XE configuration below along with exporter templates details for the Option options interface-table.

IOS XE exporter configurations:

flow exporter Telegraf
 destination <ip>
 source Loopback0
 transport udp 2055
 option interface-table

# show flow exporter templates details 
Flow Exporter Telegraf:
  Client: Option options interface-table
  Exporter Format: NetFlow Version 9
  Template ID    : 256
  Source ID      : 6
  Record Size    : 110
  Template layout
  _________________________________________________________________________________________
  |                           Field                             |  Type | Offset |  Size  |
  -----------------------------------------------------------------------------------------
  | v9-scope system                                             |     1 |     0  |     4  |
  | interface input snmp                                        |    10 |     4  |     4  |
  | interface name short                                        |    82 |     8  |    33  |
  | interface name long                                         |    83 |    41  |    65  |
  | interface output snmp                                       |    14 |   106  |     4  |
  -----------------------------------------------------------------------------------------

  Client: Flow Monitor FLOW-MONITOR-IPV4-v2
  Exporter Format: NetFlow Version 9
  Template ID    : 256
  Source ID      : 256
  Record Size    : 52
  Template layout
  _________________________________________________________________________________________
  |                           Field                             |  Type | Offset |  Size  |
  -----------------------------------------------------------------------------------------
  | ip fragmentation id                                         |    54 |     0  |     2  |
  | ipv4 source address                                         |     8 |     2  |     4  |
  | ipv4 destination address                                    |    12 |     6  |     4  |
  | ip tos                                                      |     5 |    10  |     1  |
  | ip dscp                                                     |   195 |    11  |     1  |
  | ip protocol                                                 |     4 |    12  |     1  |
  | transport source-port                                       |     7 |    13  |     2  |
  | transport destination-port                                  |    11 |    15  |     2  |
  | transport tcp flags                                         |     6 |    17  |     1  |
  | interface input snmp                                        |    10 |    18  |     4  |
  | application id                                              |    95 |    22  |     4  |
  | routing next-hop address ipv4                               |    15 |    26  |     4  |
  | interface output snmp                                       |    14 |    30  |     4  |
  | flow direction                                              |    61 |    34  |     1  |
  | flow sampler                                                |    48 |    35  |     1  |
  | counter bytes                                               |     1 |    36  |     4  |
  | counter packets                                             |     2 |    40  |     4  |
  | timestamp sys-uptime first                                  |    22 |    44  |     4  |
  | timestamp sys-uptime last                                   |    21 |    48  |     4  |
  -----------------------------------------------------------------------------------------

  Client: Flow Monitor FLOW-MONITOR-IPV6-v2
  Exporter Format: NetFlow Version 9
  Template ID    : 257
  Source ID      : 512
  Record Size    : 85
  Template layout
  _________________________________________________________________________________________
  |                           Field                             |  Type | Offset |  Size  |
  -----------------------------------------------------------------------------------------
  | ipv6 source address                                         |    27 |     0  |    16  |
  | ipv6 destination address                                    |    28 |    16  |    16  |
  | ip dscp                                                     |   195 |    32  |     1  |
  | ip protocol                                                 |     4 |    33  |     1  |
  | transport source-port                                       |     7 |    34  |     2  |
  | transport destination-port                                  |    11 |    36  |     2  |
  | transport tcp flags                                         |     6 |    38  |     1  |
  | interface input snmp                                        |    10 |    39  |     4  |
  | application id                                              |    95 |    43  |     4  |
  | routing next-hop address ipv6                               |    62 |    47  |    16  |
  | interface output snmp                                       |    14 |    63  |     4  |
  | flow direction                                              |    61 |    67  |     1  |
  | flow sampler                                                |    48 |    68  |     1  |
  | counter bytes                                               |     1 |    69  |     4  |
  | counter packets                                             |     2 |    73  |     4  |
  | timestamp sys-uptime first                                  |    22 |    77  |     4  |
  | timestamp sys-uptime last                                   |    21 |    81  |     4  |
  -----------------------------------------------------------------------------------------

Should this work out of the box? or is this a new feature request?

[SirBreadc]

Telegraf logs
log.log

[srebhan]

@SirBreadc currently option data-flow-sets are not processed by the plugin. Could you please provide some samples of the data using the dump_packets = true setting and Telegraf's debug mode?

[SirBreadc]

log_telegarf_snmp_options_table.zip
@srebhan Sorry for the delay here is the dump file you request with one device enabled sending the snmp table option:

` Exporter Format: NetFlow Version 9
Template ID : 256
Source ID : 6
Record Size : 110
Template layout

---

| Field | Type | Offset | Size |

| v9-scope system | 1 | 0 | 4 |
| interface input snmp | 10 | 4 | 4 |
| interface name short | 82 | 8 | 33 |
| interface name long | 83 | 41 | 65 |
| interface output snmp | 14 | 106 | 4 |
-----------------------------------------------------------------------------------------`

[srebhan]

@SirBreadc I've added Netflow v9 options support in PR #15180, available as soon as CI finished the builds. Please give it a try and let me know if this what you expect. The options are reported as netflow_options metrics.

Just for clarification, the "missing template" warnings are an inherent problem of the netflow protocol. The warning will disappear once the device will resend the templates. This cannot be triggered by Telegraf and without the templates the data cannot be interpreted... This mostly happens if you start/restart after the device established a connection to Telegraf...

[SirBreadc]

@srebhan Thanks that works well, can this also be added for IPFIX too? as I am only seeing the netflow_option packet for our V9 devices. Will get you a dump for one of those devices if that's needed

[srebhan]

Yeah I would need another dump from those devices and another feature-request if possible...