Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(inputs.netflow): Add sFlow decoder #13047

Merged
merged 8 commits into from
Jun 1, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/LICENSE_OF_DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ following works:
- github.com/google/go-github [BSD 3-Clause "New" or "Revised" License](https://github.com/google/go-github/blob/master/LICENSE)
- github.com/google/go-querystring [BSD 3-Clause "New" or "Revised" License](https://github.com/google/go-querystring/blob/master/LICENSE)
- github.com/google/gofuzz [Apache License 2.0](https://github.com/google/gofuzz/blob/master/LICENSE)
- github.com/google/gopacket [BSD 3-Clause "New" or "Revised" License](https://github.com/google/gopacket/blob/master/LICENSE)
- github.com/google/s2a-go [Apache License 2.0](https://github.com/google/s2a-go/blob/main/LICENSE.md)
- github.com/google/uuid [BSD 3-Clause "New" or "Revised" License](https://github.com/google/uuid/blob/master/LICENSE)
- github.com/googleapis/enterprise-certificate-proxy [Apache License 2.0](https://github.com/googleapis/enterprise-certificate-proxy/blob/main/LICENSE)
Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ require (
github.com/google/gnxi v0.0.0-20221016143401-2aeceb5a2901
github.com/google/go-cmp v0.5.9
github.com/google/go-github/v32 v32.1.0
github.com/google/gopacket v1.1.19
powersj marked this conversation as resolved.
Show resolved Hide resolved
github.com/google/licensecheck v0.3.1
github.com/google/uuid v1.3.0
github.com/gopcua/opcua v0.3.7
Expand Down
2 changes: 2 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -724,6 +724,8 @@ github.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gopacket v1.1.19 h1:ves8RnFZPGiFnTS0uPQStjwru6uO6h+nlr9j6fL7kF8=
github.com/google/gopacket v1.1.19/go.mod h1:iJ8V8n6KS+z2U1A8pUwu8bW5SyEMkXJB8Yo/Vo+TKTo=
github.com/google/licensecheck v0.3.1 h1:QoxgoDkaeC4nFrtGN1jV7IPmDCHFNIVh54e5hSt6sPs=
github.com/google/licensecheck v0.3.1/go.mod h1:ORkR35t/JjW+emNKtfJDII0zlciG9JgbT7SmsohlHmY=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
Expand Down
26 changes: 18 additions & 8 deletions plugins/inputs/netflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
```toml @sample.conf
# Netflow v5, Netflow v9 and IPFIX collector
[[inputs.netflow]]
## Address to listen for netflow/ipfix packets.
## Address to listen for netflow,ipfix or sflow packets.
## example: service_address = "udp://:2055"
## service_address = "udp4://:2055"
## service_address = "udp6://:2055"
Expand All @@ -53,9 +53,10 @@ See the [CONFIGURATION.md][CONFIGURATION.md] for more details.

## Protocol version to use for decoding.
## Available options are
## "ipfix" -- IPFIX / Netflow v10 protocol (also works for Netflow v9)
## "netflow v5" -- Netflow v5 protocol
## "netflow v9" -- Netflow v9 protocol (also works for IPFIX)
## "ipfix" -- IPFIX / Netflow v10 protocol (also works for Netflow v9)
## "sflow v5" -- sFlow v5 protocol
# protocol = "ipfix"

## Dump incoming packets to the log
Expand Down Expand Up @@ -92,6 +93,16 @@ following information
The specific fields vary for the different protocol versions, here are some
examples

### IPFIX

```text
netflow,source=127.0.0.1,version=IPFIX protocol="tcp",vlan_src=0u,src_tos="0x00",flow_end_ms=1666345513807u,src="192.168.119.100",dst="44.233.90.52",src_port=51008u,total_bytes_exported=0u,flow_end_reason="end of flow",flow_start_ms=1666345513807u,in_total_bytes=52u,in_total_packets=1u,dst_port=443u
netflow,source=127.0.0.1,version=IPFIX src_tos="0x00",src_port=54330u,rev_total_bytes_exported=0u,last_switched=9u,vlan_src=0u,flow_start_ms=1666345513807u,in_total_packets=1u,flow_end_reason="end of flow",flow_end_ms=1666345513816u,in_total_bytes=40u,dst_port=443u,src="192.168.119.100",dst="104.17.240.92",total_bytes_exported=0u,protocol="tcp"
netflow,source=127.0.0.1,version=IPFIX flow_start_ms=1666345513807u,flow_end_ms=1666345513977u,src="192.168.119.100",dst_port=443u,total_bytes_exported=0u,last_switched=170u,src_tos="0x00",in_total_bytes=40u,dst="44.233.90.52",src_port=51024u,protocol="tcp",flow_end_reason="end of flow",in_total_packets=1u,rev_total_bytes_exported=0u,vlan_src=0u
netflow,source=127.0.0.1,version=IPFIX src_port=58246u,total_bytes_exported=1u,flow_start_ms=1666345513806u,flow_end_ms=1666345513806u,in_total_bytes=156u,src="192.168.119.100",rev_total_bytes_exported=0u,last_switched=0u,flow_end_reason="forced end",dst="192.168.119.17",dst_port=53u,protocol="udp",in_total_packets=2u,vlan_src=0u,src_tos="0x00"
netflow,source=127.0.0.1,version=IPFIX protocol="udp",vlan_src=0u,src_port=58879u,dst_port=53u,flow_end_ms=1666345513832u,src_tos="0x00",src="192.168.119.100",total_bytes_exported=1u,rev_total_bytes_exported=0u,flow_end_reason="forced end",last_switched=33u,in_total_bytes=221u,in_total_packets=2u,flow_start_ms=1666345513799u,dst="192.168.119.17"
```

### Netflow v5

```text
Expand All @@ -118,12 +129,11 @@ netflow,source=127.0.0.1,version=NetFlowV9 protocol="tcp",src="192.168.119.100",
netflow,source=127.0.0.1,version=NetFlowV9 protocol="tcp",src="192.168.119.100",src_port=49398u,dst="140.82.114.26",dst_port=443u,in_bytes=697u,in_packets=4u,flow_start_ms=1666350481030u,flow_end_ms=1666350481362u,tcp_flags="...PA...",engine_type="17",engine_id="0x01",icmp_type=0u,icmp_code=0u,fwd_status="unknown",fwd_reason="unknown",src_tos="0x00"
```

### IPFIX
### sFlow v5

```text
netflow,source=127.0.0.1,version=IPFIX protocol="tcp",vlan_src=0u,src_tos="0x00",flow_end_ms=1666345513807u,src="192.168.119.100",dst="44.233.90.52",src_port=51008u,total_bytes_exported=0u,flow_end_reason="end of flow",flow_start_ms=1666345513807u,in_total_bytes=52u,in_total_packets=1u,dst_port=443u
netflow,source=127.0.0.1,version=IPFIX src_tos="0x00",src_port=54330u,rev_total_bytes_exported=0u,last_switched=9u,vlan_src=0u,flow_start_ms=1666345513807u,in_total_packets=1u,flow_end_reason="end of flow",flow_end_ms=1666345513816u,in_total_bytes=40u,dst_port=443u,src="192.168.119.100",dst="104.17.240.92",total_bytes_exported=0u,protocol="tcp"
netflow,source=127.0.0.1,version=IPFIX flow_start_ms=1666345513807u,flow_end_ms=1666345513977u,src="192.168.119.100",dst_port=443u,total_bytes_exported=0u,last_switched=170u,src_tos="0x00",in_total_bytes=40u,dst="44.233.90.52",src_port=51024u,protocol="tcp",flow_end_reason="end of flow",in_total_packets=1u,rev_total_bytes_exported=0u,vlan_src=0u
netflow,source=127.0.0.1,version=IPFIX src_port=58246u,total_bytes_exported=1u,flow_start_ms=1666345513806u,flow_end_ms=1666345513806u,in_total_bytes=156u,src="192.168.119.100",rev_total_bytes_exported=0u,last_switched=0u,flow_end_reason="forced end",dst="192.168.119.17",dst_port=53u,protocol="udp",in_total_packets=2u,vlan_src=0u,src_tos="0x00"
netflow,source=127.0.0.1,version=IPFIX protocol="udp",vlan_src=0u,src_port=58879u,dst_port=53u,flow_end_ms=1666345513832u,src_tos="0x00",src="192.168.119.100",total_bytes_exported=1u,rev_total_bytes_exported=0u,flow_end_reason="forced end",last_switched=33u,in_total_bytes=221u,in_total_packets=2u,flow_start_ms=1666345513799u,dst="192.168.119.17"
netflow,source=127.0.0.1,version=sFlowV5 out_errors=0i,out_bytes=3946i,status="up",in_unknown_protocol=4294967295i,out_unicast_packets_total=29i,agent_subid=100000i,interface_type=6i,in_unicast_packets_total=28i,out_dropped_packets=0i,in_bytes=3910i,in_broadcast_packets_total=4294967295i,ip_version="IPv4",agent_ip="192.168.119.184",in_snmp=3i,in_errors=0i,promiscuous=0i,interface=3i,in_mcast_packets_total=4294967295i,in_dropped_packets=0i,sys_uptime=12414i,seq_number=2i,speed=1000000000i,out_mcast_packets_total=4294967295i,out_broadcast_packets_total=4294967295i 12414000000
netflow,source=127.0.0.1,version=sFlowV5 sys_uptime=17214i,agent_ip="192.168.119.184",agent_subid=100000i,seq_number=2i,in_phy_interface=1i,ip_version="IPv4" 17214000000
netflow,source=127.0.0.1,version=sFlowV5 in_errors=0i,out_unicast_packets_total=36i,interface=3i,in_broadcast_packets_total=4294967295i,ip_version="IPv4",speed=1000000000i,out_bytes=4408i,out_mcast_packets_total=4294967295i,status="up",in_snmp=3i,in_mcast_packets_total=4294967295i,out_broadcast_packets_total=4294967295i,promiscuous=0i,in_bytes=5568i,out_dropped_packets=0i,sys_uptime=22014i,agent_subid=100000i,in_unknown_protocol=4294967295i,interface_type=6i,in_dropped_packets=0i,in_unicast_packets_total=37i,out_errors=0i,agent_ip="192.168.119.184",seq_number=3i 22014000000

```
7 changes: 5 additions & 2 deletions plugins/inputs/netflow/netflow.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,10 @@ func (n *NetFlow) Init() error {
n.decoder = &netflowDecoder{Log: n.Log}
case "netflow v5":
n.decoder = &netflowv5Decoder{}
case "sflow", "sflow v5":
n.decoder = &sflowv5Decoder{Log: n.Log}
default:
return fmt.Errorf("invalid protocol %q, only supports 'netflow v5', 'netflow v9' and 'ipfix'", n.Protocol)
return fmt.Errorf("invalid protocol %q, only supports 'sflow', 'netflow v5', 'netflow v9' and 'ipfix'", n.Protocol)
}
return n.decoder.Init()
}
Expand Down Expand Up @@ -123,7 +125,8 @@ func (n *NetFlow) read(acc telegraf.Accumulator) {
}
metrics, err := n.decoder.Decode(src.IP, buf[:count])
if err != nil {
acc.AddError(err)
errWithData := fmt.Errorf("%w; raw data: %s", err, hex.EncodeToString(buf[:count]))
acc.AddError(errWithData)
continue
}
for _, m := range metrics {
Expand Down
10 changes: 5 additions & 5 deletions plugins/inputs/netflow/netflow_decoder.go
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ var fieldMappingsIPFIX = map[uint16][]fieldMapping{
194: {{"mpls_payload_len", decodeUint}}, // mplsPayloadLength
195: {{"dscp", decodeUint}}, // ipDiffServCodePoint
196: {{"precedence", decodeUint}}, // ipPrecedence
197: {{"fragement_flags", decodeFragmentFlags}}, // fragmentFlags
197: {{"fragment_flags", decodeFragmentFlags}}, // fragmentFlags
198: {{"bytes_sqr_sum", decodeUint}}, // octetDeltaSumOfSquares
199: {{"bytes_sqr_sum_total", decodeUint}}, // octetTotalSumOfSquares
200: {{"mpls_top_label_ttl", decodeUint}}, // mplsTopLabelTTL
Expand All @@ -256,10 +256,10 @@ var fieldMappingsIPFIX = map[uint16][]fieldMapping{
211: {{"collector", decodeIP}}, // collectorIPv4Address
212: {{"collector", decodeIP}}, // collectorIPv6Address
213: {{"export_interface", decodeUint}}, // exportInterface
214: {{"export_proto_version", decodeUint}}, //exportProtocolVersion
215: {{"export_transport_proto", decodeUint}}, //exportTransportProtocol
216: {{"collector_transport_port", decodeUint}}, //collectorTransportPort
217: {{"exporter_transport_port", decodeUint}}, //exporterTransportPort
214: {{"export_proto_version", decodeUint}}, // exportProtocolVersion
215: {{"export_transport_proto", decodeUint}}, // exportTransportProtocol
216: {{"collector_transport_port", decodeUint}}, // collectorTransportPort
217: {{"exporter_transport_port", decodeUint}}, // exporterTransportPort
218: {{"tcp_syn_total", decodeUint}}, // tcpSynTotalCount
219: {{"tcp_fin_total", decodeUint}}, // tcpFinTotalCount
220: {{"tcp_rst_total", decodeUint}}, // tcpRstTotalCount
Expand Down
3 changes: 2 additions & 1 deletion plugins/inputs/netflow/netflow_v5.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,10 @@ import (
"net"
"time"

"github.com/netsampler/goflow2/decoders/netflowlegacy"

"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/metric"
"github.com/netsampler/goflow2/decoders/netflowlegacy"
)

// Decoder structure
Expand Down
5 changes: 3 additions & 2 deletions plugins/inputs/netflow/sample.conf
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Netflow v5, Netflow v9 and IPFIX collector
[[inputs.netflow]]
## Address to listen for netflow/ipfix packets.
## Address to listen for netflow,ipfix or sflow packets.
## example: service_address = "udp://:2055"
## service_address = "udp4://:2055"
## service_address = "udp6://:2055"
Expand All @@ -13,9 +13,10 @@

## Protocol version to use for decoding.
## Available options are
## "ipfix" -- IPFIX / Netflow v10 protocol (also works for Netflow v9)
## "netflow v5" -- Netflow v5 protocol
## "netflow v9" -- Netflow v9 protocol (also works for IPFIX)
## "ipfix" -- IPFIX / Netflow v10 protocol (also works for Netflow v9)
## "sflow v5" -- sFlow v5 protocol
# protocol = "ipfix"

## Dump incoming packets to the log
Expand Down
Loading