Add support for goflow2 to support IPFIX as sFlow alternative

[jbemmel]

Is your feature request related to a problem? Please describe.
I'm always frustrated when I cannot use IXP manager because a device only supports IPFIX and not sFlow

Describe the solution you'd like
In addition to sflowtool, I'd like to see support for goflow2 (https://github.com/netsampler/goflow2).
goflow2 produces JSON output (amongst others) and the sflow processing Perl scripts could be adapted to read that

Describe alternatives you've considered
I've submitted a PR to goflow2 (netsampler/goflow2#163) to adapt its output to the format that IXP Manager expects (CSV), but the maintainers prefer to keep their side simple, and suggested to add JSON parsing support to IXP Manager instead. I tend to agree with them

Additional context
The devices in question are Nokia routers based on SR OS - these are used by the biggest IXPs in the world.
For example, at NL-IX: https://www.nokia.com/about-us/news/releases/2022/10/19/nokia-provides-nl-ix-with-ip-platforms-to-enable-400ge-and-800ge-access-and-interconnection-services/

[barryo]

I've assigned the PR to @nickhilliard because he's the Perl expert.

I'm always frustrated when I cannot use IXP manager because a device only supports IPFIX and not sFlow

There's no need to be frustrated. IXP Manager is open source. If you've written the parser per your PR, why don't you just use it? Maybe I'm missing it but I'm seeing an actual problem here that you haven't / couldn't solve to alleviate said frustration?

[jbemmel]

@barryo just a tongue-in-cheek comment derived from the issue template - "frustrated" is a big word

I was reading https://docs.ixpmanager.org/features/sflow/#netflow and thought that MAC addresses are not required for the BGP peering use case

[barryo]

I was reading https://docs.ixpmanager.org/features/sflow/#netflow and thought that MAC addresses are not required for the BGP peering use case

We definitely need MAC addresses for both p2p graphs.

From memory I think we do peering matrix based on IP address. But we'd still need your / other script to parse the export.

Not sure if that addresses your comment there?

[jbemmel]

Looking at https://github.com/inex/IXP-Manager/blob/master/tools/runtime/sflow/sflow-to-rrd-handler#L159 it seems the MAC addresses are used to lookup the interfaces

My IPFIX events don't include MACs, but they do include interface info:

{"Type":"NETFLOW_V9","TimeReceived":1682607002,"SequenceNum":1,"SamplingRate":10,"FlowDirection":0,"SamplerAddress":"192.168.121.102","TimeFlowStart":1682606929,"TimeFlowEnd":1682606929,"TimeFlowStartMs":1682606928010,"TimeFlowEndMs":1682606928010,"Bytes":52,"Packets":1,"SrcAddr":"80.81.192.10","DstAddr":"80.81.194.114","Etype":2048,"Proto":6,"SrcPort":179,"DstPort":49618,"InIf":1,"OutIf":0,"SrcMac":"00:00:00:00:00:00","DstMac":"00:00:00:00:00:00","SrcVlan":0,"DstVlan":0,"VlanId":0,"IngressVrfId":0,"EgressVrfId":0,"IpTos":192,"ForwardingStatus":64,"IpTtl":1,"TcpFlags":16,"IcmpType":0,"IcmpCode":0,"Ipv6FlowLabel":0,"FragmentId":0,"FragmentOffset":0,"BiFlowDirection":0,"SrcAs":12552,"DstAs":12552,"NextHop":"0.0.0.0","NextHopAs":0,"SrcNet":22,"DstNet":32,"BgpNextHop":[80,81,194,114],"BgpCommunities":[],"AsPath":[],"HasMpls":false,"MplsCount":0,"Mpls_1Ttl":0,"Mpls_1Label":0,"Mpls_2Ttl":0,"Mpls_2Label":0,"Mpls_3Ttl":0,"Mpls_3Label":0,"MplsLastTtl":0,"MplsLastLabel":0,"MplsLabelIp":[],"ObservationDomainId":0,"ObservationPointId":0,"CustomInteger_1":0,"CustomInteger_2":0,"CustomInteger_3":0,"CustomInteger_4":0,"CustomInteger_5":0,"CustomBytes_1":[],"CustomBytes_2":[],"CustomBytes_3":[],"CustomBytes_4":[],"CustomBytes_5":[],"CustomList_1":[],"XXX_NoUnkeyedLiteral":{},"XXX_unrecognized":[],"XXX_sizecache":0}

{"Type":"NETFLOW_V9","TimeReceived":1682607002,"SequenceNum":1,"SamplingRate":10,"FlowDirection":0,"SamplerAddress":"192.168.121.102","TimeFlowStart":1682606986,"TimeFlowEnd":1682606986,"TimeFlowStartMs":1682606985030,"TimeFlowEndMs":1682606985030,"Bytes":71,"Packets":1,"SrcAddr":"80.81.193.202","DstAddr":"80.81.194.114","Etype":2048,"Proto":6,"SrcPort":179,"DstPort":49617,"InIf":1,"OutIf":0,"SrcMac":"00:00:00:00:00:00","DstMac":"00:00:00:00:00:00","SrcVlan":0,"DstVlan":0,"VlanId":0,"IngressVrfId":0,"EgressVrfId":0,"IpTos":192,"ForwardingStatus":64,"IpTtl":1,"TcpFlags":24,"IcmpType":0,"IcmpCode":0,"Ipv6FlowLabel":0,"FragmentId":0,"FragmentOffset":0,"BiFlowDirection":0,"SrcAs":12552,"DstAs":12552,"NextHop":"0.0.0.0","NextHopAs":0,"SrcNet":22,"DstNet":32,"BgpNextHop":[80,81,194,114],"BgpCommunities":[],"AsPath":[],"HasMpls":false,"MplsCount":0,"Mpls_1Ttl":0,"Mpls_1Label":0,"Mpls_2Ttl":0,"Mpls_2Label":0,"Mpls_3Ttl":0,"Mpls_3Label":0,"MplsLastTtl":0,"MplsLastLabel":0,"MplsLabelIp":[],"ObservationDomainId":0,"ObservationPointId":0,"CustomInteger_1":0,"CustomInteger_2":0,"CustomInteger_3":0,"CustomInteger_4":0,"CustomInteger_5":0,"CustomBytes_1":[],"CustomBytes_2":[],"CustomBytes_3":[],"CustomBytes_4":[],"CustomBytes_5":[],"CustomList_1":[],"XXX_NoUnkeyedLiteral":{},"XXX_unrecognized":[],"XXX_sizecache":0}

I believe a Perl script to process these JSON events could be written, similar to the sflow one but not requiring/using MAC addresses

[jbemmel]

I guess my questions are:

1. Would the project be willing to consider adding this functionality?
2. If so, what would be preferred:
   a) Modify the 2 Perl scripts to support both sflowtool ASCII parsing and goflow2 JSON parsing
   b) Create separate Perl scripts for goflow2

[niekvanderven]

Dear community,
Is there somebody that can help Jeroen and me to get this IPFIX / Cflowd available as another option inside IXP-Manager ?
So we can write the code etc.. we only need support on the way to integrate it into IXP-Manager as a logging/visibility option.
We started to work on a Containerlab setup for this so we could integrate it easily to any Vendor needed that is already in Containerlab present nowadays, this will bring us a light weigh CI/CD testing/development platform.
Thank you Jeroen and Niek