Implement SNMP trap reception mechanism

[lunkwill42]

Design considerations

Zino 1 uses a trap daemon model taken from Scotty, wherein an SNMP trap multiplexer daemon (straps or nmtrapd) is run as root, and any Scotty program (like Zino) that wants a copy of all received traps can run as unprivileged users and connect to a UNIX socket to receive a stream of the received trap messages.

While this is a really nice model for both debugging and for having multiple small and independent programs that can run on the same machine and react to traps, we don't see this as an essential model to replicate.

In Zino 2, we will instead opt for a model where the Zino program itself will listen for incoming SNMP trap messages (on UDP port 162). This does have the side-effect that the Zino program must be started by root, and needs to be able to switch to a less privileged user immediately after opening the privileged port 162.

A model where Zino connects to a trap message multiplexer can be added later if deemed necessary.

Sub-tasks

• Add initial trap receiver functionality to Zino #189

• Implement a generic trap dispatcher framework in zino.trapd.TrapReceiver #193

• Traps that are handled by the original Zino in some way or the other

  • These traps cause Zino to take action:

    • Implement handling of link state traps (IF-MIB::linkUp and IF-MIB::linkDown) #232
    • Implement handling of BFD traps (BFD-STD-MIB::bfdSessUp and BFD-STD-MIB::bfdSessDown) #292
    • Implement handling of Juniper BGP traps (jnxBgpM2BackwardTransition and jnxBgpM2Established from BGP4-V2-MIB-JUNIPER) #288

  • These traps are only parsed and logged (in simple or less simple ways):

    • Implement SNMPv2-MIB::coldStart and SNMPv2-MIB::warmStart trap loggers #226
    • Add custom trap logging #319

  • These traps seem to be just ignored entirely:

    • bgpBackwardTransition (a.k.a. BGP4-MIB::bgpBackwardTransition)
    • BGP4-MIB::bgpBackwardTransNotification
    • jnxBgpM2Experiment.1.1.0.2 (a.k.a BGP4-V2-MIB-JUNIPER::jnxBgpM2BackwardTransition)
    • SNMPv2-MIB::authenticationFailure
    • CISCOTRAP-MIB::tcpConnectionClose (but contains some commented out logging code)
    • BGP4-V2-MIB-JUNIPER::jnxBgpM2BackwardTransition

  • The default trap handler just logs incoming trap details as an "unknown trap"

[lunkwill42]

The straps SNMP trap multiplexer is described here: https://linuxcertif.com/man/8/straps/ - but I am hard pressed to find the actual package or source code, at the moment...

[lunkwill42]

Some more digging, and it turns out straps was renamed to nmtrapd in 1996 (!). Source code seems to be available here: https://github.com/flightaware/scotty/blob/23645564467704119dca051b6c8b16a4c56b74af/tnm/unix/nmtrapd.c

The man page source corresponding to the link above is here: https://github.com/flightaware/scotty/blob/23645564467704119dca051b6c8b16a4c56b74af/tnm/doc/nmtrapd.8 . The manpage describes the output format from the multiplexer's UNIX socket; the original SNMP trap packets can be gotten from this, and if PySNMP can be made to process raw packets, Zino 2.0 could be adapted to support both a built-in trap daemon, or fetch data from a multiplexer daemon like nmtrapd.

[lunkwill42]

A PySNMP reference: https://pysnmp.readthedocs.io/en/latest/examples/v3arch/asyncore/manager/ntfrcv/transport-tweaks.html

[he32]

Design considerations

Zino 1 uses a trap daemon model taken from Scotty, wherein an SNMP trap multiplexer daemon (straps or nmtrapd) is run as root, and any Scotty program (like Zino) that wants a copy of all received traps can run as unprivileged users and connect to a UNIX socket to receive a stream of the received trap messages.

While this is a really nice model for both debugging and for having multiple small and independent programs that can run on the same machine and react to traps, we don't see this as an essential model to replicate.

In Zino 2, we will instead opt for a model where the Zino program itself will listen for incoming SNMP trap messages (on UDP port 162). This does have the side-effect that the Zino program must be started by root, and needs to be able to switch to a less privileged user immediately after opening the privileged port 162.

A model where Zino connects to a trap message multiplexer can be added later if deemed necessary.

Please note that we're already using this multiplexing method actively.
We have a separate SNMP trap listening program running on our main
status monitoring host which takes note of configuration change trap
messages, and appropriately delayed posts messages using curl to our
oxidized instance to pick up the newly modified configuration in a somewhat
timely manner.

The above method means that you can only have a single program listening
for SNMP trap messages on this host, and you need to "burn a new IP address"
for another host and configure all your routers to also send trap messages that
direction if you want to consume SNMP trap messages for more than one single
purpose. I know also that in the past there has been bugs in our preferred router
OS when more than N (where N was 2?) trap recipients were configured.

[lunkwill42]

The above method means that you can only have a single program listening for SNMP trap messages on this host, and you need to "burn a new IP address" for another host and configure all your routers to also send trap messages that direction if you want to consume SNMP trap messages for more than one single purpose. I know also that in the past there has been bugs in our preferred router OS when more than N (where N was 2?) trap recipients were configured.

We definitely understand the advantages, @he32, but we don't want to make Zino 2 immediately depend on this 3rd party software for trap reception as its default implementation. This is why we've made the basic trap daemon from scratch, so that we can concentrate on the actual trap management code - and then down the road, we can add support for a trap multiplexer as an alternative mode of operation. An interesting question in that regard: Could the nmtrapd program be easily forked out of the Scotty codebase without having lots of dependencies back into that codebase? Or must the full Scotty package be compiled to build nmtrapd?

[lunkwill42]

All associated sub-tasks have been closed, this can now also be closed