[Multi-Asic] Forward SNMP requests received on front panel interface to SNMP agent in host. #5420
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…g in through the front panel interface
present in the network namespace, to SNMP agent running in the linux host.
abdosi
requested changes
Sep 21, 2020
judyjoseph
requested review from
jleveque,
SuvarnaMeenakshi,
arlakshm and
lguohan
abdosi
reviewed
Sep 21, 2020
abdosi
previously approved these changes
Sep 23, 2020
jleveque
reviewed
Sep 23, 2020
…created by some other process.
abdosi
reviewed
Sep 23, 2020
files/image_config/caclmgrd/caclmgrd
Outdated
| @@ -235,9 +231,6 @@ class ControlPlaneAclManager(daemon_base.DaemonBase): | |||
| (self.ACL_SERVICES['SNMP']['dst_ports'][0], self.namespace_docker_mgmt_ip[namespace])) | |||
|
|
|||
| # IPv6 rules | |||
| fwd_snmp_traffic_from_namespace_to_host_cmds.append(self.iptables_cmd_ns_prefix[namespace] + "ip6tables -t nat -X") | |||
| fwd_snmp_traffic_from_namespace_to_host_cmds.append(self.iptables_cmd_ns_prefix[namespace] + "ip6tables -t nat -F") | |||
There was a problem hiding this comment.
@judyjoseph why we remove this ? If calclmgrd starts again we will keep on adding rules.
For namespace we should not remove it.
There was a problem hiding this comment.
Ok Thanks for pointing out .. Let me keep it back, I mistook the caclmgrd service as a one shot service, but it has restart option if it fails for some reason/exception.
There was a problem hiding this comment.
@abdosi, instead of keeping the original code, have added new delete rules with the "iptables -D" option. This will delete only the rules which we create/own. I feel this is always better than doing "ip6tables/iptables -t nat -F/-X" which will delete all the rules present in the system ( created by other processes also) at the time when caclmgrd restarts.
There was a problem hiding this comment.
@judyjoseph that is fine but I feel that might not be helpful since if there are other process installing nat rules we have other challenges how to maintain order of rules since other rules can be higher priority then these rules.
I feel once we nat feature in namespace or any other process then there will be more thought and refactoring will be needed.
I still feel since it is multi asic specific change we can start from clean slate on every restart
Another benefit is that we delete only our rules, rather than earlier approach of "iptables -F" which cleans up all rules.
…AT feature added in namespace.
|
retest vsimage please |
abdosi
previously approved these changes
Sep 24, 2020
abdosi
approved these changes
Sep 26, 2020
abdosi
pushed a commit
that referenced
this pull request
Sep 28, 2020
…to SNMP agent in host. (#5420) * [Multi-Asic] Forward SNMP requests destined to loopback IP, and coming in through the front panel interface present in the network namespace, to SNMP agent running in the linux host. * Updates based on comments * Further updates in docker_image_ctl.j2 and caclmgrd * Change the variable for net config file. * Updated the comments in the code. * No need to clean up the exising NAT rules if present, which could be created by some other process. * Delete our rule first and add it back, to take care of caclmgrd restart. Another benefit is that we delete only our rules, rather than earlier approach of "iptables -F" which cleans up all rules. * Keeping the original logic to clean the NAT entries, to revist when NAT feature added in namespace. * Missing updates to log_info call.
santhosh-kt
pushed a commit
to santhosh-kt/sonic-buildimage
that referenced
this pull request
Feb 25, 2021
…to SNMP agent in host. (sonic-net#5420) * [Multi-Asic] Forward SNMP requests destined to loopback IP, and coming in through the front panel interface present in the network namespace, to SNMP agent running in the linux host. * Updates based on comments * Further updates in docker_image_ctl.j2 and caclmgrd * Change the variable for net config file. * Updated the comments in the code. * No need to clean up the exising NAT rules if present, which could be created by some other process. * Delete our rule first and add it back, to take care of caclmgrd restart. Another benefit is that we delete only our rules, rather than earlier approach of "iptables -F" which cleans up all rules. * Keeping the original logic to clean the NAT entries, to revist when NAT feature added in namespace. * Missing updates to log_info call.
abdosi
added a commit
to sonic-net/sonic-mgmt
that referenced
this pull request
Mar 2, 2021
Enhanced test_cacl_application.py for multi-asic platforms. Also it adds new test case to cover all multi-asic specific changes as done in PR's:- sonic-net/sonic-buildimage#5022 sonic-net/sonic-buildimage#5420 sonic-net/sonic-buildimage#5364 sonic-net/sonic-buildimage#6765 Also fix some of API in common/devices.py and bug in config_facts
10 tasks
qiluo-msft
pushed a commit
that referenced
this pull request
Jul 12, 2023
…15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: #5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
Merged
10 tasks
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Jul 13, 2023
…onic-net#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: sonic-net#5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Jul 13, 2023
…onic-net#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: sonic-net#5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
Merged
10 tasks
mssonicbld
pushed a commit
that referenced
this pull request
Jul 13, 2023
…15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: #5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
yxieca
pushed a commit
that referenced
this pull request
Jul 14, 2023
…15487) (#15826) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: #5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> Co-authored-by: SuvarnaMeenakshi <[email protected]>
Merged
10 tasks
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Jul 17, 2023
…onic-net#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: sonic-net#5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
Closed
10 tasks
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Jul 26, 2023
…onic-net#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: sonic-net#5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
SuvarnaMeenakshi
added a commit
to SuvarnaMeenakshi/sonic-buildimage
that referenced
this pull request
Aug 29, 2023
…onic-net#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: sonic-net#5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> (cherry picked from commit 9864dfe)
11 tasks
sonic-otn
pushed a commit
to sonic-otn/sonic-buildimage
that referenced
this pull request
Sep 20, 2023
…onic-net#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: sonic-net#5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
- Why I did it
The SNMP loopback health check and other SNMP requests coming through the front panel interfaces where not getting serviced by the SNMP agent running in the SNMP container in the linux host network namespace. This change is to create translation rules so that the SNMP packets coming in would be translated and forwarded to the agent running in linux host.
- How I did it
Create NAT rules for trapping the SNMP packets coming in the front panel interface in the linux network namespace and sent to the docker0 subnet 240.12.1.x. The NAT rules added will be for SNMP packets, which are UDP + dest port 161
change the destination IP to that of the docker0 eth0 IP address in the host which is 240.127.1.1. This is DNAT rule
change the Source IP to that of the docker0 eth0 IP address in the namespace which could be any of 240.127.1.x. This is SNAT rule
With these rules, the SNMP packets coming from front panel interface destined to the Loopback IP in the namespace gets translated to 240.127.1.x subnet, reaches the SNMP agents running in HOST. The agent's response is in 240.127.1.x subnet, reaches the respective network namespace. The NAT module finds entries in connection tracking table and changes it back to the external IP address, which is then routed out through front panel interface.
When the NAT translation happens with SNAT/DNAT rules, it creates connection tracking entry. For eg: the connection tracking entry created is here, where the values 171,164 is the entry timeout value for which it will exists, Original 5 tuple value incoming SNMP packet, the invert tuple is stored to be matched in the reverse direction.
- How to verify it
Verified with SNMP requests through the front panel interface working correctly for both IPv4 and IPv6.
- Which release branch to backport (provide reason below if selected)
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)