[Dual-ToR] Tunnel route creation/removal causes packet duplication during interfaces recovering from DOWN to UP #16161

ayurkiv-nvda · 2023-08-15T08:44:14Z

Description

Problem is related to #16085
Found during running community test dualtor_io/test_link_failure.py::test_active_link_admin_down_config_reload_link_up_downstream_standby[active-active]

Steps to reproduce the issue:

Config Active-Active Dual-ToR setup, make sure that both switches have all mux ports active.
Shutdown ALL mux interface:

config interface shutdown Ethernet4
config interface shutdown Ethernet8
config interface shutdown Ethernet12
config interface shutdown Ethernet16
config interface shutdown Ethernet20
config interface shutdown Ethernet24
config interface shutdown Ethernet28
config interface shutdown Ethernet32
config interface shutdown Ethernet36
config interface shutdown Ethernet40
config interface shutdown Ethernet44
config interface shutdown Ethernet48
config interface shutdown Ethernet52
config interface shutdown Ethernet56
config interface shutdown Ethernet60
config interface shutdown Ethernet64
config interface shutdown Ethernet68
config interface shutdown Ethernet72
config interface shutdown Ethernet76
config interface shutdown Ethernet80
config interface shutdown Ethernet84
config interface shutdown Ethernet88
config interface shutdown Ethernet92
config interface shutdown Ethernet96

config save -y
config reload -y
Run continuous traffic from T1 to the server via Standby ToR via all interfaces
Statup ALL mux interfaces:

config interface startup Ethernet4
config interface startup Ethernet8
config interface startup Ethernet12
config interface startup Ethernet16
config interface startup Ethernet20
config interface startup Ethernet24
config interface startup Ethernet28
config interface startup Ethernet32
config interface startup Ethernet36
config interface startup Ethernet40
config interface startup Ethernet44
config interface startup Ethernet48
config interface startup Ethernet52
config interface startup Ethernet56
config interface startup Ethernet60
config interface startup Ethernet64
config interface startup Ethernet68
config interface startup Ethernet72
config interface startup Ethernet76
config interface startup Ethernet80
config interface startup Ethernet84
config interface startup Ethernet88
config interface startup Ethernet92
config interface startup Ethernet96

Describe the results you received:

E       Failed:
E       Traffic to server 192.168.0.6 was duplicated 3 times. Allowed number of duplications: 0
E       Traffic to server 192.168.0.18 was duplicated 23 times. Allowed number of duplications: 0
E       Traffic to server 192.168.0.36 was duplicated 18 times. Allowed number of duplications: 0

After step#4 config reload -y according to #16085, routes to tunnel are not created ( traffic generated on step#5 will not reach the destination). There are no routes for mux IPs (192.168.0.2, 192.168.0.4, 192.168.0.6, ... 192.168.0.48):

Interface startup (config interface startup Ethernet4, etc) will trigger a tunnel route creation event for all mux interfaces.
Shortly after this, when port is UP the tunnel route will be removed.
DUT also will start sending ARP requests. Or it may receive an ARP reply during GARP notification (once in 10s)

There is a time gap between removing tunnel route and new FDB entry (~150ms)
If, during this period neighbor is known, there is no FDB in the table, and packets are sent - these packets will be flooded on all interfaces currently UP on VLAN.

In our case, traffic is flooded on a few interfaces (192.168.0.6, 192.168.0.18, 192.168.0.36)

Flow:

13:31:43.466725 r-tigon-20 NOTICE swss#orchagent: :- create_route: Created tunnel route to 192.168.0.18/32 
----> Created tunnel route to active ToR

13:31:43.934 ---> last dropped packet (traffic if forwarded via tunnel now)
.....
13:31:50.784794 00:aa:bb:cc:dd:ee > 62:b6:5a:e7:5e:09, ethertype IPv4 (0x0800), length 117: 192.168.79.109.1234 > 192.168.0.18.5000: Flags [S], seq 0:46, win 8192, length 46
13:31:51.038380 00:aa:bb:cc:dd:ee > 62:b6:5a:e7:5e:09, ethertype IPv4 (0x0800), length 117: 192.168.72.180.1234 > 192.168.0.18.5000: Flags [S], seq 0:46, win 8192, length 46
13:31:51.292395 00:aa:bb:cc:dd:ee > 62:b6:5a:e7:5e:09, ethertype IPv4 (0x0800), length 117: 192.168.92.144.1234 > 192.168.0.18.5000: Flags [S], seq 0:46, win 8192, length 46
13:31:51.546852 00:aa:bb:cc:dd:ee > 62:b6:5a:e7:5e:09, ethertype IPv4 (0x0800), length 117: 192.168.95.214.1234 > 192.168.0.18.5000: Flags [S], seq 0:46, win 8192, length 46
13:31:51.800451 00:aa:bb:cc:dd:ee > 62:b6:5a:e7:5e:09, ethertype IPv4 (0x0800), length 117: 192.168.110.134.1234 > 192.168.0.18.5000: Flags [S], seq 0:46, win 8192, length 46

13:31:51.989434 62:b6:5a:e7:5e:09 > 00:aa:bb:cc:dd:ee, ethertype ARP (0x0806), length 60: Reply 192.168.0.18 is-at 62:b6:5a:e7:5e:09, length 46 
---> ARP sent to DUT, FDB entry will be created soon.

13:31:52.010704 r-tigon-20 NOTICE swss#orchagent: :- remove_route: Removed tunnel route to 192.168.0.18/32 
---> remove tunnel route to Active ToR

13:31:52.053375 62:b6:5a:e7:5e:09 > 00:aa:bb:cc:dd:ee, ethertype ARP (0x0806), length 60: Reply 192.168.0.18 is-at 62:b6:5a:e7:5e:09, length 46  
----> another one ARP packet is sent to DUT
13:31:52.053383 00:aa:bb:cc:dd:ee > 62:b6:5a:e7:5e:09, ethertype IPv4 (0x0800), length 117: 192.168.72.197.1234 > 192.168.0.18.5000: Flags [S], seq 0:46, win 8192, length 46 
---> this packet is flooded on all interfaces in VLAN that are UP (We see a lot of copies of this packet on ports).
At this moment, tunnel already removed, but we don't have an FDB entry with MAC 62:b6:5a:e7:5e:09 

13:31:52.093 - still no FDB entry in the table

13:31:52.211 - FDB entry "62:b6:5a:e7:5e:09 Ethernet36" appeared in the table

13:31:52.305934 00:aa:bb:cc:dd:ee > 62:b6:5a:e7:5e:09, ethertype IPv4 (0x0800), length 117: 192.168.69.188.1234 > 192.168.0.18.5000: Flags [S], seq 0:46, win 8192, length 46  
---> Traffic successfully reached the server directly (without a tunnel route), continuing the normal flow without flooding

NOTE:
Dehavior described for 192.168.0.18.
It is identical to 192.168.0.36
BUT there are no create_route/remove_route logs for 192.168.0.6 even though the traffic behavior for 192.168.0.6 is identical to 192.168.0.18 and 192.168.0.36. It is currently not clear to me what is the reason of the missing logs. I think I will do few more tests and will update the ticket.

Describe the results you expected:

No duplication.

Output of `show version`:

202211.66-1e1974709_Internal

Output of `show techsupport`:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

The text was updated successfully, but these errors were encountered:

yxieca · 2023-08-21T18:15:55Z

@ayurkiv-nvda can you confirm that the reported 'duplicate' was from vlan flooding?

If this is the case, in real life, these IO would be flooded to different servers and being dropped by all but the actual receiver.

@prsunny I think in dualtor case, the vlan flooding should be disabled?

lolyu · 2023-08-22T09:23:19Z

Hi @ayurkiv-nvda, could you please share the logs?
With proxy_arp enabled on the Vlan interface, the flooding should be disabled, could you please check if proxy_arp is enabled like the below?

> hgetall VLAN_INTERFACE|Vlan1000
1) "grat_arp"
2) "enabled"
3) "proxy_arp"
4) "enabled"

ayurkiv-nvda · 2023-08-22T09:43:33Z

@ayurkiv-nvda can you confirm that the reported 'duplicate' was from vlan flooding?

If this is the case, in real life, these IO would be flooded to different servers and being dropped by all but the actual receiver.

@prsunny I think in dualtor case, the vlan flooding should be disabled?

Yes, this is a simultaneous flood to all ports in VLAN that are currently in UP state (no FDB but neighbor exists)

ayurkiv-nvda · 2023-08-22T23:27:39Z

Yes, we have proxy arp enabled on our Dual-ToR setup

127.0.0.1:6379[4]> hgetall VLAN_INTERFACE|Vlan1000
1) "grat_arp"
2) "enabled"
3) "proxy_arp"
4) "enabled"

But as far as I know, proxy arp packet flooding is related to broadcast flooding
In our case unicast packet is flooded

lolyu · 2023-08-23T08:21:55Z

With the log, SONiC behaved as expected.

lolyu · 2023-08-24T16:14:51Z

dualtor_io/test_link_failure.py::test_active_link_admin_down_config_reload_link_up_downstream_standby[active-active] PASSED                                                                                                          [100%]

I could not reproduce this with public 202211 image: https://sonic-build.azurewebsites.net/ui/sonic/pipelines/1/builds/342905/artifacts?branchName=202211

$ show version

SONiC Software Version: SONiC.202211.342905-adb43ff1f
SONiC OS Version: 11
Distribution: Debian 11.7
Kernel: 5.10.0-18-2-amd64
Build commit: adb43ff1f
Build date: Sun Aug 20 15:13:07 UTC 2023
Built by: AzDevOps@vmss-soni001TF1

@ayurkiv-nvda, could you please share the link to the 202211 image you are using?

ayurkiv-nvda · 2023-08-29T11:52:49Z

@lolyu I used 202211.66-1e1974709_Internal

I think it is here:
https://sonic-build.azurewebsites.net/ui/sonic/pipelines/1/builds/332681/artifacts/502366?branchName=202211&artifactName=sonic-buildimage.mellanox

I will check SONiC.202211.342905-adb43ff1f also, maybe it was fixed in earliest build

ayurkiv-nvda · 2023-09-18T16:18:09Z

I checked it on SONiC.202211.342905-adb43ff1f , 202211.362946-9ffa92cc6 and 202211.70-e7ce179b7_Internal
The issue is no longer reproduced on this branch
But on 202205 problem is still exist

ayurkiv-nvda · 2023-11-06T15:55:32Z

Managed to reproduce it on 202211.72-cf66a45b8_Internal
It looks like issue is still exist on 202211, reproduce rate is ~30%

Do we have some progress with this bug?

lolyu · 2023-11-16T07:55:37Z

@ayurkiv-nvda, the issue is that, after the port admin up, when the neighbor is added and tunnel route is removed, the fdb is not there, right?
And this could be confirmed from the techsupport you shared:
192.168.0.18 Ethernet32 62:B6:5A:E7:5E:09

2023-08-14.13:31:52.002726|c|SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"192.168.0.18","rif":"oid:0x6000000000b59","switch_id":"oid:0x21000000000000"}|SAI_NEIGHBOR_ENTRY_ATTR_DST_MAC_ADDRESS=62:B6:5A:E7:5E:09
2023-08-14.13:31:52.006085|c|SAI_OBJECT_TYPE_NEXT_HOP:oid:0x4000000000c24|SAI_NEXT_HOP_ATTR_TYPE=SAI_NEXT_HOP_TYPE_IP|SAI_NEXT_HOP_ATTR_IP=192.168.0.18|SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID=oid:0x6000000000b59
2023-08-14.13:31:52.008355|r|SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"192.168.0.18/32","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000002"}
2023-08-14.13:31:52.026528|c|SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"192.168.0.19","rif":"oid:0x6000000000b59","switch_id":"oid:0x21000000000000"}|SAI_NEIGHBOR_ENTRY_ATTR_DST_MAC_ADDRESS=A6:5D:44:EC:7B:84
2023-08-14.13:31:52.029167|c|SAI_OBJECT_TYPE_NEXT_HOP:oid:0x4000000000c25|SAI_NEXT_HOP_ATTR_TYPE=SAI_NEXT_HOP_TYPE_IP|SAI_NEXT_HOP_ATTR_IP=192.168.0.19|SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID=oid:0x6000000000b59
2023-08-14.13:31:52.031427|r|SAI_OBJECT_TYPE_ROUTE_ENTRY:{"dest":"192.168.0.19/32","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000002"}
**2023-08-14.13:31:52.095415|n|fdb_event|[{"fdb_entry":"{\"bvid\":\"oid:0x26000000000b24\",\"mac\":\"62:B6:5A:E7:5E:09\",\"switch_id\":\"oid:0x21000000000000\"}","fdb_event":"SAI_FDB_EVENT_LEARNED","list":[{"id":"SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID","value":"oid:0x3a000000000b31"},{"id":"SAI_FDB_ENTRY_ATTR_TYPE","value":"SAI_FDB_ENTRY_TYPE_DYNAMIC"},{"id":"SAI_FDB_ENTRY_ATTR_PACKET_ACTION","value":"SAI_PACKET_ACTION_FORWARD"}]}]|**

The tunnel route is removed at 2023-08-14.13:31:52.031427, and the fdb event comes at 2023-08-14.13:31:52.095415

ayurkiv-nvda · 2023-12-07T08:52:12Z

Hello @lolyu
Yes, sometimes there is a time gap between removing tunnel route and new FDB entry (~150ms)
If, during this period neighbor is known, there is no FDB in the table, and packets are sent - these packets will be flooded on all interfaces currently UP on VLAN.

lolyu · 2023-12-07T09:56:58Z

Hi @ayurkiv-nvda, thanks for the clarification. The current dualtor-io test infrastructure cannot tell which port the packet is received, and the testcase will complain with error if this unknown unicast flood happens. This should be a test infrastructure gap.

So let's mark this as a to-do item.

lolyu · 2023-12-12T02:54:38Z

Config Active-Active Dual-ToR setup, make sure that both switches have all mux ports active.

Hi @ayurkiv-nvda, could you please have this fixed as it is specific to mlnx platform.

ayurkiv-nvda added Dual ToR Platform ♊ Issues found on dual ToR platforms Issue for 202205 Issue for 202211 labels Aug 15, 2023

gechiang assigned lolyu Aug 16, 2023

gechiang added Triaged this issue has been triaged MSFT labels Aug 16, 2023

ayurkiv-nvda changed the title ~~[Dual-ToR] Tunnel creation/removal causes packet duplication during interfaces recovering from DOWN to UP~~ [Dual-ToR] Tunnel route creation/removal causes packet duplication during interfaces recovering from DOWN to UP Aug 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dual-ToR] Tunnel route creation/removal causes packet duplication during interfaces recovering from DOWN to UP #16161

[Dual-ToR] Tunnel route creation/removal causes packet duplication during interfaces recovering from DOWN to UP #16161

ayurkiv-nvda commented Aug 15, 2023 •

edited

Loading

yxieca commented Aug 21, 2023

lolyu commented Aug 22, 2023

ayurkiv-nvda commented Aug 22, 2023

ayurkiv-nvda commented Aug 22, 2023

lolyu commented Aug 23, 2023 •

edited

Loading

lolyu commented Aug 24, 2023 •

edited

Loading

ayurkiv-nvda commented Aug 29, 2023

ayurkiv-nvda commented Sep 18, 2023 •

edited

Loading

ayurkiv-nvda commented Nov 6, 2023 •

edited

Loading

lolyu commented Nov 16, 2023

ayurkiv-nvda commented Dec 7, 2023

lolyu commented Dec 7, 2023 •

edited

Loading

lolyu commented Dec 12, 2023

[Dual-ToR] Tunnel route creation/removal causes packet duplication during interfaces recovering from DOWN to UP #16161

[Dual-ToR] Tunnel route creation/removal causes packet duplication during interfaces recovering from DOWN to UP #16161

Comments

ayurkiv-nvda commented Aug 15, 2023 • edited Loading

Description

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of show version:

Output of show techsupport:

Additional information you deem important (e.g. issue happens only occasionally):

yxieca commented Aug 21, 2023

lolyu commented Aug 22, 2023

ayurkiv-nvda commented Aug 22, 2023

ayurkiv-nvda commented Aug 22, 2023

lolyu commented Aug 23, 2023 • edited Loading

lolyu commented Aug 24, 2023 • edited Loading

ayurkiv-nvda commented Aug 29, 2023

ayurkiv-nvda commented Sep 18, 2023 • edited Loading

ayurkiv-nvda commented Nov 6, 2023 • edited Loading

lolyu commented Nov 16, 2023

ayurkiv-nvda commented Dec 7, 2023

lolyu commented Dec 7, 2023 • edited Loading

lolyu commented Dec 12, 2023

ayurkiv-nvda commented Aug 15, 2023 •

edited

Loading

Output of `show version`:

Output of `show techsupport`:

lolyu commented Aug 23, 2023 •

edited

Loading

lolyu commented Aug 24, 2023 •

edited

Loading

ayurkiv-nvda commented Sep 18, 2023 •

edited

Loading

ayurkiv-nvda commented Nov 6, 2023 •

edited

Loading

lolyu commented Dec 7, 2023 •

edited

Loading