diff --git a/L3/L3_performance_and_scaling_enhancements/L3_Vlan_interface_scaling.md b/L3/L3_performance_and_scaling_enhancements/L3_Vlan_interface_scaling.md index 27818e529883..3b692539168f 100644 --- a/L3/L3_performance_and_scaling_enhancements/L3_Vlan_interface_scaling.md +++ b/L3/L3_performance_and_scaling_enhancements/L3_Vlan_interface_scaling.md @@ -43,15 +43,17 @@ SONIC currently supports a maximum of 1k L3 Vlan interfaces on Broadcom hardware ## 1.1 Functional Requirements Scaling improvements -|--------------------------------------|------|-----|-----|-----| -| MAC | TD3 | TD4 | TH | TH2 | -|--------------------------------------|------|-----|-----|-----| -|System MAC configured SVI | 4k | 4k | 4k | 4k | -|MCLAG Gateway MAC with same ip SVI | 4k | 4k | 4k | 4k | -|MCLAG with unique ip SVI | 4k | 4k | 1k | 1k | -|SAG MAC configured SVI | 4k | 4k | 1k | 1k | -|VRRP MAC SVI | 4k | 4k | 1k | 1k | -|--------------------------------------|------|-----|-----|-----| +|--------------------------------------|------|-----|-----|-----|-----| +| MAC | TD3 | TD4 | TH | TH2 | TH3 | +|--------------------------------------|------|-----|-----|-----|-----| +|System MAC configured SVI | 4k | 4k | 4k | 4k | 4k | +|MCLAG Gateway MAC with same ip SVI | 4k | 4k | 4k | 4k | 4k | +|MCLAG with unique ip SVI | 4k | 4k | 1k | 1k | 1k | +|MCLAG with peer gateway enabled SVI | 4k | 4k | 1k | 1k | 1k | +|SAG MAC configured SVI | 4k | 4k | 1k | 1k | 1k | +|VRRP MAC SVI | 4k | 4k | 1k | 1k | 1k | +|--------------------------------------|------|-----|-----|-----|-----| + ## 1.2 Configuration and Management Requirements No new configuration or show commands introduced. @@ -60,7 +62,8 @@ No new configuration or show commands introduced. Covered in Functional requirements ## 1.4 Warm Boot Requirements -Upgrade to higher version should take care of installing one entry per MAC and delete the individual station tcam entries per SVI on TH/TH2 platforms + +Upgrade to higher version should take care of installing one entry per MAC and delete the individual station tcam entries per SVI on TH/TH2/TH3 platforms Upgrade to higher version should take care of moving to l2 table from myStationTcam for all SVI interfaces on TD3 platform. # 2 Design @@ -68,9 +71,12 @@ Upgrade to higher version should take care of moving to l2 table from myStationT ## 2.1 Use L2 table for inner mac termination for TD3 & TD4 TD4 already uses L2 table for inner mac termination. No change for TD4 platform. In TD3, for all SVI interfaces, L2 table will be programmed instead of MyStationTcam. -Entry will be added with respective mac,vfi and flags as BCM_L2_STATIC. One entry per each SVI will be programmed.If 4k VRRP SVI interfaces are configured ,8k L2 table entries will be consumed.Rest of the cases,4k L2 tables entries will be consumed. + +Entry will be added with respective mac,vfi and flags as BCM_L2_STATIC. One entry per each SVI will be programmed. +Note :- If 4k VRRP SVI interfaces are configured ,8k L2 table entries will be consumed.Rest of the cases 4k L2 table entries are consumed. ## 2.2 MyStationTcam lookup key modification +This is applicable for only TH/TH2/TH3 platforms. Use only MAC as the key for MyStationTcam entries, this will give way to use only one MyStationTcam entry for all SVI interfaces, i.e, we could be able to scale to 4K L3 Interfaces. This change is for the following 2 MACs only. 1. System MAC 2. McLAG Gateway MAC @@ -79,11 +85,12 @@ For SVI interfaces the lookup keys is only (DMAC) For Phy interfaces / RouterPorts the lookup key is retained the same [Port Id + DMAC] For LAG router interfaces the lookup key is retained the same [Trunk Id + DMAC] -## 2.2.1 MyStationTcam programming + +### 2.2.1 MyStationTcam programming A new data structure is added and is managed by the sai data manager.This data structure handles the list of Vlan interfaces associated with the given mac and the corresponding MyStationTcam hardware id. When the first SVI interface configured with any of the above macs the corresponding tcam entry gets added. When the last SVI interface associated with the mac is deleted , the corresponding tcam entry will be deleted. -## 2.2.2 Routing behaviour differences +### 2.2.2 Routing behaviour differences With the key change to only (MAC) from (MAC,Vlan), if the SVI interfaces are configured each with one type of MAC. Station tcam will look like below now. For example 1. System MAC - 3c2c992ed875 @@ -99,7 +106,9 @@ If a packet comes with (DA=MCLAG Gateway MAC, Vlan = !MCLAG Vlan). It will be su If a packet comes with (DA=Any of the above two macs, Vlan=L2). It will be subjected to routing. Inorder to address these cases, added below IFP drop rules during system bringup. Also, counters attached to each of the drop rules. -### 2.3 VSI Profile + + +## 2.3 VSI Profile For each SVI interface there is a VSI profile maintained to classify the SVI interfaces. 2 VSI profile bits are consumed. One to identify the SAG MAC vlan and another to identify the MCLAG Gateway MAC vlan. @@ -109,36 +118,73 @@ For each SVI interface there is a VSI profile maintained to classify the SVI int #define _BRCM_SAI_VSI_ROUTER_MAC_L3_INTF (1<<3)------->MCLAG Gateway MAC VSI #define _BRCM_SAI_MAX_VSI_PROFILE_BITS_LENGTH 4 -### 2.4 IFP Drop rules -Below drop rules will be programmed only in TH/TH2 devices. -1. DA=System MAC Vlan = SAG VSI profile ---> Drop -2. DA=System MAC Vlan = MCLAG VSI profile ---> Drop -3. DA=MCLAG GWMAC Vlan = !MCLAG VSI profile ---> Drop + +## 2.4 IFP Drop rules +Below drop rules will be programmed only in TH/TH2/TH3 devices. +1. DA=System MAC Vlan = SAG VSI profile ---> Drop +2. DA=System MAC Vlan = MCLAG VSI profile ---> Drop +3. DA=MCLAG GWMAC Vlan = !MCLAG VSI profile ---> Drop 4. DA=System MAC Vlan = L2 vlan ---> Drop +### 2.4.1 Debug commands to check the drop rule counters are below +debugsh>show system internal sai vlan ifp drop counters +debugsh>clear system internal vlan ifp drop counters +root@sonic:/home/admin# debugsh +=========================================================================================== + Welcome to SONiC Debug shell. + +WARNING: The commands available in this shell are for internal use only. Use with caution. +=========================================================================================== +debugsh> show system internal sai vlan ifp drop counters + +| Drop entry | Counters | +------------------------------------------------------------ +| DA = SystemMAC, Vlan = SAG-VSI | 0 | +| DA = SystemMAC, Vlan = MclagGW-VSI | 0 | +| MyStationHit, Vlan = L2-VSI | 20 | +| DA = MclagGW-MAC, Vlan = Non-MclagGw-VSI | 0 | +debugsh> +debugsh> +debugsh> clear system internal vlan ifp drop counters +debugsh> +debugsh> +debugsh> show system internal sai vlan ifp drop counters + +| Drop entry | Counters | +------------------------------------------------------------ +| DA = SystemMAC, Vlan = SAG-VSI | 0 | +| DA = SystemMAC, Vlan = MclagGW-VSI | 0 | +| MyStationHit, Vlan = L2-VSI | 0 | +| DA = MclagGW-MAC, Vlan = Non-MclagGw-VSI | 0 | +debugsh> + # 4 Warm Boot Support -Upgrade scenario will be supported. It will be make before break. On upgrade to higher version, in TH/TH2 devices one entry per each of the MACs will be programmed first, followed by deletion of all the individual entries programmed per each SVI interface. Incase of TD3 device, l2 table entry will be added first and then corresponding myStationTcam entry will be deleted. +Upgrade scenario will be supported. It will be make before break. On upgrade to higher version, in TH/TH2/TH3 devices one entry per each of the MACs will be programmed first, followed by deletion of all the individual entries programmed per each SVI interface. Incase of TD3 device, l2 table entry will be added first and then corresponding myStationTcam entry will be deleted. -# 5 Unit Test +# 5 Unit Tests Configure setup with base MCLAG config. Configure SAG MAC,MCLAG Gateway MAC. Configure SAG SVI interface, MCLAG Gateway MAC SVI interface and plain L3 SVI interface. - -Testcase | Validate | Remarks | ----------|----------------------------------------------------|--------------------------| - 1. | Send l3 pkt with DA= System MAC vlan=SAG vlan | Packet should be dropped | - 2. | Send l3 pkt with DA= System MAC vlan=MCLAG vlan | Packet should be dropped | - 3. | Send l3 pkt with DA= System MAC vlan=L2 vlan | Packet should be dropped | - 4. | Send l3 pkt with DA= System MAC vlan=L3 vlan | Packet should be routed | - 5. | Send l3 pkt with DA= SAG MAC vlan=MCLAG vlan | Packet should be dropped | - 6. | Send l3 pkt with DA= SAG MAC vlan=SAG vlan | Packet should be routed | - 7. | Send l3 pkt with DA= SAG MAC vlan=L3 vlan | Packet should be dropped | - 8. | Send l3 pkt with DA= SAG MAC vlan=L2 vlan | Packet should be dropped | - 9. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L2 vlan | Packet should be dropped | - 10. | Send l3 pkt with DA= MCLAG Gw MAC vlan=SAG vlan | Packet should be dropped | - 11. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L3 vlan | Packet should be dropped | - 12. | Send l3 pkt with DA= MCLAG Gw MAC vlan=MCLAG vlan | Packet should be routed | - +Testcase | Validate | Remarks | +---------|----------------------------------------------------|--------------------------------| + 1. | Send l3 pkt with DA= System MAC vlan=SAG vlan | Packet should be dropped | + 2. | Send l3 pkt with DA= System MAC vlan=MCLAG vlan | Packet should be dropped | + 3. | Send l3 pkt with DA= System MAC vlan=L2 vlan | Packet should be dropped | + 4. | Send l3 pkt with DA= System MAC vlan=L3 vlan | Packet should be routed | + 5. | Send l3 pkt with DA= SAG MAC vlan=MCLAG vlan | Packet should be dropped | + 6. | Send l3 pkt with DA= SAG MAC vlan=SAG vlan | Packet should be routed | + 7. | Send l3 pkt with DA= SAG MAC vlan=L3 vlan | Packet should be dropped | + 8. | Send l3 pkt with DA= SAG MAC vlan=L2 vlan | Packet should be dropped | + 9. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L2 vlan | Packet should be dropped | + 10. | Send l3 pkt with DA= MCLAG Gw MAC vlan=SAG vlan | Packet should be dropped | + 11. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L3 vlan | Packet should be dropped | + 12. | Send l3 pkt with DA= MCLAG Gw MAC vlan=MCLAG vlan | Packet should be routed | + 13. | Configure 4k Mclag Svi.Perform reboot. | | + | Apply 4k static arp/nd.Start traffic | Validate no traffic disruption | + 14. | Configure 4k Mclag Svi.Perform fast reboot. | | + | Apply 4k static arp/nd.Start traffic | Validate no traffic disruption | + 15. | Configure 4k Mclag Svi.Perform warm reboot. | | + | Apply 4k static arp/nd.Start traffic | Validate no traffic disruption | # 6 Internal Design Information NA