Skip to content

Commit

Permalink
Merge pull request sonic-net#260 from BRCM-SONIC/amulya_sviscale
Browse files Browse the repository at this point in the history 
Amulya sviscale
  • Loading branch information
@NarendraBandarupalli
NarendraBandarupalli authored Aug 18, 2021
2 parents a8fc0c2 + 502a2d5 commit b988188
Showing 1 changed file with 83 additions and 37 deletions.
120 changes: 83 additions & 37 deletions L3/L3_performance_and_scaling_enhancements/L3_Vlan_interface_scaling.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,17 @@ SONIC currently supports a maximum of 1k L3 Vlan interfaces on Broadcom hardware

## 1.1 Functional Requirements
Scaling improvements
|--------------------------------------|------|-----|-----|-----|
| MAC | TD3 | TD4 | TH | TH2 |
|--------------------------------------|------|-----|-----|-----|
|System MAC configured SVI | 4k | 4k | 4k | 4k |
|MCLAG Gateway MAC with same ip SVI | 4k | 4k | 4k | 4k |
|MCLAG with unique ip SVI | 4k | 4k | 1k | 1k |
|SAG MAC configured SVI | 4k | 4k | 1k | 1k |
|VRRP MAC SVI | 4k | 4k | 1k | 1k |
|--------------------------------------|------|-----|-----|-----|
|--------------------------------------|------|-----|-----|-----|-----|
| MAC | TD3 | TD4 | TH | TH2 | TH3 |
|--------------------------------------|------|-----|-----|-----|-----|
|System MAC configured SVI | 4k | 4k | 4k | 4k | 4k |
|MCLAG Gateway MAC with same ip SVI | 4k | 4k | 4k | 4k | 4k |
|MCLAG with unique ip SVI | 4k | 4k | 1k | 1k | 1k |
|MCLAG with peer gateway enabled SVI | 4k | 4k | 1k | 1k | 1k |
|SAG MAC configured SVI | 4k | 4k | 1k | 1k | 1k |
|VRRP MAC SVI | 4k | 4k | 1k | 1k | 1k |
|--------------------------------------|------|-----|-----|-----|-----|


## 1.2 Configuration and Management Requirements
No new configuration or show commands introduced.
Expand All @@ -60,17 +62,21 @@ No new configuration or show commands introduced.
Covered in Functional requirements

## 1.4 Warm Boot Requirements
Upgrade to higher version should take care of installing one entry per MAC and delete the individual station tcam entries per SVI on TH/TH2 platforms

Upgrade to higher version should take care of installing one entry per MAC and delete the individual station tcam entries per SVI on TH/TH2/TH3 platforms
Upgrade to higher version should take care of moving to l2 table from myStationTcam for all SVI interfaces on TD3 platform.

# 2 Design

## 2.1 Use L2 table for inner mac termination for TD3 & TD4
TD4 already uses L2 table for inner mac termination. No change for TD4 platform.
In TD3, for all SVI interfaces, L2 table will be programmed instead of MyStationTcam.
Entry will be added with respective mac,vfi and flags as BCM_L2_STATIC. One entry per each SVI will be programmed.If 4k VRRP SVI interfaces are configured ,8k L2 table entries will be consumed.Rest of the cases,4k L2 tables entries will be consumed.

Entry will be added with respective mac,vfi and flags as BCM_L2_STATIC. One entry per each SVI will be programmed.
Note :- If 4k VRRP SVI interfaces are configured ,8k L2 table entries will be consumed.Rest of the cases 4k L2 table entries are consumed.

## 2.2 MyStationTcam lookup key modification
This is applicable for only TH/TH2/TH3 platforms.
Use only MAC as the key for MyStationTcam entries, this will give way to use only one MyStationTcam entry for all SVI interfaces, i.e, we could be able to scale to 4K L3 Interfaces. This change is for the following 2 MACs only.
1. System MAC
2. McLAG Gateway MAC
Expand All @@ -79,11 +85,12 @@ For SVI interfaces the lookup keys is only (DMAC)
For Phy interfaces / RouterPorts the lookup key is retained the same [Port Id + DMAC]
For LAG router interfaces the lookup key is retained the same [Trunk Id + DMAC]

## 2.2.1 MyStationTcam programming

### 2.2.1 MyStationTcam programming
A new data structure is added and is managed by the sai data manager.This data structure handles the list of Vlan interfaces associated with the given mac and the corresponding MyStationTcam hardware id.
When the first SVI interface configured with any of the above macs the corresponding tcam entry gets added. When the last SVI interface associated with the mac is deleted , the corresponding tcam entry will be deleted.

## 2.2.2 Routing behaviour differences
### 2.2.2 Routing behaviour differences
With the key change to only (MAC) from (MAC,Vlan), if the SVI interfaces are configured each with one type of MAC. Station tcam will look like below now.
For example
1. System MAC - 3c2c992ed875
Expand All @@ -99,7 +106,9 @@ If a packet comes with (DA=MCLAG Gateway MAC, Vlan = !MCLAG Vlan). It will be su
If a packet comes with (DA=Any of the above two macs, Vlan=L2). It will be subjected to routing.
Inorder to address these cases, added below IFP drop rules during system bringup. Also, counters attached to each of the drop rules.

### 2.3 VSI Profile


## 2.3 VSI Profile
For each SVI interface there is a VSI profile maintained to classify the SVI interfaces.
2 VSI profile bits are consumed. One to identify the SAG MAC vlan and another to identify the MCLAG Gateway MAC vlan.

Expand All @@ -109,36 +118,73 @@ For each SVI interface there is a VSI profile maintained to classify the SVI int
#define _BRCM_SAI_VSI_ROUTER_MAC_L3_INTF (1<<3)------->MCLAG Gateway MAC VSI
#define _BRCM_SAI_MAX_VSI_PROFILE_BITS_LENGTH 4

### 2.4 IFP Drop rules
Below drop rules will be programmed only in TH/TH2 devices.
1. DA=System MAC Vlan = SAG VSI profile ---> Drop
2. DA=System MAC Vlan = MCLAG VSI profile ---> Drop
3. DA=MCLAG GWMAC Vlan = !MCLAG VSI profile ---> Drop

## 2.4 IFP Drop rules
Below drop rules will be programmed only in TH/TH2/TH3 devices.
1. DA=System MAC Vlan = SAG VSI profile ---> Drop
2. DA=System MAC Vlan = MCLAG VSI profile ---> Drop
3. DA=MCLAG GWMAC Vlan = !MCLAG VSI profile ---> Drop
4. DA=System MAC Vlan = L2 vlan ---> Drop

### 2.4.1 Debug commands to check the drop rule counters are below
debugsh>show system internal sai vlan ifp drop counters
debugsh>clear system internal vlan ifp drop counters
root@sonic:/home/admin# debugsh
===========================================================================================
Welcome to SONiC Debug shell.

WARNING: The commands available in this shell are for internal use only. Use with caution.
===========================================================================================
debugsh> show system internal sai vlan ifp drop counters

| Drop entry | Counters |
------------------------------------------------------------
| DA = SystemMAC, Vlan = SAG-VSI | 0 |
| DA = SystemMAC, Vlan = MclagGW-VSI | 0 |
| MyStationHit, Vlan = L2-VSI | 20 |
| DA = MclagGW-MAC, Vlan = Non-MclagGw-VSI | 0 |
debugsh>
debugsh>
debugsh> clear system internal vlan ifp drop counters
debugsh>
debugsh>
debugsh> show system internal sai vlan ifp drop counters

| Drop entry | Counters |
------------------------------------------------------------
| DA = SystemMAC, Vlan = SAG-VSI | 0 |
| DA = SystemMAC, Vlan = MclagGW-VSI | 0 |
| MyStationHit, Vlan = L2-VSI | 0 |
| DA = MclagGW-MAC, Vlan = Non-MclagGw-VSI | 0 |
debugsh>

# 4 Warm Boot Support
Upgrade scenario will be supported. It will be make before break. On upgrade to higher version, in TH/TH2 devices one entry per each of the MACs will be programmed first, followed by deletion of all the individual entries programmed per each SVI interface. Incase of TD3 device, l2 table entry will be added first and then corresponding myStationTcam entry will be deleted.
Upgrade scenario will be supported. It will be make before break. On upgrade to higher version, in TH/TH2/TH3 devices one entry per each of the MACs will be programmed first, followed by deletion of all the individual entries programmed per each SVI interface. Incase of TD3 device, l2 table entry will be added first and then corresponding myStationTcam entry will be deleted.

# 5 Unit Test
# 5 Unit Tests
Configure setup with base MCLAG config.
Configure SAG MAC,MCLAG Gateway MAC.
Configure SAG SVI interface, MCLAG Gateway MAC SVI interface and plain L3 SVI interface.

Testcase | Validate | Remarks |
---------|----------------------------------------------------|--------------------------|
1. | Send l3 pkt with DA= System MAC vlan=SAG vlan | Packet should be dropped |
2. | Send l3 pkt with DA= System MAC vlan=MCLAG vlan | Packet should be dropped |
3. | Send l3 pkt with DA= System MAC vlan=L2 vlan | Packet should be dropped |
4. | Send l3 pkt with DA= System MAC vlan=L3 vlan | Packet should be routed |
5. | Send l3 pkt with DA= SAG MAC vlan=MCLAG vlan | Packet should be dropped |
6. | Send l3 pkt with DA= SAG MAC vlan=SAG vlan | Packet should be routed |
7. | Send l3 pkt with DA= SAG MAC vlan=L3 vlan | Packet should be dropped |
8. | Send l3 pkt with DA= SAG MAC vlan=L2 vlan | Packet should be dropped |
9. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L2 vlan | Packet should be dropped |
10. | Send l3 pkt with DA= MCLAG Gw MAC vlan=SAG vlan | Packet should be dropped |
11. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L3 vlan | Packet should be dropped |
12. | Send l3 pkt with DA= MCLAG Gw MAC vlan=MCLAG vlan | Packet should be routed |

Testcase | Validate | Remarks |
---------|----------------------------------------------------|--------------------------------|
1. | Send l3 pkt with DA= System MAC vlan=SAG vlan | Packet should be dropped |
2. | Send l3 pkt with DA= System MAC vlan=MCLAG vlan | Packet should be dropped |
3. | Send l3 pkt with DA= System MAC vlan=L2 vlan | Packet should be dropped |
4. | Send l3 pkt with DA= System MAC vlan=L3 vlan | Packet should be routed |
5. | Send l3 pkt with DA= SAG MAC vlan=MCLAG vlan | Packet should be dropped |
6. | Send l3 pkt with DA= SAG MAC vlan=SAG vlan | Packet should be routed |
7. | Send l3 pkt with DA= SAG MAC vlan=L3 vlan | Packet should be dropped |
8. | Send l3 pkt with DA= SAG MAC vlan=L2 vlan | Packet should be dropped |
9. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L2 vlan | Packet should be dropped |
10. | Send l3 pkt with DA= MCLAG Gw MAC vlan=SAG vlan | Packet should be dropped |
11. | Send l3 pkt with DA= MCLAG Gw MAC vlan=L3 vlan | Packet should be dropped |
12. | Send l3 pkt with DA= MCLAG Gw MAC vlan=MCLAG vlan | Packet should be routed |
13. | Configure 4k Mclag Svi.Perform reboot. | |
| Apply 4k static arp/nd.Start traffic | Validate no traffic disruption |
14. | Configure 4k Mclag Svi.Perform fast reboot. | |
| Apply 4k static arp/nd.Start traffic | Validate no traffic disruption |
15. | Configure 4k Mclag Svi.Perform warm reboot. | |
| Apply 4k static arp/nd.Start traffic | Validate no traffic disruption |

# 6 Internal Design Information
NA
Expand Down

0 comments on commit b988188

Please sign in to comment.