diff --git a/doc/sag/images/sag-db-arch.png b/doc/sag/images/sag-db-arch.png deleted file mode 100644 index 6005995eef..0000000000 Binary files a/doc/sag/images/sag-db-arch.png and /dev/null differ diff --git a/doc/sag/images/sag-flow.png b/doc/sag/images/sag-flow.png new file mode 100644 index 0000000000..eaf02cac89 Binary files /dev/null and b/doc/sag/images/sag-flow.png differ diff --git a/doc/sag/images/sag-flows.png b/doc/sag/images/sag-flows.png deleted file mode 100644 index 0913a5d525..0000000000 Binary files a/doc/sag/images/sag-flows.png and /dev/null differ diff --git a/doc/sag/sag-HLD.md b/doc/sag/sag-HLD.md index 9e7c29304b..59eca923b6 100644 --- a/doc/sag/sag-HLD.md +++ b/doc/sag/sag-HLD.md @@ -4,182 +4,321 @@ - [SAG high level design for SONiC](#sag-high-level-design-for-sonic) - [Table of Contents](#table-of-contents) - [Revision History](#revision-history) +- [Scope](#scope) - [Abbreviations](#abbreviations) -- [Requirements Overview](#requirements-overview) -- [Modules Design](#modules-design) - - [Config DB](#config-db) - - [SAG Global Table](#sag-global-table) - - [SAG Interface Table](#sag-interface-table) - - [Config DB Schemas](#config-db-schemas) - - [APP DB](#config-db) - - [Interface Table](#interface-table) - - [APP DB Schemas](#app-db-schemas) - - [Orchestration Agent](#orchestration-agent) - - [SagOrch](#sagorch) - - [SagMgrD](#sagmgrd) - - [IntfsOrch](#intfsorch) - - [SAI](#sai) -- [Flows](#flows) -- [CLI](#cli) -- [Test](#test) - - [Functional Test Cases](#functional-test-cases) +- [Overview](#overview) +- [Requirements](#requirements) +- [Architecture Design](#architecture-design) +- [High-Level Design](#high-level-design) + - [sonic-swss-common](#sonic-swss-common) + - [sonic-swss](#sonic-swss) + - [sonic-utilities](#sonic-utilities) + - [DB](#db) + - [SAI API](#sai-api) +- [Configuration and management](#configuration-and-management) +- [CLI/YANG model Enhancements](#cliyang-model-enhancements) + - [CLI](#cli) + - [Yang model](#yang-model) + - [Config DB Enhancements](#config-db-enhancements) +- [Warmboot and Fastboot Design Impact](#warmboot-and-fastboot-design-impact) +- [Restrictions/Limitations](#restrictionslimitations) +- [Testing Requirements/Design](#testing-requirementsdesign) + - [System Test Cases](#system-test-cases) # Revision History | Rev | Date | Author | Change Description | | :---: | :--------- | :---------: | --------------------------------- | | 0.1 | 08/13/2021 | Jimi Chen | Initial version | +| 0.2 | 08/30/2021 | Jimi Chen | Update community review comments | + +# Scope +This HLD extends SONiC to support static anycast gateway # Abbreviations | Abbreviations | Description | |---------------|------------------------| | SAG | Static Anycast Gateway | -# Requirements Overview +# Overview Anycast gateway feature for EVPN/VXLAN is a default gateway addressing mechanism that enables the use of the same gateway IP address across all the leaf switches that are part of a VXLAN network. This ensures that every leaf switch can function as the default gateway for the workloads directly connected to it. The feature failitates flexible workload placement, host mobility and optimal traffic forwarding across the VXLAN fabric. -This feature should be used together with VxLAN EVPN. -It should always to configure VxLAN to make SAG work. -SAG announces its IP and MAC only towards the host facing Ethernet ports but not towards the fabric (Overlay). -As a result, the IP/MAC for the SAG is only seen and reachable from the host facing side and never from the fabric (Overlay) facing side. - -The following is the requirements: -* Support one global virtual MAC address on SAG interface -* Support multiple IPv4/IPv6 address on SAG interface -* Support global IPv4/IPv6 enable knob for SAG address - -# Modules Design -## Config DB -### SAG Global Table -``` -SAG_GLOBAL|IP - "gwmac": {{mac_address}} - "IPv4": {{enable}} - "IPv6": {{enable}} -``` -### SAG Interface Table -``` -SAG|{{vlan_intf_name}}|IPv4 - "gwip@": {{ipv4_list}} -SAG|{{vlan_intf_name}}|IPv6 - "gwip@": {{ipv6_ilst}} -``` - -### Config DB Schemas -``` -; Defines schema for SAG configuration attributes -key = SAG_GLOBAL|IP ; SAG global configuration -; field = value -gwmac = mac_address ; global anycast gateway MAC address -IPv4 = enable/disable ; enable anycast gateway IPv4 address or not -IPv6 = enable/disable ; enable anycast gateway IPv6 address or not - -``` -## APP DB -### Interface Table -``` -INTF_TABLE:{{intf_name}} - "mac_addr": {{mac_address}} - -INTF_TABLE:{{intf_name}}:{{ip_prefix}} - "scope": {{addr_scope} - "family": {{ip_family}} -``` -### APP DB Schemas -``` -; Defines schema for SAG configuration attributes -key = INTF_TABLE:sag_name ; SAG interface -; field = value -mac_addr = mac_address ; global anycast mac MAC address - -key = INTF_TABLE:sag_name:ip_prefix ; SAG IP address -; field = value -scope = global/local/host ; IP address scope -family = IPv4/IPv6 ; IP address family -``` - -## Orchestration Agent -Following orchagents shall be modified. Flow diagrams are captured in a later section. - -![SAG arch](./images/sag-db-arch.png) - -### SagMgrD -SagMgrD creates the macvlan kernel interfaces on top of the vlan interface. -SagMgrD waits for macvlan interface creation update in STATE_DB and updates the APP_DB INTF_TABLE with the macvlan interface name. -SagMgrD creates IP address on the SAG interface. -SagMgrD gets the INTERFACE Table config to create route entry in SAI from APP_DB updates - -### IntfsOrch -Add SagOrch as a member of IntfsOrch. IntfsOrch creates Router Interfaces based on interface table (INTF_TABLE). For SAG usecase, IntfOrch calls SagOrch API to handle router interface creation. - -### PortOrch -Add SAG related port interface and corresponding attributes. -Also handle the port referenece count for the SAG port interface. - -# SAI -There are no changes to SAI headers/implementation to support this feature. Existing SAI router interface APIs are used to handle creating SAG interface. SAG needs to configure SAI_ROUTER_INTERFACE_ATTR_IS_VIRTUAL to true when creating router interface. - -# Flows -The overall data flow diagram is captured below for all TABLE updates. - - ![SAG flows](./images/sag-flows.png) - -# CLI -Commands summary - - - config sag mac_address add - - Add global mac address for sag - - config sag mac_address del - - Remove global mac address for sag - - config sag ipv4 enable - - Enable the ipv4 knob for sag - - config sag ipv4 disable - - Disable the ipv4 knob for sag - - config sag ipv6 enable - - Enable the ipv6 knob for sag - - config sag ipv6 disable - - Disable the ipv6 knob for sag - - config interface sag ip add - - Create an sag interface and set ip on vlan interface - - config interface sag ip del - - Delete the ip on sag interface - - show sag - - Show global mac for sag +This feature is used together with VxLAN EVPN usually, it can also be used individually. +In common use case, SAG announces its IP and MAC towards the host facing Ethernet ports but not towards the fabric. +As a result, the IP/MAC for the SAG is only seen and reachable from the host facing side and never from the fabric facing side. + +# Requirements +User can configure global virtual MAC address to apply on VLAN interfaces. +Also provides a per VLAN interface's knob to determine if VLAN interface use this global virtual MAC address or system CPU MAC address. + + +# Architecture Design +The overall SONiC architecture will not be changed and no new sub-modules will be introduced. + +# High-Level Design +## sonic-swss-common +sonic-swss-common's schema will be updated to include **SAG** define for the new table name. + +## sonic-swss +sonic-swss's Intfs Orch and Intf Mgr will be updated to include a new handler for static anycast gateway configuration, and also handles new enable/disable field for VLAN interface in original handler. + +The IP address configured on the VLAN interface is not changed. +The VLAN interface will use static anycast gateway MAC address to replace CPU MAC address if static anycast gateway MAC address is specified and it's enabled on the VLAN interface. +The enable/disable knob on the VLAN interface can let user to determine to use CPU MAC or static anycast gateway MAC address. + +The static anycast gateway on the VLAN interface will be disabled by default. + +The following diagram describes the sequence between DBs and modules. +![SAG flow](./images/sag-flow.png) +## sonic-utilities +sonic-utilities will be updated to offer the following CLI commands for the user +* Configure global static anycast gateway MAC address +* Enable/Disable to use static anycast gateway MAC address on the VLAN interface + +## DB +The CONFIG_DB will be updated to include a new **SAG**. +This will have the following format: +``` +### SAG + ; SAG global configuration + key = "SAG|GLOBAL" + ; field = value + gwmac = mac_address + +Example: +127.0.0.1:6379[4]> hgetall "SAG|GLOBAL" +1) "gwmac" +2) "00:11:22:33:44:0f" +``` + +It also add new field **static_anycast_gateway** to include in existing **VLAN_INTERFACE** table +This will have the following format: + +``` +Example: +127.0.0.1:6379[4]> hgetall "VLAN_INTERFACE|Vlan201" +1) "vrf_name" +2) "Vrf1" +3) "static_anycast_gateway" +4) "false" +``` +# SAI API +There are no changes to SAI headers/implementation to support this feature. + +# Configuration and management +# CLI/YANG model Enhancements + +## CLI +**Add static anycast gateway MAC address** +``` +config static-anycast-gateway mac_address add +``` +* mac_address: mandatory, the virtual MAC address of static anycast gateway + +The format is in the following + +``` +admin@edgecore:~$ sudo config static-anycast-gateway mac_address add -h +Usage: config static-anycast-gateway mac_address add [OPTIONS] + + Add global static-anycast-gateway mac address + +Options: + -?, -h, --help Show this message and exit. + +Example: +admin@edgecore:~$ sudo config static-anycast-gateway mac_address add 00:11:22:33:44:0f +``` + +**Delete static anycast gateway MAC address** + +``` +config static-anycast-gateway mac_address del +``` +* mac_address: mandatory, the virtual MAC address of static anycast gateway + +The format is in the following + +``` +admin@edgecore:~$ sudo config static-anycast-gateway mac_address del -h +Usage: config static-anycast-gateway mac_address add [OPTIONS] + + Delete global static-anycast-gateway mac address + +Options: + -?, -h, --help Show this message and exit. + +Example: +admin@edgecore:~$ sudo config static-anycast-gateway mac_address del 00:11:22:33:44:0f +``` + +**Enable static anycast gateway on VLAN interface** ``` +config vlan static-anycast-gateway add +``` +* vlan_id: mandatory, the VLAN ID of the interface + +The format is in the following + +``` +admin@edgecore:~$ sudo config vlan static-anycast-gateway add -h +Usage: config vlan static-anycast-gateway add [OPTIONS] + + Enable static-anycast-gateway on VLAN interface + +Options: + -?, -h, --help Show this message and exit. + +Example: +admin@edgecore:~$ sudo config vlan static-anycast-gateway add 100 +``` + +**Disable static anycast gateway on VLAN interface** +``` +config vlan static-anycast-gateway del +``` +* vlan_id: mandatory, the VLAN ID of the interface + +The format is in the following + +``` +admin@edgecore:~$ sudo config vlan static-anycast-gateway del -h +Usage: config vlan static-anycast-gateway del [OPTIONS] + + Disable static-anycast-gateway on VLAN interface + +Options: + -?, -h, --help Show this message and exit. + +Example: +admin@edgecore:~$ sudo config vlan static-anycast-gateway del 100 +``` +**Display static anycast gateway configuration** +``` +show static-anycast-gateway +``` + +The display format is in the following +``` +admin@edgecore:~$ show sag Static Anycast Gateway Information -MacAddress IPv4 IPv6 ------------------ ------ ------ -00:11:22:33:44:55 enable enable +MacAddress +----------------- +00:11:22:33:44:55 +``` + +**Display enable/disable on the VLAN interface** +Add extra column to display the SAG enable/disable status +``` +show vlan brief ``` - - show sag ip - - Show ipv4 on sag interface +The display format is in the following ``` -Vlan Interface Name IPv4 address/mask ---------------------- ------------------- -Vlan100 1.1.1.254/24 +admin@edgecore:~$ show vlan brief ++-----------+----------------+---------------+----------------+-----------------------+-------------+------------------------+ +| VLAN ID | IP Address | Ports | Port Tagging | DHCP Helper Address | Proxy ARP | Static Anycast Gateway | ++===========+================+===============+================+=======================+=============+========================+ +| 10 | 192.10.10.1/24 | PortChannel02 | tagged | | disabled | enabled | ++-----------+----------------+---------------+----------------+-----------------------+-------------+------------------------+ +| 200 | | Ethernet60 | untagged | | disabled | disabled | +| | | PortChannel01 | tagged | | | | +| | | PortChannel02 | tagged | | | | ++-----------+----------------+---------------+----------------+-----------------------+-------------+------------------------+ ``` - - show sag ipv6 - - Show ipv6 on sag interface + + +**IP address configuration** +Use the the following existed IPv4/IPv6 command to configure and display on the VLAN interface, no new arguments or commands added. +* config interface ip add +* config interface ip del +* show ip interfaces +* show ipv6 interfaces + +## Yang model +**sonic-static-anycast-gateway.yang** +Add new yang model to describe static anycast gateway configuration +``` +container sonic-static-anycast-gateway { + container SAG { + description "static anycast gateway configuration"; + + leaf gwmac { + type yang:mac-address; + } + } +} ``` -Vlan Interface Name IPv6 address/mask ---------------------- ------------------- -Vlan100 2001:1000::FE/64 +**sonic-vlan.yang** +Add **static-anycast-gateway** field under **VLAN_INTERFACE_LIST**, this field controls enable static-anycast-gateway on the VLAN interface or not. ``` +... +list VLAN_INTERFACE_LIST { + + description "VLAN INTERFACE part of config_db.json with vrf"; + + key "name"; + + leaf name { + type leafref { + path /vlan:sonic-vlan/vlan:VLAN/vlan:VLAN_LIST/vlan:name; + } + } + + leaf vrf_name { + type leafref{ + path "/vrf:sonic-vrf/vrf:VRF/vrf:VRF_LIST/vrf:name"; + } + } + + leaf nat_zone { + description "NAT Zone for the vlan interface"; + type uint8 { + range "0..3" { + error-message "Invalid nat zone for the vlan interface."; + error-app-tag nat-zone-invalid; + } + } + default "0"; + } + + leaf staic-anycast-gateway { + description "Enable/Disable static anycast gateway" + type boolean; + default false; + } +} +... + +``` + +## Config DB Enhancements +The relevant changes have been described in HLD's DB sub-section. + +# Warmboot and Fastboot Design Impact +No impacted by the changes. +# Restrictions/Limitations +For the router interfaces resources, it's the same as other IP address configure on the interface, and it can be monitored by CRM. +Due to lack of router interfaces monitoring in current CRM implementation. +It should be supported in other enhancement. -# Test -## Functional Test Cases -1. Verify that SAG interface can be created with virtual MAC address in kernel. -2. Verify that SAG router interface is programmed to switch ASIC. -3. Verify that IPv4 address can be created on SAG interface in kernel. -4. Verify that IPv4 IP2ME route is programmed to switch ASIC. -5. Verify that packets destined to SAG IPv4 address are trapped to CPU - using ping utility. -6. Verify that IPv6 address can be created on SAG interface in kernel. -7. Verify that IPv6 IP2ME route is programmed to switch ASIC. -8. Verify that packets destined to SAG IPv6 address are trapeed to CPU - using ping utility. -10. Verify that neighbor learns the SAG virutial MAC address - using ping/ARP/ND utility. -11. Verify that SAG global IPv4/IPv6 enable/disable command can create/destroy all SAG IP address -using ping utility. -12. Verify that change SAG virtual MAC address when IP is configured first - using ping/ARP/ND utility. \ No newline at end of file +# Testing Requirements/Design +## System Test Cases +1. When global SAG MAC is configured and SAG is enabled on the VLAN interface + * Verify that VLAN interface can be created with SAG MAC address in kernel. + * Verify that VLAN router interface with SAG MAC address is programmed to switch ASIC. + * Verify that IPv4 address can be created on the VLAN interface in kernel. + * Verify that IPv4 IP2ME route is programmed to switch ASIC. + * Verify that packets destined to SAG IPv4 address are trapped to CPU + * Verify that IPv6 address can be created on the VLAN interface in kernel. + * Verify that IPv6 IP2ME route is programmed to switch ASIC. + * Verify that packets destined to SAG IPv6 address are trapped to CPU + * Verify that host learns the SAG virutial MAC address + * Verify that switch learns neighbor on the VLAN interface +2. When global SAG MAC is configured and SAG is enabled on the VLAN interface, disable SAG on the VLAN interface + * Verify the VLAN interface's MAC change to CPU MAC address in kernel + * Verify the VLAN router interface with CPU MAC address is programmed to switch ASIC + * Verify that neighbor learns the CPU MAC address \ No newline at end of file diff --git a/doc/sag/sag-flow.wsd b/doc/sag/sag-flow.wsd new file mode 100644 index 0000000000..c36572dc56 --- /dev/null +++ b/doc/sag/sag-flow.wsd @@ -0,0 +1,35 @@ +@startuml +boundary "CLI" as UI +box "Orchagent" #LightGrey + control IntfMgrD + control IntfsOrch +end box +participant kernel +box "DB" #LightBlue + database CONFIG_DB + database APPL_DB + database ASIC_DB +end box + +UI -> CONFIG_DB: config SAG global MAC address +IntfMgrD <-- CONFIG_DB +UI -> CONFIG_DB: enable SAG on the VLAN interface +IntfMgrD <-- CONFIG_DB +UI -> CONFIG_DB: config interface IP address on the VLAN interface +IntfMgrD <-- CONFIG_DB +IntfMgrD -> kernel: create VLAN interface and change to SAG MAC address +IntfMgrD -> APPL_DB: create INTF_TABLE entry with SAG MAC address and IP address entry +IntfsOrch <-- APPL_DB +IntfsOrch -> ASIC_DB: create router interface and change to SAG MAC address +IntfMgrD -> APPL_DB: config VLAN interface IP +IntfsOrch <-- APPL_DB +IntfsOrch -> ASIC_DB: add IP2ME route and net route +group Disable SAG on the VLAN interface + UI -> CONFIG_DB: disable SAG on the VLAN interface + IntfMgrD <-- CONFIG_DB + IntfMgrD -> kernel: change VLAN interface's MAC address to CPU MAC address + IntfMgrD -> APPL_DB: Modify INTF_TABLE entry from SAG MAC address to CPU MAC address + IntfsOrch <-- APPL_DB + IntfsOrch -> ASIC_DB: change router interface's MAC address to CPU MAC address +end +@enduml \ No newline at end of file