From 71c16b766a9aafebf3b18c7551a881f0cbaa92fa Mon Sep 17 00:00:00 2001 From: Mykola Faryma Date: Mon, 26 Nov 2018 14:34:12 +0200 Subject: [PATCH 1/6] [doc] RIF counters design Signed-off-by: Mykola Faryma --- doc/RIF_counters.md | 233 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) create mode 100644 doc/RIF_counters.md diff --git a/doc/RIF_counters.md b/doc/RIF_counters.md new file mode 100644 index 0000000000..3539a7cbb9 --- /dev/null +++ b/doc/RIF_counters.md @@ -0,0 +1,233 @@ +# Router interface counters in SONiC +### Rev 0.1 + +# Table of Contents + * [List of Tables](#list-of-tables) + * [Revision](#revision) + * [About this Manual](#about-this-manual) + * [Scope](#scope) + * [Definitions/Abbreviation](#definitionsabbreviation) + + * [1 Requirements](#1-overview) + + * [2 CLI format](#2-cli-format) + + * [3 Modules Design](#3-modules-design) + * [3.1 Modules that need to be updated](#31-modules-that-need-to-be-updated) + * [3.1.1 Counter DB](#311-counter-db) + * [3.1.2 CLI](#312-cli) + * [3.1.3 Orchestration Agent](#313-orchestration-agent) + * [3.1.4 Flex Counter](#314-flex-counter) + * [3.1.5 SAI](#318-sai) + + * [4 Open questions](#4-open-questions) + +# List of Tables +* [Table 1: Revision](#revision) +* [Table 2: Abbreviations](#definitionsabbreviation) + +###### Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:------------------:|-----------------------------------| +| 0.1 | | Mykola Faryma | Initial version | + +# About this Manual +This document provides general information about RIF counter feature in SONiC. +# Scope +This document describes the high level design of the RIF counters. +# Definitions/Abbreviation +###### Table 2: Abbreviations +| Definitions/Abbreviation | Description | +|--------------------------|--------------------------------------------| +| RIF | Router interface | +| FC | Flex counter | +| API | Application Programmable Interface | +| SAI | Switch Abstraction Interface | + +# 1 Requirements + +Add new CLI commands to show L3 interfaces counters. L3 interface counters refers to router port as well as vlan interface. +The information should be: RX: OK, ERR in packets and bytes. TX: OK, ERR in packets and bytes + +# 2 CLI format +The CLI command format and example output + +``` +$ show interfaces counters rif [OPTOINS] [interface_name] + + Show interface counters + +Options: + + -p, --period TEXT + + --verbose Enable verbose output + + -?, -h, --help Show this message and exit. +``` + +The period option gives the ability to see the counters and RX/TX BPS, PPS and utilization rates. + +The rates are calculated the following way: + +BPS = (new_*_OCTETS - old_*_OCTETS) / delta, + +Where delta is the period specified. The BPS is printed in MB/s, KB/s or B/s depending on the value. + +PPS = (new_*_PACKETS - old_*_PACKETS) / delta + +To clear the counters, the following CLI is available: + +$ sonic-clear interface rifcounters [interface_name] + +We will support both clearing the counters for a single RIF as well as all at once. + +# Example output: + +If the interface name is not provided, all the RIFs counters will be printed: + +$ show interfaces counters rif +``` + IFACE RX_OK RX_BPS RX_PPS RX_ERR TX_OK TX_BPS Tx_PPS TX_ERR +----------- ------- -------- ---------------- ------- -------- --------- -------- +Ethernet0 2180 N/A N/A 0 44697 N/A N/A 0 +Ethernet4 47342 N/A N/A 0 8362 N/A N/A 0 +Ethernet8 2188 N/A N/A 0 9379 N/A N/A 0 +Ethernet12 46882 N/A N/A 0 44448 N/A N/A 0 + ... +Portchannel0002 2833 N/A N/A 0 0 N/A N/A 0 + ... +Vlan2 2833 N/A N/A 0 4170 N/A N/A 0 +``` + +Providing a RIF name will print the counters in a more debug-friendly manner. + +If the interface name does not exist in COUNTERS_RIF_NAME_MAP table, we'll output a warning message. + +$ show interfaces counters rif Portchannel0002 + +``` +Portchannel0002 +--------------- + RX: + 0 packets + 0 bytes + 0 error packets + 0 error bytes + + TX: + 0 packets + 0 bytes + 0 error packets + 0 error bytes +``` +# SAI counters definition: +``` + RX: + packets - SAI_ROUTER_INTERFACE_STAT_IN_PACKETS + bytes - SAI_ROUTER_INTERFACE_STAT_IN_OCTETS + error packets - SAI_ROUTER_INTERFACE_STAT_IN_ERROR_PACKETS + error bytes - SAI_ROUTER_INTERFACE_STAT_IN_ERROR_OCTETS + + TX: + packets - SAI_ROUTER_INTERFACE_STAT_OUT_PACKETS + bytes - SAI_ROUTER_INTERFACE_STAT_OUT_OCTETS + error packets - SAI_ROUTER_INTERFACE_STAT_OUT_ERROR_PACKETS + error bytes - SAI_ROUTER_INTERFACE_STAT_OUT_ERROR_OCTETS +``` + +# 3 Modules Design +## 3.1 Modules that need to be updated + +### 3.1.1 Counter DB + +#### The following new Queue counters should be available for each queue entry in the DB: +- "COUNTERS:rif_vid" + - SAI_ROUTER_INTERFACE_STAT_IN_PACKETS + - SAI_ROUTER_INTERFACE_STAT_IN_OCTETS + - SAI_ROUTER_INTERFACE_STAT_IN_ERROR_PACKETS + - SAI_ROUTER_INTERFACE_STAT_IN_ERROR_OCTETS + - SAI_ROUTER_INTERFACE_STAT_OUT_PACKETS + - SAI_ROUTER_INTERFACE_STAT_OUT_OCTETS + - SAI_ROUTER_INTERFACE_STAT_OUT_ERROR_PACKETS + - SAI_ROUTER_INTERFACE_STAT_OUT_ERROR_OCTETS + +#### Additionally a few mappings should be added: +- "COUNTERS_RIF_TYPE_MAP" - map RIF to its type (e.g. LAG, Vlan) +- "COUNTERS_RIF_NAME_MAP" - map RIF oid to its name + +### 3.1.2 CLI + +Add new show interfaces counters subcommand, add rifstat script, try to reuse portstat code. + +### 3.1.3 SWSS + +IntfsOrch should be updated: + - Add/remove RIFs to Flex counter + - implement the Counters Db maps generation + +### 3.1.4 Flex counter + +Add router interface stats support. Implement new FC group for RIFs. + +Let's set the default FC interval to 1 second, same as for port counters. + +### 3.1.5 SAI + +The sai APIs and calls are: + +sai_router_interface_api -> sai_get_router_interface_stats() + +``` +/** + * @brief Router interface counter IDs in sai_get_router_interface_stats() call + */ +typedef enum _sai_router_interface_stat_t +{ + /** Ingress byte stat count */ + SAI_ROUTER_INTERFACE_STAT_IN_OCTETS, + + /** Ingress packet stat count */ + SAI_ROUTER_INTERFACE_STAT_IN_PACKETS, + + /** Egress byte stat count */ + SAI_ROUTER_INTERFACE_STAT_OUT_OCTETS, + + /** Egress packet stat count */ + SAI_ROUTER_INTERFACE_STAT_OUT_PACKETS, + + /** Byte stat count for packets having errors on router ingress */ + SAI_ROUTER_INTERFACE_STAT_IN_ERROR_OCTETS, + + /** Packet stat count for packets having errors on router ingress */ + SAI_ROUTER_INTERFACE_STAT_IN_ERROR_PACKETS, + + /** Byte stat count for packets having errors on router egress */ + SAI_ROUTER_INTERFACE_STAT_OUT_ERROR_OCTETS, + + /** Packet stat count for packets having errors on router egress */ + SAI_ROUTER_INTERFACE_STAT_OUT_ERROR_PACKETS + +} sai_router_interface_stat_t; +``` + +### 3.1.5 SNMP + +RIF counters can be exposed vie RFC1213 interface MIB. + +They will differ from ports by the ifType and base index. + +IANAifType: + + 6 - ethernetCsmacd + + 136 - l3ipvlan + + 161 - ieee8023adLag + +Base index : TBD + +### 4 Open questions + +What should be the base index (SNMP) for Vlan Interfaces and Portchannels? + From 00453ad4be0574e8158e7671659b53ebb5870d0d Mon Sep 17 00:00:00 2001 From: Mykola Faryma Date: Thu, 29 Nov 2018 19:25:44 +0200 Subject: [PATCH 2/6] [doc] Update RIF_counters design Signed-off-by: Mykola Faryma --- doc/RIF_counters.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/doc/RIF_counters.md b/doc/RIF_counters.md index 3539a7cbb9..839a5d55ab 100644 --- a/doc/RIF_counters.md +++ b/doc/RIF_counters.md @@ -66,6 +66,10 @@ Options: -?, -h, --help Show this message and exit. ``` +Note: the 'show interfaces counters' has '-c, --clear' option. From the CLI point of view, this should be available from 'sonic-clear' CLI. (There already is 'sonic-clear interfaces counters') + +If we would like to allign the options, I suggest to remove the clear option from the 'show interfaces counters' command. + The period option gives the ability to see the counters and RX/TX BPS, PPS and utilization rates. The rates are calculated the following way: @@ -219,15 +223,24 @@ They will differ from ports by the ifType and base index. IANAifType: - 6 - ethernetCsmacd - - 136 - l3ipvlan - - 161 - ieee8023adLag +| Value | Type | +| --- | ------ | +| 6 | ethernetCsmacd | +| 136 | l3ipvlan | +| 161 | ieee8023adLag | Base index : TBD +Currently defined base indexes: + +| Type | Base index | +| --- | ------ | +| ethernet | 1 | +| portchannel | 1000 | +| mgmt_port | 10000 | + ### 4 Open questions -What should be the base index (SNMP) for Vlan Interfaces and Portchannels? +#### 4.1 What should be the base index (SNMP) for Vlan Interfaces? +#### 4.2 Allign the CLi options with 'show interfaces counters' --clear option. Remove existing clear option from the show command? From 64c626dd60a98356ef9ccd0f2396ff6a8fcd7b17 Mon Sep 17 00:00:00 2001 From: Mykola Faryma Date: Thu, 6 Dec 2018 14:18:28 +0200 Subject: [PATCH 3/6] [doc] Update RIF_counters design Signed-off-by: Mykola Faryma --- doc/RIF_counters.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/doc/RIF_counters.md b/doc/RIF_counters.md index 839a5d55ab..6dbcc07c77 100644 --- a/doc/RIF_counters.md +++ b/doc/RIF_counters.md @@ -234,13 +234,28 @@ Base index : TBD Currently defined base indexes: | Type | Base index | -| --- | ------ | +| ---- | ---------- | | ethernet | 1 | | portchannel | 1000 | | mgmt_port | 10000 | +Proposed SNMP OIDs to be used: + +| OID | SNMP counter | Description | SAI stat | +| --- | ------------ | ----------- | -------- | +| 1.3.6.1.2.1.2.2.1.10 | ifInOctets | The total number of octets received on the interface, including framing characters. | SAI_ROUTER_INTERFACE_STAT_IN_OCTETS | +| 1.3.6.1.2.1.2.2.1.11 | ifInUcastPkts | The number of packets, delivered by this sub-layer to a higher (sub-)layer, which were not addressed to a multicast or broadcast address at this sub-layer. | SAI_ROUTER_INTERFACE_STAT_IN_PACKETS | +| 1.3.6.1.2.1.2.2.1.14 | ifInErrors | For packet-oriented interfaces, the number of inbound packets that contained errors preventing them from being deliverable to a higher-layer protocol. | SAI_ROUTER_INTERFACE_STAT_IN_ERROR_PACKETS | +| 1.3.6.1.2.1.2.2.1.16 | ifOutOctets | The total number of octets transmitted out of the interface, including framing characters. | SAI_ROUTER_INTERFACE_STAT_OUT_OCTETS | +| 1.3.6.1.2.1.2.2.1.17 | ifOutUcastPkts | The total number of packets that higher-level protocols requested be transmitted, and which were not addressed to a multicast or broadcast address at this sub-layer, including those that were discarded or not sent. | SAI_ROUTER_INTERFACE_STAT_OUT_PACKETS | +| 1.3.6.1.2.1.2.2.1.20 | ifOutErrors| For packet-oriented interfaces, the number of outbound packets that could not be transmitted because of errors. For character-oriented or fixed-length interfaces, the number of outbound transmission units that could not be transmitted because of errors. | SAI_ROUTER_INTERFACE_STAT_OUT_ERROR_PACKETS | + +It looks like the SAI_ROUTER_INTERFACE_STAT_IN_ERROR_PACKETS and SAI_ROUTER_INTERFACE_STAT_OUT_ERROR_PACKETS include bad packets, error and discard. The MIB has separate OIDs - ifInDiscards, ifInErrors, ifOutDiscards, ifOutErrors. Not sure how to handle this. + ### 4 Open questions #### 4.1 What should be the base index (SNMP) for Vlan Interfaces? #### 4.2 Allign the CLi options with 'show interfaces counters' --clear option. Remove existing clear option from the show command? + +#### 4.3 SNMP support and OIDs to be used. From dae4e7fdeaf978fa5fa5962b352d8faa67310dd3 Mon Sep 17 00:00:00 2001 From: Mykola F <37578614+mykolaf@users.noreply.github.com> Date: Tue, 8 Jan 2019 16:27:24 +0200 Subject: [PATCH 4/6] Update RIF_counters.md --- doc/RIF_counters.md | 62 ++++++++++++++------------------------------- 1 file changed, 19 insertions(+), 43 deletions(-) diff --git a/doc/RIF_counters.md b/doc/RIF_counters.md index 6dbcc07c77..fb4babfbac 100644 --- a/doc/RIF_counters.md +++ b/doc/RIF_counters.md @@ -18,7 +18,7 @@ * [3.1.2 CLI](#312-cli) * [3.1.3 Orchestration Agent](#313-orchestration-agent) * [3.1.4 Flex Counter](#314-flex-counter) - * [3.1.5 SAI](#318-sai) + * [3.1.5 SAI](#318-telemetry) * [4 Open questions](#4-open-questions) @@ -66,11 +66,7 @@ Options: -?, -h, --help Show this message and exit. ``` -Note: the 'show interfaces counters' has '-c, --clear' option. From the CLI point of view, this should be available from 'sonic-clear' CLI. (There already is 'sonic-clear interfaces counters') - -If we would like to allign the options, I suggest to remove the clear option from the 'show interfaces counters' command. - -The period option gives the ability to see the counters and RX/TX BPS, PPS and utilization rates. +The period option gives the ability to see the counters and RX/TX BPS and PPS. The rates are calculated the following way: @@ -215,47 +211,27 @@ typedef enum _sai_router_interface_stat_t } sai_router_interface_stat_t; ``` -### 3.1.5 SNMP - -RIF counters can be exposed vie RFC1213 interface MIB. +### 3.1.5 Telemetry -They will differ from ports by the ifType and base index. - -IANAifType: - -| Value | Type | -| --- | ------ | -| 6 | ethernetCsmacd | -| 136 | l3ipvlan | -| 161 | ieee8023adLag | - -Base index : TBD +We can expose the rif counters via sonic telemetry. +The telemetry is able to access the data in SONiC databases according to the schema, +but for more simplicity virtual path can be used. Virtual path allows to querry for counters by interface name, instead of VID. -Currently defined base indexes: +Example virtual paths -| Type | Base index | -| ---- | ---------- | -| ethernet | 1 | -| portchannel | 1000 | -| mgmt_port | 10000 | - -Proposed SNMP OIDs to be used: - -| OID | SNMP counter | Description | SAI stat | -| --- | ------------ | ----------- | -------- | -| 1.3.6.1.2.1.2.2.1.10 | ifInOctets | The total number of octets received on the interface, including framing characters. | SAI_ROUTER_INTERFACE_STAT_IN_OCTETS | -| 1.3.6.1.2.1.2.2.1.11 | ifInUcastPkts | The number of packets, delivered by this sub-layer to a higher (sub-)layer, which were not addressed to a multicast or broadcast address at this sub-layer. | SAI_ROUTER_INTERFACE_STAT_IN_PACKETS | -| 1.3.6.1.2.1.2.2.1.14 | ifInErrors | For packet-oriented interfaces, the number of inbound packets that contained errors preventing them from being deliverable to a higher-layer protocol. | SAI_ROUTER_INTERFACE_STAT_IN_ERROR_PACKETS | -| 1.3.6.1.2.1.2.2.1.16 | ifOutOctets | The total number of octets transmitted out of the interface, including framing characters. | SAI_ROUTER_INTERFACE_STAT_OUT_OCTETS | -| 1.3.6.1.2.1.2.2.1.17 | ifOutUcastPkts | The total number of packets that higher-level protocols requested be transmitted, and which were not addressed to a multicast or broadcast address at this sub-layer, including those that were discarded or not sent. | SAI_ROUTER_INTERFACE_STAT_OUT_PACKETS | -| 1.3.6.1.2.1.2.2.1.20 | ifOutErrors| For packet-oriented interfaces, the number of outbound packets that could not be transmitted because of errors. For character-oriented or fixed-length interfaces, the number of outbound transmission units that could not be transmitted because of errors. | SAI_ROUTER_INTERFACE_STAT_OUT_ERROR_PACKETS | - -It looks like the SAI_ROUTER_INTERFACE_STAT_IN_ERROR_PACKETS and SAI_ROUTER_INTERFACE_STAT_OUT_ERROR_PACKETS include bad packets, error and discard. The MIB has separate OIDs - ifInDiscards, ifInErrors, ifOutDiscards, ifOutErrors. Not sure how to handle this. +| DB target | Virtual Path | Description | +| --- | --- | --- | +| COUNTERS_DB | "RIF_COUNTERS/" | All counters on all interfaces | +| COUNTERS_DB | "RIF_COUNTERS/*/" | One counter on all interfaces | +| COUNTERS_DB | "RIF_COUNTERS//" | One counter on one interface | ### 4 Open questions -#### 4.1 What should be the base index (SNMP) for Vlan Interfaces? +#### 4.1 SNMP support? -#### 4.2 Allign the CLi options with 'show interfaces counters' --clear option. Remove existing clear option from the show command? - -#### 4.3 SNMP support and OIDs to be used. + - is SNMP support needed? + - Which MIB to use? + +Inreface MIB (RFC1213) has some issues: + - already provides l2 counters for Ethernet*, adding l3 overlaps, no separate iftype + - Interface MIB has more specific OIDs. it contains OIDs for InUnicast / InMcast pkts; eroor, drop pkts, while rif counters only have InPkts, error packets. From dbaf949b5905d1fe5b06c7f95e5aaf0fdb063ada Mon Sep 17 00:00:00 2001 From: Mykola F <37578614+mykolaf@users.noreply.github.com> Date: Tue, 8 Jan 2019 16:31:16 +0200 Subject: [PATCH 5/6] Update RIF_counters.md --- doc/RIF_counters.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/RIF_counters.md b/doc/RIF_counters.md index fb4babfbac..c2714c9377 100644 --- a/doc/RIF_counters.md +++ b/doc/RIF_counters.md @@ -222,8 +222,8 @@ Example virtual paths | DB target | Virtual Path | Description | | --- | --- | --- | | COUNTERS_DB | "RIF_COUNTERS/" | All counters on all interfaces | -| COUNTERS_DB | "RIF_COUNTERS/*/" | One counter on all interfaces | -| COUNTERS_DB | "RIF_COUNTERS//" | One counter on one interface | +| COUNTERS_DB | "RIF_COUNTERS/``*``/````" | One counter on all interfaces | +| COUNTERS_DB | "RIF_COUNTERS/````/````" | One counter on one interface | ### 4 Open questions From 56443ca1e554b81189c630beb38758df94a495f0 Mon Sep 17 00:00:00 2001 From: Mykola F <37578614+mykolaf@users.noreply.github.com> Date: Wed, 9 Jan 2019 16:55:12 +0200 Subject: [PATCH 6/6] [rif_counters hld] remove snmp support from open questions Discussion on community meeting decided we don't need to expose rif counters via SNMP. If the need arises we can always add it in the future. --- doc/RIF_counters.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/doc/RIF_counters.md b/doc/RIF_counters.md index c2714c9377..d756721deb 100644 --- a/doc/RIF_counters.md +++ b/doc/RIF_counters.md @@ -226,12 +226,3 @@ Example virtual paths | COUNTERS_DB | "RIF_COUNTERS/````/````" | One counter on one interface | ### 4 Open questions - -#### 4.1 SNMP support? - - - is SNMP support needed? - - Which MIB to use? - -Inreface MIB (RFC1213) has some issues: - - already provides l2 counters for Ethernet*, adding l3 overlaps, no separate iftype - - Interface MIB has more specific OIDs. it contains OIDs for InUnicast / InMcast pkts; eroor, drop pkts, while rif counters only have InPkts, error packets.