[Nokia-BRCM-DNX]: CLI show dropcounter counts - retains the stats after clearing

[amitpawar12]

Description

Although the device says that drop counter IS NOT supported on this device, I see that drop counters do increment.

admin@ixre-egl-board71:~$ show dropcounters capabilities
Current device does not support drop counters

After clearing the drop counter stats for a given ASIC, if the stats of other ASIC is cleared, the stats for the first ASIC are brought back.

Steps to reproduce the issue:

1. Find the drop counter stats for a given ASIC. These drops are brought about by oversubscribing lossy priority traffic.

admin@ixre-egl-board71:~$ date;sudo ip netns exec asic0 show dropcounters counts | grep -E "IFACE|Ethernet0 |Ethernet8 "
Thu 08 Aug 2024 07:30:36 PM UTC
        IFACE    STATE    RX_ERR     RX_DROPS    TX_ERR    TX_DROPS
    Ethernet0        U         0  13707840988         0           0
    Ethernet8        U         0  13703485502         0           0

2. Clear the drop counter stats for ASIC0 and check the stats.

admin@ixre-egl-board71:~$ date;sudo ip netns exec asic0 sonic-clear dropcounters
Thu 08 Aug 2024 07:30:53 PM UTC
Cleared drop counters

admin@ixre-egl-board71:~$ date;sudo ip netns exec asic0 show dropcounters counts | grep -E "IFACE|Ethernet0 |Ethernet8 "
Thu 08 Aug 2024 07:30:56 PM UTC
        IFACE    STATE    RX_ERR    RX_DROPS    TX_ERR    TX_DROPS
    Ethernet0        U         0           0         0           0
    Ethernet8        U         0           0         0           0

3. Clear the stats for ASIC1

admin@ixre-egl-board71:~$ date;sudo ip netns exec asic1 sonic-clear dropcounters
Thu 08 Aug 2024 07:31:05 PM UTC
Cleared drop counters

4. Check the stats for ASIC0

admin@ixre-egl-board71:~$ date;sudo ip netns exec asic1 sonic-clear dropcounters
Thu 08 Aug 2024 07:31:31 PM UTC
Cleared drop counters

admin@ixre-egl-board71:~$ date;sudo ip netns exec asic0 show dropcounters counts | grep -E "IFACE|Ethernet0 |Ethernet8 "
Thu 08 Aug 2024 07:31:59 PM UTC
        IFACE    STATE    RX_ERR     RX_DROPS    TX_ERR    TX_DROPS
    Ethernet0        U         0  13707840988         0           0
    Ethernet8        U         0  13703485502         0           0

5. The drop counter stats are identical and there was no traffic flowing through the box.
6. The same is also true if the ASICs are swapped.

Describe the results you received:

On clearing the stats for ASIC0, and checking the stats, the stats reset. However, if we clear the stats for other ASIC, the stats on first ASIC come back again.

Describe the results you expected:

Clearing of the stats on one ASIC, should not have any impact on the stats on different ASIC.

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[zjswhhh]

@saksarav-nokia - can you please triage internally first? Thanks!

[saksarav-nokia]

It is the same root cause as #19779

This seems to be very fundamental issue when the user issues sonic-clear and then show for the commands in multi-asic switch. If you issue sonic-clear for asic0, then asic1 and then do show for asic0 OR son-clear for asic1, sonic-clear for asic0 and then do show for asic1, you will see this issue. The reason is that the soni-clear command creates the files in /tmp/cache/dropstat each time you run clear command and overwrites the asic 0's, when you clear for asic 0 first and then asic 0 (or asic1 file depending on the order you issue sonic-clear). Then when you do show for asic0, it reads this file created for asic 1 and it does look up for asic 0's port counter( which does n't exist) and subtract this from counter vlaue from COUNTERS_DB. So it prints the value from COUNTER_DB.
Since this is very basic issue and applies for all the clear commands in multi-asic , need to discuss with sonic community for the fix.

[saksarav-nokia]

If we add the multi-asic support to sonic-clear command for pg drop counter, then the cache file can be created with prefixing ns and show command can read the corresponding history from cache.

[vmittal-msft]

@kenneth-arista can you/team please check this ? This is related to multi asic support for Qos commands.

[kenneth-arista]

Copying my comments issue 19779:

The solution is to not use ip netns exec before running CLI command related to "priority-group drop counters" because native multi-ASIC support has been added recently to these family of commands. Instead use the built-in -n argument. See sonic-net/sonic-utilities#3058 for further details.

The reason is that ip netns exec ... limits the Linux network namespace, which conflicts with the default use of the multi_asic decorator for adding multi-asic support to existing commands. I believe historically folks have been using ip netns exec as a hack to get around old commands that haven't been taught about multi-asic. But we are putting effort into enhancing all Qos commands to natively support multi-asic. Tracking issue: #15148.

[amitpawar12]

@kenneth-arista:

Does the drop-counter CLI support namespace tag or not?

I went ahead and tried to check if drop counter have namespace tag. I did not find it. Checked with 2205 build.

=======================================
-- Clearing the counters
=======================================
admin@ixre-egl-board71:~$ sudo sonic-clear dropcounters -h
Usage: sonic-clear dropcounters [OPTIONS]

  Clear drop counters

Options:
  -h, -?, --help  Show this message and exit.

admin@ixre-egl-board71:~$ sonic-clear dropcounters -h
Usage: sonic-clear dropcounters [OPTIONS]

  Clear drop counters

Options:
  -h, -?, --help  Show this message and exit.

=======================================
-- Check the counters in sudo mode.
=======================================   
admin@ixre-egl-board71:~$ sudo show dropcounters counts -h
Usage: show dropcounters counts [OPTIONS]

  Show drop counts

Options:
  -g, --group TEXT
  -t, --counter_type TEXT
  --verbose                Enable verbose output
  -h, -?, --help           Show this message and exit.
admin@ixre-egl-board71:~$ sudo show dropcounters counts -n asic0
Usage: show dropcounters counts [OPTIONS]
Try "show dropcounters counts -h" for help.

Error: no such option: -n
admin@ixre-egl-board71:~$ sudo show dropcounters -n asic0 counts
Usage: show dropcounters [OPTIONS] COMMAND [ARGS]...
Try "show dropcounters -h" for help.

Error: no such option: -n


=======================================
-- CLI in non-sudo mode.
=======================================
admin@ixre-egl-board71:~$ show dropcounters counts --namespace asic0
Usage: show dropcounters counts [OPTIONS]
Try "show dropcounters counts -h" for help.

Error: no such option: --namespace

Please let me know.

Thanks,
-A

[kenneth-arista]

The changes to support multi-asics merged to master and not 202205. 202205 is virtually frozen and not accepting any feature changes.

[kenneth-arista]

Please close this issue as I don't have permission to do so.

[saksarav-nokia]

@kenneth-arista , Did Arista add namespace support for sonic-clear commands also. How do you clear the counters?. I don't see -n support in 202405