Test case 2 of PFC watchdog against warm-reboot: sad path

[wendani]

PFC storm started and detected before warm-reboot
On-going storm on warm-reboot emission, and lasts past the warm-reboot finish
PFC storm stopped and restored after warm-reboot

Tested on regular pfc watchdog without break.

Infrastructure change:

loganalyzer.py & loganalyzer_analyze.yml:
Allow log analyzer to take a specified start marker

loganalyzer_init.yml:
Use lookup('pipe', 'date +%H:%M:%S') in place of ansible_date_time.time, which uses cached time for a certain period of time ansible/ansible#22561

loganalyzer_end.yml:
Dump only the current result and summary files for debugging and troubleshooting purpose
Add the capability to check if the number of exact matches is equal to the target number

Incremental commits on top of #825

Description of PR

Summary:
Fixes # (issue)

Type of change

• [] Bug fix
• [] Testbed and Framework(new/improvement)
• [] Test case(new/improvement)

Approach

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[wendani]

@stepanblyschak Let me know if the test case passes at your end.

[stepanblyschak]

@wendani sonic-net/sonic-swss#876 is missing in 201811, which is important to pass this test

[stepanblyschak]

@wendani Besides of missing commit in 201811, there is another issue with bufferorch and pfcwdorch dependency.
BufferOrch::doTask is executed on 1st restore iteration before PortsOrch and PfcWdOrch, since PortInitDone is not set yet it will retry on 2nd iteration after PfcWdOrch:;doTask is executed and will override zero buffer profiles set on stormed ports.
I assume simple swap the order of PortsOrch and BufferOrch in m_orchList will fix this issue

[wendani]

@stepanblyschak please use the 201811 image we shared with you. As said in sonic-net/sonic-swss#876, we have a patch that hardcoded the two lossless TCs, and it is in the image we shared with you #834 (comment)

[wendani]

PortsOrch will not proceed on a port if buffer profile is not configured on that port. https://github.com/Azure/sonic-swss/blob/master/orchagent/portsorch.cpp#L1694 This said, BufferOrch must be ahead of PortsOrch in initialization list. There is no question on this.

What I can envision is a transcient behavior using the zero buffer apporach that a storming port may first have a non-zero buffer profile set by BufferOrch after warm-reboot, and then have a zero buffer profile when PfcWdOrch kicks in to install the drop action.

#834 (comment)

[wendani]

PfcWdOrch will wait for PortsOrch to signal all ports ready https://github.com/Azure/sonic-swss/blob/master/orchagent/pfcwdorch.cpp#L57

I believe the dependency is well sorted at the orchagent level.

[wendani]

@stepanblyschak Other than the test failure, do you have objection to the test case itself?

[stepanblyschak]

@wendani
The order of Orch::doTask() execution is defined by m_orchList https://github.com/Azure/sonic-swss/blob/master/orchagent/orchdaemon.cpp#L189.
So the order is BufferOrch, PortsOrch, PfcWdSwOrch.
Consider the warm restoration procedure:

• BufferOrch
  Since host interfaces have not been created yet, PortsOrch::isPortInitDone() returns false, so BufferOrch will defer the processing on 2nd iteration
• PortsOrch
  m_initDone == false and m_pendingPortSet.empty() == true at this point;
  m_toSync contains the following [Ethernet0, ...., PortConfigDone, PortInitDone]
  PortsOrch processes m_toSync and when PortConfigDone is beeing processed it creates ports, creates host interfaces; right after that there is a condition if alias == "PortConfigDone" https://github.com/Azure/sonic-swss/blob/master/orchagent/portsorch.cpp#L1688 and continues the loop. Next is "PortInitDone" in m_toSync, so PortsOrch::doTask will simply return https://github.com/Azure/sonic-swss/blob/master/orchagent/portsorch.cpp#L1531.
  Note, at this stage m_initDone == true and m_pendingPortSet.empty() == true which makes PortsOrch::allPortsReady() return true.
• PfcWdSwOrch configures zero buffer
• BufferOrch overrides the buffer.

So seems that design of m_pendingPortSet has an issue that makes other orchs think that buffer configuration has been applied when it was not.
A simple chnage in m_orchList to swap BufferOrch and PortsOrch makes sense because PortsOrch will create ports, host interfaces at first iteration, BufferOrch will be able to process buffers since PortInitDone flag is set, then on second iteration PortsOrch will proceed with speed, fec, etc.
More preferable will be to change the m_pendingPortSet to be filled before PortInitDone is processed.

[stepanblyschak]

@wendani Regarding transcient state during warm restoration on Mellanox when non-zero profile is set on hw. Do you think it is an issue in production?

[stepanblyschak]

@stepanblyschak please use the 201811 image we shared with you. As said in Azure/sonic-swss#876, we have a patch that hardcoded the two lossless TCs, and it is in the image we shared with you #834 (comment)

Does it mean that sonic-net/sonic-swss#876 won't be cherry-picked to 201811?

[stepanblyschak]

Despite there is an issue with PFCWD zero buffer approach, the test looks good to me