[TH3] Skipp Control Plane Assist on WARM Reboot for TH3 HWSKUs #1861

gechiang · 2021-10-06T23:30:52Z

What I did

Added Code to Skip over Control Plane Assist (CPA) executed during Warm reboot operation for those HW that is not capable of handling CPA such as TH3 based HW SKUs.

How to verify it

Execute testcase "arp/test_wr_arp.py" to ensure the operation to program VxLAN Tunnel and Mirror ACLs are all skipped.
This is skipped during setup as well as finalizer time.
Syslog captured that shows the skipping is made when HW SKU of TH3 is detected:

admin@z9332f:~$ show platform summary
Platform: x86_64-dellemc_z9332f_d1508-r0
HwSKU: DellEMC-Z9332f-M-O16C64
ASIC: broadcom
ASIC Count: 1
admin@SONiC:~$ show logging

...
Oct  7 23:12:47.298940 z9332f NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.0.23, 98:03:9b:03:22:15 on Vlan1000
Oct  7 23:12:47.300419 z9332f NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.0.23 on Vlan1000
Oct  7 23:12:47.501984 z9332f NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.0.24, 98:03:9b:03:22:16 on Vlan1000
Oct  7 23:12:47.503181 z9332f NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.0.24 on Vlan1000
Oct  7 23:12:47.706207 z9332f NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.0.25, 98:03:9b:03:22:17 on Vlan1000
Oct  7 23:12:47.707813 z9332f NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.0.25 on Vlan1000
Oct  7 23:12:47.900366 z9332f NOTICE admin: DellEMC-Z9332f-M-O16C64 Not capable to support CPA. Skipping gracefully ...
Oct  7 23:12:47.902892 z9332f NOTICE admin: Pausing orchagent ...
Oct  7 23:12:47.909771 z9332f NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 192.168.0.26, 98:03:9b:03:22:18 on Vlan1000
Oct  7 23:12:47.911524 z9332f NOTICE swss#orchagent: :- addNextHop: Created next hop 192.168.0.26 on Vlan1000
Oct  7 23:12:48.042208 z9332f NOTICE swss#orchagent_restart_check: :- main: Wait time for response from orchagent set to 2000 milliseconds
Oct  7 23:12:48.042467 z9332f NOTICE swss#orchagent_restart_check: :- main: Number of retries for the request to orchagent is set to 5
Oct  7 23:12:48.047630 z9332f INFO swss#orchagent_restart_check: :- subscribe: subscribed to RESTARTCHECKREPLY
Oct  7 23:12:48.047630 z9332f NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 0
Oct  7 23:12:48.047841 z9332f NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent
Oct  7 23:12:48.047887 z9332f NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Oct  7 23:12:48.047931 z9332f NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: READY
Oct  7 23:12:48.048088 z9332f NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK success, orchagent is frozen and ready for warm restart
...
Oct  7 23:15:57.249871 z9332f NOTICE bgp#fpmsyncd: :- main: Warm-Restart timer expired.
Oct  7 23:15:57.249871 z9332f NOTICE bgp#fpmsyncd: :- reconcile: Warm-Restart: Initiating reconciliation process for bgp application.
Oct  7 23:15:57.287145 z9332f NOTICE bgp#fpmsyncd: :- reconcile: Warm-Restart reconciliation: updating entry fe80::/64 { nexthop: :: | ifname: Ethernet104 }
Oct  7 23:15:57.368196 z9332f NOTICE bgp#fpmsyncd: :- setWarmStartState: bgp warm start state changed to reconciled
Oct  7 23:15:57.368196 z9332f NOTICE bgp#fpmsyncd: :- reconcile: Warm-Restart: Concluded reconciliation process for bgp application.
Oct  7 23:15:57.368196 z9332f NOTICE bgp#fpmsyncd: :- main: Warm-Restart reconciliation processed.
Oct  7 23:16:01.960005 z9332f NOTICE root: WARMBOOT_FINALIZER : Tearing down control plane assistant ...
Oct  7 23:16:05.173116 z9332f NOTICE root: WARMBOOT_FINALIZER : Save in-memory database after warm reboot ...
Oct  7 23:16:05.498247 z9332f INFO finalize-warmboot.sh[6750]: Running command: /usr/local/bin/sonic-cfggen -d --print-data > /etc/sonic/config_db.json
Oct  7 23:16:05.906786 z9332f NOTICE root: WARMBOOT_FINALIZER : Finalizing warmboot...
Oct  7 23:16:06.218787 z9332f INFO systemd[1]: warmboot-finalizer.service: Succeeded.
Oct  7 23:16:06.218964 z9332f INFO systemd[1]: Finished Monitor warm recovery and disable warmboot when done.
...

Also added new test cases to specifically test out the new helper function I added.

prsunny

Can the neighbor_advertiser be skipped while invoking warmboot instead of modifying the script?

gechiang · 2021-10-06T23:59:52Z

Can the neighbor_advertiser be skipped while invoking warmboot instead of modifying the script?

Reason I am doing it inside the script is because the setup for warmboot and finalizer are two seperate files and I think it is easier to have this inside the script which is called by both cases instead of spread out to multiple files.

prsunny · 2021-10-07T00:06:36Z

Can the neighbor_advertiser be skipped while invoking warmboot instead of modifying the script?

Reason I am doing it inside the script is because the setup for warmboot and finalizer are two seperate files and I think it is easier to have this inside the script which is called by both cases instead of spread out to multiple files.

If you fix this in setup for warmboot, isn't finalizer a no-op? As in, even if it try to remove the entries in finalizer, it will not trigger anything since its non-existent, right?

gechiang · 2021-10-07T00:13:55Z

Can the neighbor_advertiser be skipped while invoking warmboot instead of modifying the script?

Reason I am doing it inside the script is because the setup for warmboot and finalizer are two seperate files and I think it is easier to have this inside the script which is called by both cases instead of spread out to multiple files.

If you fix this in setup for warmboot, isn't finalizer a no-op? As in, even if it try to remove the entries in finalizer, it will not trigger anything since its non-existent, right?

Interesting idea. Let me give it a try just on the setup for warmboot to see if that works. If so, I will create a new PULL request.
Thanks!

This reverts commit 40cb537.

…ript to achieve skip CPA for TH3

gechiang · 2021-10-07T23:35:48Z

@prsunny I have modified the code per your comment. Please help review the new changes.
Thanks!

vaibhavhd

Minor suggestions. LGTM otherwise.

vaibhavhd · 2021-10-07T23:51:52Z

scripts/fast-reboot

@@ -487,6 +495,8 @@ case "$REBOOT_TYPE" in
        ;;
 esac

+HWSKU=$(show platform summary --json | python -c 'import sys, json; print(json.load(sys.stdin)["hwsku"])')


This can be moved inside setup_reboot_variables function.
https://github.com/Azure/sonic-utilities/blob/4af3e129c600fcc226a6a87d58a5ec8503a3c65e/scripts/fast-reboot#L308

Agreed. Changed just now.
Thanks!

vaibhavhd · 2021-10-08T00:00:42Z

scripts/fast-reboot

-        debug "Setting up control plane assistant: ${ASSISTANT_IP_LIST} ..."
-        ${ASSISTANT_SCRIPT} -s ${ASSISTANT_IP_LIST} -m set
+        # TH3 HW is not capable of VxLAN programming thus skipping TH3 platforms
+        if [[ "$HWSKU" != "DellEMC-Z9332f-M-O16C64" && "$HWSKU" != "DellEMC-Z9332f-M-O16C64-lab" ]]; then


$HWSKU can be replaced by ${HWSKU} for consistency and probably bash safety too.

Agreed. Changed just now.
Thanks!

gechiang · 2021-10-08T01:31:21Z

/azp run

azure-pipelines · 2021-10-08T01:31:31Z

Azure Pipelines successfully started running 1 pipeline(s).

vaibhavhd · 2021-10-08T18:05:41Z

The failures seem to be coming due to a sonic-swss dvs switch creation error.
Yesterday this was supposedly fixed on sonic-swss - sonic-net/sonic-swss#1949

Buildimage update is in progress - sonic-net/sonic-buildimage#8915

I suggest to retry after buildimage PR is merged.

gechiang · 2021-10-08T18:53:14Z

The failures seem to be coming due to a sonic-swss dvs switch creation error. Yesterday this was supposedly fixed on sonic-swss - Azure/sonic-swss#1949

Buildimage update is in progress - Azure/sonic-buildimage#8915

I suggest to retry after buildimage PR is merged.

Thanks! I was about to ask how to address the tests failure which are unrelated to my changes...
Will wait for that swss PR pulled into sonic-buildimage before attempting restart azp run.

gechiang · 2021-10-11T04:25:42Z

/azp run

azure-pipelines · 2021-10-11T04:25:54Z

Azure Pipelines successfully started running 1 pipeline(s).

* [TH3] Skip Control Plane Assist on WARM Reboot for TH3 HWSKUs

8768089 (HEAD -> 202012, origin/202012) Remove exec from platform_reboot_plugin call to handle any hang issue. (sonic-net#1879) ae5d90c Validate input of ```config mirror_session add``` (sonic-net#1825) 44d3a3b [show][config] fix the muxcable commands for interface naming mode (sonic-net#1862) 0a4933e [TH3] Skipp Control Plane Assist on WARM Reboot for TH3 HWSKUs (sonic-net#1861) Signed-off-by: vaibhav-dahiya <[email protected]>

[TH3] Skipp Control Plane Assist on WARM Reboot for TH3 HWSKUs

40cb537

gechiang requested review from vaibhavhd, lguohan, prsunny, yxieca and rlhui October 6, 2021 23:54

gechiang added the Request for 202012 Branch label Oct 6, 2021

prsunny reviewed Oct 6, 2021

View reviewed changes

gechiang added 2 commits October 7, 2021 22:46

Revert "[TH3] Skipp Control Plane Assist on WARM Reboot for TH3 HWSKUs"

549b070

This reverts commit 40cb537.

Modify fast-reboot script instead of the control plane assist(CPA) sc…

4af3e12

…ript to achieve skip CPA for TH3

gechiang requested a review from prsunny October 7, 2021 23:34

vaibhavhd previously approved these changes Oct 8, 2021

View reviewed changes

Moved HWSKU variable init to inside setup_reboot_variables()

771f501

gechiang dismissed vaibhavhd’s stale review via 771f501 October 8, 2021 00:28

gechiang requested a review from vaibhavhd October 8, 2021 00:30

Merge remote-tracking branch 'upstream/master'

c1ff268

vaibhavhd approved these changes Oct 8, 2021

View reviewed changes

yxieca approved these changes Oct 11, 2021

View reviewed changes

gechiang merged commit 1edf934 into sonic-net:master Oct 11, 2021

qiluo-msft pushed a commit that referenced this pull request Oct 12, 2021

[TH3] Skipp Control Plane Assist on WARM Reboot for TH3 HWSKUs (#1861)

0a4933e

* [TH3] Skip Control Plane Assist on WARM Reboot for TH3 HWSKUs

qiluo-msft added the Included in 202012 Branch label Oct 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TH3] Skipp Control Plane Assist on WARM Reboot for TH3 HWSKUs #1861

[TH3] Skipp Control Plane Assist on WARM Reboot for TH3 HWSKUs #1861

gechiang commented Oct 6, 2021 •

edited

Loading

prsunny left a comment

gechiang commented Oct 6, 2021

prsunny commented Oct 7, 2021

gechiang commented Oct 7, 2021

gechiang commented Oct 7, 2021

vaibhavhd left a comment

vaibhavhd Oct 7, 2021

gechiang Oct 8, 2021

vaibhavhd Oct 8, 2021

gechiang Oct 8, 2021

gechiang commented Oct 8, 2021

azure-pipelines bot commented Oct 8, 2021

vaibhavhd commented Oct 8, 2021

gechiang commented Oct 8, 2021

gechiang commented Oct 11, 2021

azure-pipelines bot commented Oct 11, 2021

[TH3] Skipp Control Plane Assist on WARM Reboot for TH3 HWSKUs #1861

[TH3] Skipp Control Plane Assist on WARM Reboot for TH3 HWSKUs #1861

Conversation

gechiang commented Oct 6, 2021 • edited Loading

What I did

How to verify it

prsunny left a comment

Choose a reason for hiding this comment

gechiang commented Oct 6, 2021

prsunny commented Oct 7, 2021

gechiang commented Oct 7, 2021

gechiang commented Oct 7, 2021

vaibhavhd left a comment

Choose a reason for hiding this comment

vaibhavhd Oct 7, 2021

Choose a reason for hiding this comment

gechiang Oct 8, 2021

Choose a reason for hiding this comment

vaibhavhd Oct 8, 2021

Choose a reason for hiding this comment

gechiang Oct 8, 2021

Choose a reason for hiding this comment

gechiang commented Oct 8, 2021

azure-pipelines bot commented Oct 8, 2021

vaibhavhd commented Oct 8, 2021

gechiang commented Oct 8, 2021

gechiang commented Oct 11, 2021

azure-pipelines bot commented Oct 11, 2021

gechiang commented Oct 6, 2021 •

edited

Loading