[teamd]: wait for swss db flush done before starting teamd container

[jipanyang]

Signed-off-by: Jipan Yang [email protected]

- What I did
This a try to fix #2606

- How I did it

- How to verify it

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

[jleveque]

One problem I foresee here is that SWSS_DB_FLUSH_DONE never gets set to 0 or deleted. Take for instance the new logic I implemented in the 201803 branch (and will soon implement in the master/201811 branches, as well). If a critical process crashes in the swss container, the container will exit causing the swss service to restart itself and its dependent services. At this point, swss restarts and subsequently restarts teamd. It's possible that teamd would check the value before the databases get flushed, yet SWSS_DB_FLUSH_DONE would still be set to 1, causing teamd to start before the databases are flushed. How can we reliably set this value to 0 or delete the key when applicable? maybe deleting the key when the swss service stops is enough?

[jipanyang]

Yes, at least SWSS_DB_FLUSH_DONE should be set as 0 upon swss service stop.

[yxieca]

Did you test this change with system warm reboot?

[jipanyang]

@yxieca I tested with cold reboot and teamd docker warm restart, but not system warm reboot. Did you see any issue?

[yxieca]

Not really.

I my test. I think you change is equal to add service teamd after syncd. I don't see behavior change from one to another. Can you test that change?

diff --git a/files/build_templates/teamd.service.j2 b/files/build_templates/teamd.service.j2
index 792b824..bde55c6 100644
--- a/files/build_templates/teamd.service.j2
+++ b/files/build_templates/teamd.service.j2
@@ -2,6 +2,7 @@
Description=TEAMD container
Requires=updategraph.service
After=updategraph.service
+After=syncd.service
Before=ntp-config.service

[Service]

[jipanyang]

Yes, with recent systemd service change, this should work too.

The difference might be bootup speed (for both cold boot and warm boot), but I don't have any concrete data.

[jipanyang]

Wait for swss to finish db flushing:

root@vlab-01:/tmp# ls -l
total 8
-rw-r--r-- 1 root  root    0 Mar  1 08:19 dump.rdb
-rw-r--r-- 1 root  root  824 Mar  1 08:19 swss-syncd-debug.log
-rw-r--r-- 1 root  root    0 Mar  1 08:19 swss-syncd-lock
-rw-r--r-- 1 admin admin 638 Mar  1 08:19 teamd_debug.log
root@vlab-01:/tmp# cat teamd_debug.log 
Fri Mar 1 08:19:24 UTC 2019 - Start waiting for swss
Fri Mar 1 08:19:28 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:33 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:35 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:37 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:38 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:40 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:42 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:43 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:45 UTC 2019 - Swss db flush done

SWSS_DB_FLUSH_DONE flag and correct mtu for lag in appDB.

root@vlab-01:/tmp# redis-cli -p 6379
127.0.0.1:6379> keys SWS*
1) "SWSS_DB_FLUSH_DONE"
127.0.0.1:6379> get SWSS_DB_FLUSH_DONE
"1"
127.0.0.1:6379> exit
root@vlab-01:/tmp# redis-cli -p 6379
127.0.0.1:6379> hgetall "LAG_TABLE:PortChannel0001"
1) "mtu"
2) "9100"
3) "admin_status"
4) "up"
5) "oper_status"
6) "up"

root@vlab-01:/home/admin# systemctl stop swss
then
systemctl start swss

127.0.0.1:6379> 
127.0.0.1:6379> get SWSS_DB_FLUSH_DONE
"0"
127.0.0.1:6379> get SWSS_DB_FLUSH_DONE
"1"
127.0.0.1:6379> 

mtu gone after swss restart.

127.0.0.1:6379> hgetall "LAG_TABLE:PortChannel0001"
1) "admin_status"
2) "up"
3) "oper_status"
4) "up"
127.0.0.1:6379> 

root@vlab-01:/home/admin# systemctl restart teamd

mtu push back.

127.0.0.1:6379> hgetall "LAG_TABLE:PortChannel0001"
1) "admin_status"
2) "up"
3) "oper_status"
4) "up"
5) "mtu"
6) "9100"

root@vlab-01:/home/admin# cat /tmp/teamd_debug.log 
Fri Mar 1 08:19:24 UTC 2019 - Start waiting for swss
Fri Mar 1 08:19:28 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:33 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:35 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:37 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:38 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:40 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:42 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:43 UTC 2019 - Wait 1 second for SWSS_DB_FLUSH_DONE
Fri Mar 1 08:19:45 UTC 2019 - Swss db flush done
Fri Mar 1 08:36:05 UTC 2019 - Start waiting for swss
Fri Mar 1 08:36:06 UTC 2019 - Swss db flush done

[jleveque]

Retest this please

[jipanyang]

Close it in favor of #2724