sonic-db-cli -n <asic-ns> CHASSIS_APP_DB EVAL fails intermittently after config reload or load-minigraph #17945
Comments
|
@qiluo-msft, below is the question from Shakthi Is there any difference between these two calls ? This is the place where we refer to, currently we use sonic-db-cli : sonic-buildimage/files/scripts/swss.sh Line 143 in 4af43dc |
|
@saksarav-nokia just noticed you see this error "Unable to connect to redis: Cannot assign requested address" -- which means some issue in local connectivity, local interface not available at this time Can you check if the network-config service is still setting up local interface ? |
|
@judyjoseph , if it is connectivity issue, how come the redis-cli command does not fail. It worked fine with 300 config reload. |
|
@judyjoseph , We realized hostname-config.service is restarted every time the config reload or load-minigraph is done and this service updates the /etc/hosts and this is conflicting with the swss service chassis-db-cleanup. We didn't see the issue with adding "After=hostname-config.service" in [email protected] and we will run full OC this weekend with the change and update the PR if everything goes fine. mv -f /etc/hosts /etc/hosts.old |
|
admin@lc4: |
…re valid in CONIFG_DB before starting chassis-db-cleanup (#17962) This PR fixes the issue reported in Issu #17945 We noticed that chassis db clean up is skipped sometimes when the CHASSIS_APP_DB PING fails. Also if host_name and asic_name are not written to CONIG_DB, it could pass the empty strings to CHASSIS_APP_DB EVAL commands. The service hostname-config.service is restarted whenever the config-reload or load-minigraph is done and this services renames the file /etc/hosts to updates it with the new file. This interferes with [email protected] and when swss.sh script CHASSIS_APP_DPP when the /etc/hosts file is renamed, the error "Unable to connect to redis: Cannot assign requested address" is seen and the CHASSIS_APP_DB EVAL command fails. This causes the chassis db entries not getting cleaned up and causes orchagent crash in remote LC's. --------- Signed-off-by: saksarav <[email protected]>
…re valid in CONIFG_DB before starting chassis-db-cleanup (sonic-net#17962) This PR fixes the issue reported in Issu sonic-net#17945 We noticed that chassis db clean up is skipped sometimes when the CHASSIS_APP_DB PING fails. Also if host_name and asic_name are not written to CONIG_DB, it could pass the empty strings to CHASSIS_APP_DB EVAL commands. The service hostname-config.service is restarted whenever the config-reload or load-minigraph is done and this services renames the file /etc/hosts to updates it with the new file. This interferes with [email protected] and when swss.sh script CHASSIS_APP_DPP when the /etc/hosts file is renamed, the error "Unable to connect to redis: Cannot assign requested address" is seen and the CHASSIS_APP_DB EVAL command fails. This causes the chassis db entries not getting cleaned up and causes orchagent crash in remote LC's. --------- Signed-off-by: saksarav <[email protected]>
…re valid in CONIFG_DB before starting chassis-db-cleanup (#17962) This PR fixes the issue reported in Issu #17945 We noticed that chassis db clean up is skipped sometimes when the CHASSIS_APP_DB PING fails. Also if host_name and asic_name are not written to CONIG_DB, it could pass the empty strings to CHASSIS_APP_DB EVAL commands. The service hostname-config.service is restarted whenever the config-reload or load-minigraph is done and this services renames the file /etc/hosts to updates it with the new file. This interferes with [email protected] and when swss.sh script CHASSIS_APP_DPP when the /etc/hosts file is renamed, the error "Unable to connect to redis: Cannot assign requested address" is seen and the CHASSIS_APP_DB EVAL command fails. This causes the chassis db entries not getting cleaned up and causes orchagent crash in remote LC's. --------- Signed-off-by: saksarav <[email protected]>
…re valid in CONIFG_DB before starting chassis-db-cleanup (sonic-net#17962) This PR fixes the issue reported in Issu sonic-net#17945 We noticed that chassis db clean up is skipped sometimes when the CHASSIS_APP_DB PING fails. Also if host_name and asic_name are not written to CONIG_DB, it could pass the empty strings to CHASSIS_APP_DB EVAL commands. The service hostname-config.service is restarted whenever the config-reload or load-minigraph is done and this services renames the file /etc/hosts to updates it with the new file. This interferes with [email protected] and when swss.sh script CHASSIS_APP_DPP when the /etc/hosts file is renamed, the error "Unable to connect to redis: Cannot assign requested address" is seen and the CHASSIS_APP_DB EVAL command fails. This causes the chassis db entries not getting cleaned up and causes orchagent crash in remote LC's. --------- Signed-off-by: saksarav <[email protected]>
Description
When the config reload or load-minigraph is done , swss dockers are restarted and when swss comes back up, swss.sh script cleans the SYSYTEM_NEIGH, SYSTEM_INTERFACE, SYSTEM_LAG_MEMBER_TABLE and SYSTEM_LAG_ID_TABLE entries added by this LC+asic from the CHASSIS_APP_DB table using sonic-db-cli -n CHASSIS_APP_DB EVAL command.
Even though the EVAL command is called after sonic-db-cli -n CHASSIS_APP_DB PING is successful, the EVAL command fails sometimes.
When we replaced the "sonic-db-cli -n CHASSIS_APP_DB EVAL" with "redis-cli -h 10.6.0.100 -p 6380 -n 12 EVAL", the issues are not seen.
Steps to reproduce the issue:
Describe the results you received:
Describe the results you expected:
No errors in syslog and no crash
Output of
show version:SONiC Software Version: SONiC.20220532.54
SONiC OS Version: 11
Distribution: Debian 11.8
Kernel: 5.10.0-23-2-amd64
Build commit: b9e6caad98
Build date: Tue Jan 9 00:13:06 UTC 2024
Built by: cloudtest@95bebd0dc000000
Output of
show techsupport:Additional information you deem important (e.g. issue happens only occasionally):
The text was updated successfully, but these errors were encountered: