Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dual tor: Too Many SWSS.REC Messages Between Orchagent And Xcvrd #6796

Closed
tahmed-dev opened this issue Feb 16, 2021 · 1 comment
Closed

dual tor: Too Many SWSS.REC Messages Between Orchagent And Xcvrd #6796

tahmed-dev opened this issue Feb 16, 2021 · 1 comment
Assignees
@prsunny @vdahiya12
Labels
Issue for 202012

Comments

@tahmed-dev
Copy link
Contributor

Description

Description

While Debugging issue #6761, lot of communication took place between orchagent and xcvrd even after the underlying issue was mitigated.

Steps to reproduce the issue:

  1. Load Gemini image after 1/26 on ToRs with complete set of muxcables
  2. Reboot the ToR or config reload
  3. Notice initialization loop in the syslog and in swss.rec
  4. Fix swss ipinip issue, restart swss.service and notice communication between xcvrd and orachagent

Describe the results you received:

Plenty of communication observed in the swss.rec that lasted ~30 min

Describe the results you expected:

swss.service restart should recover the system and there should no be so many messaging between orchagent and xcvrd.

Output of show version:

SONiC Software Version: SONiC.internal-20210205-0331.post-rebase
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: f0a52b5a6
Build date: Fri Feb  5 10:12:39 UTC 2021
Built by: sonicbld@new-worker-3

Platform: x86_64-arista_7050cx3_32s
HwSKU: Arista-7050CX3-32S-D48C8
ASIC: broadcom
ASIC Count: 1
usermod: Permission denied.
usermod: cannot lock /etc/passwd; try again later.
usermod: Permission denied.
usermod: cannot lock /etc/passwd; try again later.
Serial Number: JPW20281059
Uptime: 18:49:36 up 10 days, 16:13,  1 user,  load average: 3.65, 4.25, 4.52

Docker images:
usermod: Permission denied.
usermod: cannot lock /etc/passwd; try again later.
usermod: Permission denied.
usermod: cannot lock /etc/passwd; try again later.
REPOSITORY                 TAG                                  IMAGE ID            SIZE
docker-syncd-brcm          internal-20210205-0331.post-rebase   be1ba1ebad2a        679MB
docker-syncd-brcm          latest                               be1ba1ebad2a        679MB
docker-snmp                internal-20210205-0331.post-rebase   e65a59a0e040        437MB
docker-snmp                latest                               e65a59a0e040        437MB
docker-teamd               internal-20210205-0331.post-rebase   7ca9ceb6a00a        407MB
docker-teamd               latest                               7ca9ceb6a00a        407MB
docker-router-advertiser   internal-20210205-0331.post-rebase   88bfd121d86a        397MB
docker-router-advertiser   latest                               88bfd121d86a        397MB
docker-platform-monitor    internal-20210205-0331.post-rebase   1ddce122a308        604MB
docker-platform-monitor    latest                               1ddce122a308        604MB
docker-lldp                internal-20210205-0331.post-rebase   bcc0f8bb5606        437MB
docker-lldp                latest                               bcc0f8bb5606        437MB
docker-dhcp-relay          internal-20210205-0331.post-rebase   22a0d820d12e        404MB
docker-dhcp-relay          latest                               22a0d820d12e        404MB
docker-database            internal-20210205-0331.post-rebase   83e8d19b2dac        397MB
docker-database            latest                               83e8d19b2dac        397MB
docker-orchagent           internal-20210205-0331.post-rebase   9dd87855c799        426MB
docker-orchagent           latest                               9dd87855c799        426MB
docker-macsec              internal-20210205-0331.post-rebase   7846322c296a        410MB
docker-macsec              latest                               7846322c296a        410MB
docker-sonic-telemetry     internal-20210205-0331.post-rebase   bc965837d672        471MB
docker-sonic-telemetry     latest                               bc965837d672        471MB
docker-mux                 internal-20210205-0331.post-rebase   ff18b4f11faf        451MB
docker-mux                 latest                               ff18b4f11faf        451MB
docker-fpm-frr             internal-20210205-0331.post-rebase   f2d8f8333aaf        425MB
docker-fpm-frr             latest                               f2d8f8333aaf        425MB
docker-acms                internal-20210205-0331.post-rebase   b2fef5f0d520        197MB
docker-acms                latest                               b2fef5f0d520        197MB
k8s.gcr.io/pause           3.2                                  80d28bedfe5d        683kB

Additional information you deem important (e.g. issue happens only occasionally):

2021-02-06.01:20:25.469169|MUX_CABLE_TABLE:Ethernet76|SET|state:standby
2021-02-06.01:20:25.473528|HW_MUX_CABLE_TABLE|Ethernet104|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.476099|MUX_CABLE_TABLE:Ethernet84|SET|state:standby
2021-02-06.01:20:25.479543|HW_MUX_CABLE_TABLE|Ethernet4|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.484714|MUX_CABLE_TABLE:Ethernet0|SET|state:standby
2021-02-06.01:20:25.488291|HW_MUX_CABLE_TABLE|Ethernet52|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.493799|MUX_CABLE_TABLE:Ethernet104|SET|state:standby
2021-02-06.01:20:25.497390|HW_MUX_CABLE_TABLE|Ethernet60|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.503617|MUX_CABLE_TABLE:Ethernet4|SET|state:standby
2021-02-06.01:20:25.506928|HW_MUX_CABLE_TABLE|Ethernet68|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.512636|MUX_CABLE_TABLE:Ethernet52|SET|state:standby
2021-02-06.01:20:25.517049|HW_MUX_CABLE_TABLE|Ethernet76|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.520814|MUX_CABLE_TABLE:Ethernet60|SET|state:standby
2021-02-06.01:20:25.524189|HW_MUX_CABLE_TABLE|Ethernet84|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.531676|MUX_CABLE_TABLE:Ethernet68|SET|state:standby
2021-02-06.01:20:25.535306|HW_MUX_CABLE_TABLE|Ethernet0|SET|state:standby|read_side:1|active_side:2
2021-02-06.01:20:25.543988|MUX_CABLE_TABLE:Ethernet76|SET|state:standby
2021-02-06.01:20:25.563430|HW_MUX_CABLE_TABLE|Ethernet104|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.570289|MUX_CABLE_TABLE:Ethernet84|SET|state:standby
2021-02-06.01:20:25.574078|HW_MUX_CABLE_TABLE|Ethernet4|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.579401|MUX_CABLE_TABLE:Ethernet0|SET|state:standby
2021-02-06.01:20:25.583161|HW_MUX_CABLE_TABLE|Ethernet52|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.588829|MUX_CABLE_TABLE:Ethernet104|SET|state:standby
2021-02-06.01:20:25.594047|HW_MUX_CABLE_TABLE|Ethernet60|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.602518|HW_MUX_CABLE_TABLE|Ethernet68|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.605042|MUX_CABLE_TABLE:Ethernet4|SET|state:standby
2021-02-06.01:20:25.609081|MUX_CABLE_TABLE:Ethernet52|SET|state:standby
2021-02-06.01:20:25.613308|HW_MUX_CABLE_TABLE|Ethernet76|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.620507|MUX_CABLE_TABLE:Ethernet60|SET|state:standby
2021-02-06.01:20:25.624418|HW_MUX_CABLE_TABLE|Ethernet84|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.631734|MUX_CABLE_TABLE:Ethernet68|SET|state:standby
2021-02-06.01:20:25.635805|HW_MUX_CABLE_TABLE|Ethernet0|SET|state:standby|read_side:1|active_side:2
2021-02-06.01:20:25.642890|MUX_CABLE_TABLE:Ethernet76|SET|state:standby
2021-02-06.01:20:25.646914|HW_MUX_CABLE_TABLE|Ethernet104|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.653860|MUX_CABLE_TABLE:Ethernet84|SET|state:standby
2021-02-06.01:20:25.660354|HW_MUX_CABLE_TABLE|Ethernet4|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.664613|MUX_CABLE_TABLE:Ethernet0|SET|state:standby
2021-02-06.01:20:25.668773|HW_MUX_CABLE_TABLE|Ethernet52|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.674850|MUX_CABLE_TABLE:Ethernet104|SET|state:standby
2021-02-06.01:20:25.679089|HW_MUX_CABLE_TABLE|Ethernet60|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.685846|MUX_CABLE_TABLE:Ethernet4|SET|state:standby
2021-02-06.01:20:25.690002|HW_MUX_CABLE_TABLE|Ethernet68|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.696852|MUX_CABLE_TABLE:Ethernet52|SET|state:standby
2021-02-06.01:20:25.701200|HW_MUX_CABLE_TABLE|Ethernet76|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.708410|MUX_CABLE_TABLE:Ethernet60|SET|state:standby
2021-02-06.01:20:25.712958|HW_MUX_CABLE_TABLE|Ethernet84|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.721484|MUX_CABLE_TABLE:Ethernet68|SET|state:standby
2021-02-06.01:20:25.726363|HW_MUX_CABLE_TABLE|Ethernet0|SET|state:standby|read_side:1|active_side:2
2021-02-06.01:20:25.735008|MUX_CABLE_TABLE:Ethernet76|SET|state:standby
2021-02-06.01:20:25.740571|HW_MUX_CABLE_TABLE|Ethernet104|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.747246|MUX_CABLE_TABLE:Ethernet84|SET|state:standby
2021-02-06.01:20:25.751672|HW_MUX_CABLE_TABLE|Ethernet4|SET|state:standby|read_side:2|active_side:1
2021-02-06.01:20:25.758384|MUX_CABLE_TABLE:Ethernet0|SET|state:standby
2021-02-06.01:20:25.765610|HW_MUX_CABLE_TABLE|Ethernet52|SET|state:standby|read_side:2|active_side:1
@prsunny
Copy link
Contributor

prsunny commented Apr 8, 2021

With the latest modification of orchagent for initialization, this is no longer seen. Will close this for now and can re-open if issue is seen again.

@prsunny prsunny closed this as completed Apr 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prsunny prsunny

@vdahiya12 vdahiya12

Labels
Issue for 202012
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@prsunny @tahmed-dev @vdahiya12