Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S6100/201911/WFB/BGP docker container fails to run #5909

Closed
mini-nair-dell opened this issue Nov 12, 2020 · 3 comments
Closed

S6100/201911/WFB/BGP docker container fails to run #5909

mini-nair-dell opened this issue Nov 12, 2020 · 3 comments

Comments

@mini-nair-dell
Copy link

Description

  1. The BGP docker doesnt get to run in the image.

root@sonic-s6100-10:~# show ver
SONiC Software Version: SONiC.201911.38-005702ba

  1. The issue is not seen on 36th label

  2. Could see the below fixes on 37th build. Not sure if this broke.

[multi-ASIC] BGP internal neighbor table support (#5520) — abdosi / githubweb
[multi-ASIC] util changes with the BGP_INTERNAL_NEIGHBOR table. (#5760) — abdosi / githubweb

Steps

  1. Load the image to the box.
  2. BGP docker get to run, as its exited

Syslog

root@sonic-s6100-10:# tail -f /var/log/syslog
Nov 12 09:04:45.132067 sonic-s6100-10 INFO dockerd[617]: time="2020-11-12T09:04:45.131847061Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 12 09:04:45.270667 sonic-s6100-10 INFO bgp.sh[5863]: 0
Nov 12 09:04:45.362655 sonic-s6100-10 INFO bgp.sh[5974]: bgp
Nov 12 09:05:15.534052 sonic-s6100-10 INFO systemd[1]: bgp.service: Service hold-off time over, scheduling restart.
Nov 12 09:05:15.534862 sonic-s6100-10 INFO systemd[1]: Stopped BGP container.
Nov 12 09:05:15.537147 sonic-s6100-10 INFO systemd[1]: Starting BGP container...
Nov 12 09:05:16.911174 sonic-s6100-10 INFO bgp.sh[6047]: Starting existing bgp container with HWSKU Force10-S6100-T1
Nov 12 09:05:17.061658 sonic-s6100-10 INFO containerd[493]: time="2020-11-12T09:05:17.061036646Z" level=info msg="shim containerd-shim started" address=/containerd-shim/7351848d6cfa6692e3a7131350836839a8acb52b28bf59ab50b578bd39035dca.sock debug=false pid=6081
Nov 12 09:05:17.252782 sonic-s6100-10 INFO bgp.sh[6047]: bgp
Nov 12 09:05:17.261807 sonic-s6100-10 INFO systemd[1]: Started BGP container.
Nov 12 09:05:24.517122 sonic-s6100-10 INFO bgp#rsyslogd: [origin software="rsyslogd" swVersion="8.1901.0" x-pid="39" x-info="https://www.rsyslog.com"] start
Nov 12 09:05:24.522333 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:36,770 INFO spawned: 'zebra' with pid 48
Nov 12 09:05:24.522333 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:36,770 INFO success: zebra entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Nov 12 09:05:24.522333 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:37,397 INFO spawned: 'staticd' with pid 52
Nov 12 09:05:24.522333 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:37,398 INFO success: staticd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Nov 12 09:05:24.522333 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:38,100 INFO spawned: 'bgpd' with pid 55
Nov 12 09:05:24.522333 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:38,101 INFO success: bgpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Nov 12 09:05:24.522384 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:38,732 INFO spawned: 'fpmsyncd' with pid 62
Nov 12 09:05:24.522384 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:38,733 INFO success: fpmsyncd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Nov 12 09:05:24.522429 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:39,391 INFO spawned: 'bgpcfgd' with pid 65
Nov 12 09:05:24.522429 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:39,391 INFO success: bgpcfgd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Nov 12 09:05:24.522429 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:40,020 INFO spawned: 'bgpmon' with pid 69
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:40,021 INFO success: bgpmon entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:40,062 INFO exited: start.sh (exit status 0; expected)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:40,700 INFO exited: bgpcfgd (exit status 1; not expected)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:41,711 WARN received SIGTERM indicating exit request
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:41,712 INFO waiting for bgpmon, fpmsyncd, zebra, bgpd, supervisor-proc-exit-listener, staticd, rsyslogd to die
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:42,878 INFO stopped: fpmsyncd (terminated by SIGTERM)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:42,881 INFO stopped: bgpmon (terminated by SIGTERM)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:43,885 INFO stopped: bgpd (terminated by SIGKILL)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:43,887 INFO stopped: staticd (exit status 0)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:43,940 INFO stopped: zebra (exit status 0)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:43,943 INFO stopped: rsyslogd (exit status 0)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:44,944 INFO waiting for supervisor-proc-exit-listener to die
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:04:44,947 INFO stopped: supervisor-proc-exit-listener (terminated by SIGTERM)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:17,882 CRIT Set uid to user 0
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:17,882 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:17,904 INFO RPC interface 'supervisor' initialized
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:17,904 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:17,904 INFO supervisord started with pid 1
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:18,907 INFO spawned: 'supervisor-proc-exit-listener' with pid 8
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:18,910 INFO spawned: 'start.sh' with pid 9
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:19,066 INFO success: start.sh entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:20,068 INFO success: supervisor-proc-exit-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:24,496 INFO spawned: 'rsyslogd' with pid 39
Nov 12 09:05:24.522853 sonic-s6100-10 INFO bgp#supervisord 2020-11-12 09:05:24,497 INFO success: rsyslogd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Nov 12 09:05:25.699448 sonic-s6100-10 INFO bgp#supervisord: start.sh zebra: started
Nov 12 09:05:25.776489 sonic-s6100-10 INFO bgp#supervisord: zebra 2020/11/12 09:05:25 warnings: ZEBRA: [EC 4043309105] Disabling MPLS support (no kernel support)
Nov 12 09:05:26.305507 sonic-s6100-10 INFO bgp#supervisord: start.sh staticd: started
Nov 12 09:05:26.940307 sonic-s6100-10 INFO bgp#supervisord: start.sh bgpd: started
Nov 12 09:05:27.584662 sonic-s6100-10 INFO bgp#supervisord: start.sh fpmsyncd: started
Nov 12 09:05:27.603628 sonic-s6100-10 NOTICE bgp#fpmsyncd: :- loadRedisScript: lua script loaded, sha: 88270a7c5c90583e56425aca8af8a4b8c39fe757
Nov 12 09:05:27.609873 sonic-s6100-10 INFO bgp#supervisord: fpmsyncd Waiting for fpm-client connection...
Nov 12 09:05:28.284642 sonic-s6100-10 INFO bgp#supervisord: start.sh bgpcfgd: started
Nov 12 09:05:28.694069 sonic-s6100-10 INFO bgp#bgpcfgd: Start waiting for FRR daemons: 2020-11-12 09:05:28.693676
Nov 12 09:05:28.694069 sonic-s6100-10 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'show daemons']'.
Nov 12 09:05:28.919125 sonic-s6100-10 INFO bgp#bgpcfgd: All required daemons have connected to vtysh: 2020-11-12 09:05:28.918722
Nov 12 09:05:28.941586 sonic-s6100-10 INFO bgp#supervisord: start.sh bgpmon: started
Nov 12 09:05:29.023039 sonic-s6100-10 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'show bgp vrfs json']'.
Nov 12 09:05:29.248627 sonic-s6100-10 DEBUG bgp#bgpcfgd: execute command '['vtysh', '-c', 'show bgp vrf default neighbors json']'.
Nov 12 09:05:29.528579 sonic-s6100-10 CRIT bgp#bgpcfgd: Got an exception 'module' object has no attribute 'CFG_BGP_INTERNAL_NEIGHBOR_TABLE_NAME': Traceback: Traceback (most recent call last):#12 File "/usr/local/lib/python2.7/dist-packages/bgpcfgd/main.py", line 67, in main#012 do_work()#12 File "/usr/local/lib/python2.7/dist-packages/bgpcfgd/main.py", line 47, in do_work#012 BGPPeerMgrBase(common_objs, "CONFIG_DB", swsscommon.CFG_BGP_INTERNAL_NEIGHBOR_TABLE_NAME, "internal", False),#012AttributeError: 'module' object has no attribute 'CFG_BGP_INTERNAL_NEIGHBOR_TABLE_NAME'
Nov 12 09:05:29.530972 sonic-s6100-10 INFO bgp#supervisord: bgpcfgd Traceback (most recent call last):
Nov 12 09:05:29.531159 sonic-s6100-10 INFO bgp#supervisord: bgpcfgd File "/usr/local/bin/bgpcfgd", line 8, in
Nov 12 09:05:29.531340 sonic-s6100-10 INFO bgp#supervisord: bgpcfgd sys.exit(main())
Nov 12 09:05:29.531472 sonic-s6100-10 INFO bgp#supervisord: bgpcfgd File "/usr/local/lib/python2.7/dist-packages/bgpcfgd/main.py", line 67, in main
Nov 12 09:05:29.531472 sonic-s6100-10 INFO bgp#supervisord: bgpcfgd do_work()
Nov 12 09:05:29.531472 sonic-s6100-10 INFO bgp#supervisord: bgpcfgd File "/usr/local/lib/python2.7/dist-packages/bgpcfgd/main.py", line 47, in do_work
Nov 12 09:05:29.531472 sonic-s6100-10 INFO bgp#supervisord: bgpcfgd BGPPeerMgrBase(common_objs, "CONFIG_DB", swsscommon.CFG_BGP_INTERNAL_NEIGHBOR_TABLE_NAME, "internal", False),
Nov 12 09:05:29.531498 sonic-s6100-10 INFO bgp#supervisord: bgpcfgd AttributeError: 'module' object has no attribute 'CFG_BGP_INTERNAL_NEIGHBOR_TABLE_NAME'
Nov 12 09:05:30.601431 sonic-s6100-10 INFO bgp#supervisor-proc-exit-listener: Process bgpcfgd exited unxepectedly. Terminating supervisor...
Nov 12 09:05:30.788460 sonic-s6100-10 INFO bgp#supervisord: fpmsyncd Connected!
Nov 12 09:05:31.804504 sonic-s6100-10 NOTICE bgp#staticd[51]: Terminating on signal
Nov 12 09:05:33.014293 sonic-s6100-10 INFO containerd[493]: time="2020-11-12T09:05:33.013970013Z" level=info msg="shim reaped" id=e2c39b463ca9bdda30be2e59beac416c4ded8f7d96718d7400505097464ed9b1
Nov 12 09:05:33.024303 sonic-s6100-10 INFO dockerd[617]: time="2020-11-12T09:05:33.024139218Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 12 09:05:33.141538 sonic-s6100-10 INFO bgp.sh[6112]: 0
Nov 12 09:05:33.227917 sonic-s6100-10 INFO bgp.sh[6211]: bgp
^C
root@sonic-s6100-10:#

Thanks
Mini
@liat-grozovik
Copy link
Collaborator

Thanks for reporting it. it is not relevant to specific ASIC or platform. seems like 201911 is broken as of recent commits.
@judyjoseph FYI as it seems related to merges done on master but cherry picked to 201911.

@judyjoseph
Copy link
Contributor

Thanks @abdosi for cherry-picking this PR, sonic-net/sonic-swss-common#389.
It was the DB schema change that was needed. The issue should be fixed now. Will verify a build and close.

@abdosi
Copy link
Contributor

abdosi commented Nov 12, 2020

Verified locally and it is fine. New Build should be fine

@abdosi abdosi closed this as completed Nov 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@liat-grozovik @judyjoseph @mini-nair-dell @abdosi