Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bgp container not started due to database #390

Closed
lguohan opened this issue Mar 9, 2017 · 4 comments
Closed

bgp container not started due to database #390

lguohan opened this issue Mar 9, 2017 · 4 comments
Assignees
@taoyl-ms
Labels
Bug 🐛 Critical ❗

Comments

@lguohan
Copy link
Collaborator

lguohan commented Mar 9, 2017 •
edited by stcheng
Loading 

admin@str-s6000-acs-7:~$ sudo systemctl status bgp
â—� bgp.service - BGP container
   Loaded: loaded (/etc/systemd/system/bgp.service; enabled)
   Active: failed (Result: exit-code) since Wed 2017-03-08 10:55:45 UTC; 21h ago
 Main PID: 802 (code=exited, status=125)

Mar 08 10:55:39 str-s6000-acs-7 systemd[1]: Started BGP container.
Mar 08 10:55:42 str-s6000-acs-7 bgp.sh[802]: docker: Error response from daemon: No such container: database.
Mar 08 10:55:42 str-s6000-acs-7 bgp.sh[802]: See 'docker run --help'.
Mar 08 10:55:42 str-s6000-acs-7 systemd[1]: bgp.service: main process exited, code=exited, status=125/n/a
Mar 08 10:55:45 str-s6000-acs-7 bgp.sh[903]: Error response from daemon: No such container: bgp
Mar 08 10:55:45 str-s6000-acs-7 systemd[1]: bgp.service: control process exited, code=exited status=1
Mar 08 10:55:45 str-s6000-acs-7 systemd[1]: Unit bgp.service entered failed state.
@stcheng
Copy link
Contributor

stcheng commented Mar 21, 2017 

syslog:998:Mar 21 20:42:51 str-s6000-on-2 INFO bash[739]: Error response from daemon: No such container: database
syslog:1001:Mar 21 20:42:52 str-s6000-on-2 INFO bash[739]: Error response from daemon: No such container: database
syslog:1006:Mar 21 20:42:54 str-s6000-on-2 INFO bash[739]: Error response from daemon: No such container: database
syslog:1021:Mar 21 20:42:55 str-s6000-on-2 INFO bgp.sh[731]: docker: Error response from daemon: No such container: database.
syslog:1026:Mar 21 20:42:55 str-s6000-on-2 INFO bash[739]: Error response from daemon: No such container: database
syslog:1087:Mar 21 20:42:58 str-s6000-on-2 INFO bgp.sh[832]: Error response from daemon: No such container: bgp

can be reproduced.

@taoyl-ms
Copy link
Contributor

Mar 21 20:42:50 str-s6000-on-2 INFO systemd[1]: Starting Database container...
Mar 21 20:42:50 str-s6000-on-2 INFO systemd[1]: Started Database container.
Mar 21 20:42:50 str-s6000-on-2 INFO systemd[1]: Starting BGP container...
Mar 21 20:42:51 str-s6000-on-2 INFO systemd[1]: Started BGP container.

The reason seems to be that systemd is not blocked correctly by database.sh script, and database service is considered started immediately after calling the script. Will make deeper investigation for the root cause.

@taoyl-ms taoyl-ms self-assigned this Mar 21, 2017
@taoyl-ms taoyl-ms mentioned this issue Mar 22, 2017
Merged
@taoyl-ms
Copy link
Contributor

Please refer to description of PR#421 for detail information.

stcheng pushed a commit that referenced this issue Mar 22, 2017
@taoyl-ms
[oneimage]: Fix race condition in systemd container services (#421)
42ae02b 
When Type=simple, systemd will consider the service activated immediately
after specified in ExecStart process is started. If there is downstream
service depending on the state prepared in ExecStart, there will be race condition.

For example, issue #390. In this case, database.service calls database.sh, which
calls docker run or docker start -a to start database container. However, systemd
considers database.service successfully started at the time database.sh begins,
not after docker run finishes. As database.service is consider started, bgp.service
can be started. The redis database, which bgp service depends on, might or might not
have been started at this time point.

To fix this issue (and still keeping the functionality to monitor docker status with
systemd), we split the ExecStart process into an ExecStartPre part and an ExecStart
part. docker run is splitted into docker run -d then docker attach , while docker start
-a  is splitted into docker start and then docker attach. In this way, we make sure
the downstream services are blocked until container is successfully started.
@stcheng
Copy link
Contributor

stcheng commented Mar 22, 2017

Thanks!

@stcheng stcheng closed this as completed Mar 22, 2017
Kalimuthu-Velappan pushed a commit to Kalimuthu-Velappan/sonic-buildimage that referenced this issue Sep 12, 2019
@yxieca
[mlnx|ffb]: Add fast-fast flow in fast(warm)-reboot script (sonic-net…
97d73da 
…#390)

* [mlnx|ffb] Add fast-fast flow in fast(warm)-reboot script

Signed-off-by: Stepan Blyschak <[email protected]>

* [Mellanox|FFB]: Fix review comments

* Change naming convention from "fast-fast" to "fastfast"

Signed-off-by: Volodymyr Samotiy <[email protected]>
madhanmellanox pushed a commit to madhanmellanox/sonic-buildimage that referenced this issue Mar 23, 2020
@yxieca
[Arista7260cx3] add all ports in buffer configuration (sonic-net#390)
4cb542f
dmytroxshevchuk pushed a commit to dmytroxshevchuk/sonic-buildimage that referenced this issue Aug 31, 2020
@yxieca
[SAI header] upgrade SAI header to version v1.3.6 (sonic-net#390)
06418bb 
- Pick up SAI switch attribute SAI_SWITCH_ATTR_PRE_SHUTDOWN definition

Signed-off-by: Ying Xie <[email protected]>
mssonicbld added a commit that referenced this issue Aug 11, 2023
@mssonicbld
[submodule] Update submodule sonic-platform-common to the latest HEAD…
0269e60 
… automatically (#16106)

#### Why I did it
src/sonic-platform-common
```
* ab70e66 - (HEAD -> master, origin/master, origin/HEAD) Add new SSD type support (#390) (21 hours ago) [Junchao-Mellanox]
```
#### How I did it
#### How to verify it
#### Description for the changelog
mssonicbld added a commit that referenced this issue Aug 17, 2023
@mssonicbld
[submodule] Update submodule sonic-platform-common to the latest HEAD…
651a132 
… automatically (#16180)

#### Why I did it
src/sonic-platform-common
```
* 2da4286 - (HEAD -> 202211, origin/202211) Add new SSD type support (#390) (14 hours ago) [Junchao-Mellanox]
```
#### How I did it
#### How to verify it
#### Description for the changelog
sonic-otn pushed a commit to sonic-otn/sonic-buildimage that referenced this issue Sep 20, 2023
@mssonicbld @sonic-otn
[submodule] Update submodule sonic-platform-common to the latest HEAD…
7c0972a 
… automatically (sonic-net#16106)

#### Why I did it
src/sonic-platform-common
```
* ab70e66 - (HEAD -> master, origin/master, origin/HEAD) Add new SSD type support (sonic-net#390) (21 hours ago) [Junchao-Mellanox]
```
#### How I did it
#### How to verify it
#### Description for the changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@taoyl-ms taoyl-ms

Labels
Bug 🐛 Critical ❗
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lguohan @stcheng @taoyl-ms