[dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 #7083

jleveque · 2021-03-17T18:59:04Z

Why I did it

To prevent error messages like the following from being logged:

Mar 17 02:33:48.523153 vlab-01 INFO swss#supervisord 2021-03-17 02:33:48,518 ERRO pool supervisor-proc-exit-listener event buffer overflowed, discarding event 46

This is basically an addendum to #5247, which increased the event buffer size for dependent-startup. While supervisor-proc-exit-listener doesn't subscribe to as many events as dependent-startup, there is still a chance some containers (like swss, as in the example above) have enough processes running to cause an overflow of the default buffer size of 10.

This is especially important for preventing erroneous log_analyzer failures in the sonic-mgmt repo regression tests, which have started occasionally causing PR check builds to fail. Example here.

I set all supervisor-proc-exit-listener event buffer sizes to 1024, and also updated all dependent-startup event buffer sizes to 1024, as well, to keep things simple, unified, and allow headroom so that we will not need to adjust these values frequently, if at all.

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

201811
201911
202006
202012

lguohan · 2021-03-18T19:25:34Z

not sure why different docker has different buffer size.

jleveque · 2021-03-18T19:31:33Z

not sure why different docker has different buffer size.

It varies based on the number of processes running in the container. More processes = more events. We could choose a number high enough to work for all containers and apply it everywhere, but this approach is more conservative with memory usage in containers which don't have many processes running.

qiluo-msft · 2021-03-18T23:41:19Z

How large is the one buffer slot in memory? If the buffer overflow is critical error, let's use very large buffer size.

jleveque · 2021-03-18T23:59:59Z

How large is the one buffer slot in memory? If the buffer overflow is critical error, let's use very large buffer size.

There are different types of events, so they vary, but they appear to be relatively small: https://github.com/Supervisor/supervisor/blob/master/supervisor/events.py. I considered just using a very large size for all. Thoughts?

lguohan · 2021-03-19T06:32:34Z

suggest 1024 for all.

…tener

jleveque · 2021-03-19T18:05:11Z

suggest 1024 for all.

Done in latest commit.

jleveque · 2021-03-19T22:20:04Z

This will not cherry-pick cleanly into 201911, so I opened a separate PR here.

lguohan · 2021-03-27T18:58:04Z

/azp run

azure-pipelines · 2021-03-27T18:58:15Z

Azure Pipelines successfully started running 1 pipeline(s).

…exit listener (#7106) Backport of #7083 to the 201911 branch. #### Why I did it To prevent error [messages](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802) like the following from being logged: ``` Mar 17 02:33:48.523153 vlab-01 INFO swss#supervisord 2021-03-17 02:33:48,518 ERRO pool supervisor-proc-exit-listener event buffer overflowed, discarding event 46 ``` This is basically an addendum to #5247, which increased the event buffer size for dependent-startup. While supervisor-proc-exit-listener doesn't subscribe to as many events as dependent-startup, there is still a chance some containers (like swss, as in the example above) have enough processes running to cause an overflow of the default buffer size of 10. This is especially important for preventing erroneous log_analyzer failures in the sonic-mgmt repo regression tests, which have started occasionally causing PR check builds to fail. Example [here](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802). I set all supervisor-proc-exit-listener event buffer sizes to 1024, and also updated all dependent-startup event buffer sizes to 1024, as well, to keep things simple, unified, and allow headroom so that we will not need to adjust these values frequently, if at all.

daall · 2021-03-31T15:45:29Z

@jleveque can you also open a separate PR for 202012 please? There is a conflict.

jleveque · 2021-03-31T17:52:45Z

@jleveque can you also open a separate PR for 202012 please? There is a conflict.

202012 PR opened here: #7203

…exit listener; Set all event buffer sizes to 1024 (#7203) #### Why I did it Backport of #7083 to the 202012 branch. To prevent error [messages](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802) like the following from being logged: ``` Mar 17 02:33:48.523153 vlab-01 INFO swss#supervisord 2021-03-17 02:33:48,518 ERRO pool supervisor-proc-exit-listener event buffer overflowed, discarding event 46 ``` This is basically an addendum to #5247, which increased the event buffer size for dependent-startup. While supervisor-proc-exit-listener doesn't subscribe to as many events as dependent-startup, there is still a chance some containers (like swss, as in the example above) have enough processes running to cause an overflow of the default buffer size of 10. This is especially important for preventing erroneous log_analyzer failures in the sonic-mgmt repo regression tests, which have started occasionally causing PR check builds to fail. Example [here](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802). I set all supervisor-proc-exit-listener event buffer sizes to 1024, and also updated all dependent-startup event buffer sizes to 1024, as well, to keep things simple, unified, and allow headroom so that we will not need to adjust these values frequently, if at all.

…tener; Set all event buffer sizes to 1024 (sonic-net#7083) To prevent error [messages](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802) like the following from being logged: ``` Mar 17 02:33:48.523153 vlab-01 INFO swss#supervisord 2021-03-17 02:33:48,518 ERRO pool supervisor-proc-exit-listener event buffer overflowed, discarding event 46 ``` This is basically an addendum to sonic-net#5247, which increased the event buffer size for dependent-startup. While supervisor-proc-exit-listener doesn't subscribe to as many events as dependent-startup, there is still a chance some containers (like swss, as in the example above) have enough processes running to cause an overflow of the default buffer size of 10. This is especially important for preventing erroneous log_analyzer failures in the sonic-mgmt repo regression tests, which have started occasionally causing PR check builds to fail. Example [here](https://dev.azure.com/mssonic/build/_build/results?buildId=2254&view=logs&j=9a13fbcd-e92d-583c-2f89-d81f90cac1fd&t=739db6ba-1b35-5485-5697-de102068d650&l=802). I set all supervisor-proc-exit-listener event buffer sizes to 1024, and also updated all dependent-startup event buffer sizes to 1024, as well, to keep things simple, unified, and allow headroom so that we will not need to adjust these values frequently, if at all.

jleveque added Enhancement ➕ Request for 201911 Branch Request for 202012 Branch labels Mar 17, 2021

jleveque requested a review from lguohan March 17, 2021 18:59

jleveque requested a review from qiluo-msft as a code owner March 17, 2021 18:59

jleveque self-assigned this Mar 17, 2021

jleveque mentioned this pull request Mar 17, 2021

System Health: Fix ASIC key issue in Dell platform #6556

Merged

4 tasks

jleveque added the Bug 🐛 label Mar 18, 2021

jleveque added 2 commits March 19, 2021 17:55

[dockers][supervisor] Increase event buffer size for process exit lis…

05f93dc

…tener

Set all buffer_size values to 1024

1b3b6be

jleveque changed the title ~~[dockers][supervisor] Increase event buffer size for process exit listener~~ [dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 Mar 19, 2021

lguohan previously approved these changes Mar 19, 2021

View reviewed changes

qiluo-msft previously approved these changes Mar 19, 2021

View reviewed changes

Add 'buffer_size' parameter to all event listeners which were missing it

5f351a7

jleveque dismissed stale reviews from qiluo-msft and lguohan via 5f351a7 March 19, 2021 22:00

qiluo-msft approved these changes Mar 19, 2021

View reviewed changes

jleveque mentioned this pull request Mar 19, 2021

[201911][dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 #7106

Merged

jleveque removed the Request for 201911 Branch label Mar 19, 2021

lguohan merged commit c651a9a into sonic-net:master Mar 28, 2021

jleveque deleted the docker_event_buffer_exit_listener branch March 29, 2021 15:37

jleveque mentioned this pull request Mar 31, 2021

[202012][dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 #7203

Merged

jleveque removed the Request for 202012 Branch label Mar 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 #7083

[dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 #7083

jleveque commented Mar 17, 2021 •

edited

Loading

lguohan commented Mar 18, 2021

jleveque commented Mar 18, 2021

qiluo-msft commented Mar 18, 2021

jleveque commented Mar 18, 2021

lguohan commented Mar 19, 2021

jleveque commented Mar 19, 2021

jleveque commented Mar 19, 2021

lguohan commented Mar 27, 2021

azure-pipelines bot commented Mar 27, 2021

daall commented Mar 31, 2021

jleveque commented Mar 31, 2021

[dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 #7083

[dockers][supervisor] Increase event buffer size for process exit listener; Set all event buffer sizes to 1024 #7083

Conversation

jleveque commented Mar 17, 2021 • edited Loading

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

lguohan commented Mar 18, 2021

jleveque commented Mar 18, 2021

qiluo-msft commented Mar 18, 2021

jleveque commented Mar 18, 2021

lguohan commented Mar 19, 2021

jleveque commented Mar 19, 2021

jleveque commented Mar 19, 2021

lguohan commented Mar 27, 2021

azure-pipelines bot commented Mar 27, 2021

daall commented Mar 31, 2021

jleveque commented Mar 31, 2021

jleveque commented Mar 17, 2021 •

edited

Loading