[202012][teammgrd]: Improve LAGs cleanup on shutdown #1916
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…c-net#1841) This PR is intended to fix LAGs cleanup degradation caused by python2.7 -> python3 migration. The approach is to replace `teamd -k -t` call with the raw `SIGTERM` and add PID alive check. This will make sure the `teammgrd` is stopped only after all managed processes are being killed. resolves: sonic-net/sonic-buildimage#8071 **What I did** * Replaced `teamd -k -t` call with raw `SIGTERM` * Added PID alive check **Why I did it** * To fix LAGs cleanup timeout issue caused by python2.7 -> python3 upgrade **How I verified it** 1. Configure 64 LAG RIFs 2. Reload config
|
A cherry-pick of: #1841 for |
nazariig
changed the title
nazariig
requested review from
prsunny,
lguohan,
abdosi,
qiluo-msft,
yxieca,
liat-grozovik and
volodymyrsamotiy
|
/azpw run |
|
/AzurePipelines run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@judyjoseph Could you check? |
I don't think it is related to this change -- triggering an azp run again, |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azpw run |
|
/AzurePipelines run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azpw run |
|
/AzurePipelines run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Only LGTM check is failing: [2021-09-17 15:46:48] [build-stderr] main.cpp:2:10: fatal error: sai.h: No such file or directory
[2021-09-17 15:46:48] [build-stderr] 2 | #include "sai.h"
[2021-09-17 15:46:48] [build-stderr] | ^~~~~~~
[2021-09-17 15:46:48] [build-stderr] compilation terminated.
[2021-09-17 15:46:48] [build-stderr] make[2]: *** [Makefile:623: orchagent-main.o] Error 1
[2021-09-17 15:46:48] [build-stdout] make[2]: Leaving directory '/opt/src/orchagent'
[2021-09-17 15:46:48] [build-stderr] make[1]: *** [Makefile:410: all-recursive] Error 1
[2021-09-17 15:46:48] [build-stdout] make[1]: Leaving directory '/opt/src'
[2021-09-17 15:46:48] [build-stderr] make: *** [Makefile:342: all] Error 2
[2021-09-17 15:46:48] [build-stderr] + '[' -f build.ninja ']'
[2021-09-17 15:46:48] [build-stderr] + '[' -d ../_lgtm_build_dir ']'
[2021-09-17 15:46:48] [build-stdout] Semmle autobuild: no supported build system detected.
[2021-09-17 15:46:48] [build-stderr] + for f in build build.sh
[2021-09-17 15:46:48] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/dist/tools/linux64/preload_tracer, /opt/dist/cpp/tools/do-build])
[2021-09-17 15:46:48] [build-stderr] + '[' -x build ']'
[2021-09-17 15:46:48] [build-stderr] + for f in build build.sh
[2021-09-17 15:46:48] [build-stderr] + '[' -x build.sh ']'
[2021-09-17 15:46:48] [build-stderr] + '[' -f setup.py ']'
[2021-09-17 15:46:48] [build-stderr] + echo 'Semmle autobuild: no supported build system detected.'
[2021-09-17 15:46:48] [build-stderr] + exit 1
[2021-09-17 15:46:48] [build-stderr] A fatal error occurred: Exit status 1 from command: [/opt/dist/cpp/tools/do-build]
[2021-09-17 15:46:48] [build-stderr] deptrace-server: received exit command
[2021-09-17 15:46:48] [ERROR] Spawned process exited abnormally (code 2; tried to run: [/opt/work/lgtm-workspace/lgtm/extract.sh])
A fatal error occurred: Exit status 2 from command: [/opt/work/lgtm-workspace/lgtm/extract.sh]Potential fix: @judyjoseph / @qiluo-msft taking into consideration that this PR is a cherry-pick and LGTM failures are not relevant, can we proceed with the merge? |
|
Override lgtm on 202012, which is still under investigation. |
|
directory for 202012 SAI headers should be |
|
some how SAI headers are not installed in that directory for 202012 |
@kcudnik please have a look at: [2021-09-17 15:46:48] [build-stdout] make[2]: Entering directory '/opt/src/orchagent'
[2021-09-17 15:46:48] [build-stdout] g++ -DHAVE_CONFIG_H -I. -I.. -I ../lib -I .. -I ../warmrestart -I flex_counter -I debug_counter -g -DNDEBUG -std=c++14 -Wall -fPIC -Wno-write-strings -I/usr/include/libnl3 -I/usr/include/swss -Werror -Wno-reorder -Wcast-align -Wcast-qual -Wconversion -Wdisabled-optimization -Wextra -Wfloat-equal -Wformat=2 -Wformat-nonliteral -Wformat-security -Wformat-y2k -Wimport -Winit-self -Winvalid-pch -Wlong-long -Wmissing-field-initializers -Wmissing-format-attribute -Wno-aggregate-return -Wno-padded -Wno-switch-enum -Wno-unused-parameter -Wpacked -Wpointer-arith -Wredundant-decls -Wstack-protector -Wstrict-aliasing=3 -Wswitch -Wswitch-default -Wunreachable-code -Wunused -Wvariadic-macros -Wno-switch-default -Wno-long-long -Wno-redundant-decls -I /usr/include/sai -I/opt/work/lgtm-workspace/usr/include -I/opt/work/lgtm-workspace/usr/include/swss -I/opt/work/lgtm-workspace/usr/include/sai -g -O2 -MT orchagent-main.o -MD -MP -MF .deps/orchagent-main.Tpo -c -o orchagent-main.o `test -f 'main.cpp' || echo './'`main.cpp
[2021-09-17 15:46:48] [build-stderr] main.cpp:2:10: fatal error: sai.h: No such file or directory
[2021-09-17 15:46:48] [build-stderr] 2 | #include "sai.h"
[2021-09-17 15:46:48] [build-stderr] | ^~~~~~~
[2021-09-17 15:46:48] [build-stderr] compilation terminated.
[2021-09-17 15:46:48] [build-stderr] make[2]: *** [Makefile:623: orchagent-main.o] Error 1
[2021-09-17 15:46:48] [build-stdout] make[2]: Leaving directory '/opt/src/orchagent'
[2021-09-17 15:46:48] [build-stderr] make[1]: *** [Makefile:410: all-recursive] Error 1
[2021-09-17 15:46:48] [build-stdout] make[1]: Leaving directory '/opt/src'
[2021-09-17 15:46:48] [build-stderr] make: *** [Makefile:342: all] Error 2
[2021-09-17 15:46:48] [build-stderr] + '[' -f build.ninja ']'
[2021-09-17 15:46:48] [build-stderr] + '[' -d ../_lgtm_build_dir ']'
[2021-09-17 15:46:48] [build-stdout] Semmle autobuild: no supported build system detected.
[2021-09-17 15:46:48] [build-stderr] + for f in build build.sh
[2021-09-17 15:46:48] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/dist/tools/linux64/preload_tracer, /opt/dist/cpp/tools/do-build])
[2021-09-17 15:46:48] [build-stderr] + '[' -x build ']'
[2021-09-17 15:46:48] [build-stderr] + for f in build build.sh
[2021-09-17 15:46:48] [build-stderr] + '[' -x build.sh ']'
[2021-09-17 15:46:48] [build-stderr] + '[' -f setup.py ']'
[2021-09-17 15:46:48] [build-stderr] + echo 'Semmle autobuild: no supported build system detected.'
[2021-09-17 15:46:48] [build-stderr] + exit 1
[2021-09-17 15:46:48] [build-stderr] A fatal error occurred: Exit status 1 from command: [/opt/dist/cpp/tools/do-build]
[2021-09-17 15:46:48] [build-stderr] deptrace-server: received exit command
[2021-09-17 15:46:48] [ERROR] Spawned process exited abnormally (code 2; tried to run: [/opt/work/lgtm-workspace/lgtm/extract.sh])
A fatal error occurred: Exit status 2 from command: [/opt/work/lgtm-workspace/lgtm/extract.sh]Basically we have: -I/usr/include/swss
-I /usr/include/sai
-I/opt/work/lgtm-workspace/usr/include
-I/opt/work/lgtm-workspace/usr/include/sai
-I/opt/work/lgtm-workspace/usr/include/swss |
|
yes, i noticed that, but none of those directories contains sai.h, dont know yet why that happens, otherwise gcc would pick that include |
|
im testing this issue on this branch: #1921 this builds master binaries, and probably only swss is compiled on 202012 branch PR |
|
output: directories sai don't even exists after instaling libsaivs-dev, and headers dont seems to be instale d at all, since the only sai.h is in sairedis source |
|
@kcudnik -> this builds master binaries, and probably only swss is compiled on 202012 branch PR |
|
don't know how to force lgtm to use different commands on different branches :( |
libsaivs-dev don't contain any sai headers :( |
@kcudnik nice catch! Seems to be a buildsystem issue |
seems like autogen fails, and dont even configure/build required libraries, not sure how this passes on master |
|
https://lgtm.com/help/lgtm/lgtm.yml-configuration-file does not specify any conditions for branches :( |
|
@kcudnik can you please check with master: prepare:
packages:
- libxml-simple-perl
- aspell
- aspell-en
- libhiredis-dev
- libnl-3-dev
- libnl-genl-3-dev
- libnl-route-3-dev
- libnl-nf-3-dev
- libzmq3-dev
- libzmq5
- swig3.0
- libpython2.7-dev
- libgtest-dev
- dh-exec
- doxygen
- cdbs
- bison
- flex
- graphviz
- autoconf-archive202012: prepare:
packages:
- libxml-simple-perl
- aspell
- aspell-en
- libhiredis-dev
- libnl-3-dev
- libnl-genl-3-dev
- libnl-route-3-dev
- libnl-nf-3-dev
- libzmq3-dev
- libzmq5
- swig3.0
- libpython2.7-dev
- libgtest-dev
- dh-exec
- doxygen
- graphviz |
|
sure |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Nazarii Hnydyn [email protected]
This PR is intended to fix LAGs cleanup degradation caused by python2.7 -> python3 migration.
The approach is to replace
teamd -k -tcall with the rawSIGTERMand add PID alive check.This will make sure the
teammgrdis stopped only after all managed processes are being killed.resolves: sonic-net/sonic-buildimage#8071
What I did
teamd -k -tcall with rawSIGTERMWhy I did it
How I verified it
Details if related