Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rsyslog pushing "old" supervisor logs to syslog #1652

Closed
prsunny opened this issue Apr 26, 2018 · 3 comments · Fixed by #2439
Closed

rsyslog pushing "old" supervisor logs to syslog #1652

prsunny opened this issue Apr 26, 2018 · 3 comments · Fixed by #2439
Assignees

Comments

@prsunny
Copy link
Contributor

prsunny commented Apr 26, 2018

Description
While stopping and starting snmp service, it was observed that the same error message is repeatedly pushed to syslog.

Initial analysis suggest the logic here may have issues.

/etc/rsyslog.d/supervisor.conf => InputFileStateFile

Steps to reproduce the issue:

  1. systemctl stop snmp, systemctl start snmp multiple times
  2. tail -F /var/log/syslog | grep "not expected"
  3. The following logging happens everytime snmp service start. Notice the syslog time vs supervisord time

Describe the results you received:

Apr 26 22:43:48.757317 str-a7060cx-acs-1 INFO supervisord 2018-04-26 21:07:20,342 INFO exited: snmpd (terminated by SIGHUP; not expected)
Apr 26 22:45:58.609038 str-a7060cx-acs-1 INFO supervisord 2018-04-26 21:07:20,342 INFO exited: snmpd (terminated by SIGHUP; not expected)
Apr 26 23:04:25.800888 str-a7060cx-acs-1 INFO supervisord 2018-04-26 21:07:20,342 INFO exited: snmpd (terminated by SIGHUP; not expected)

Describe the results you expected:
After deleting the above line from supervisor/supervisord.log, and repeat the same steps of stopping and starting snmp, the issue was not observed

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

```
admin@str-a7060cx-acs-1:/$ show version 
SONiC Software Version: SONiC.20180330.16
```
@lguohan
Copy link
Collaborator

lguohan commented Aug 17, 2018

we can potentially classify the logs based on the docker, then the problem is that we do not have merged events logs

@lguohan
Copy link
Collaborator

lguohan commented Aug 17, 2018

another solution is the let supervisord to send the syslog directly to outside the docker, this will also solve the dual timestamp issues.

@lguohan
Copy link
Collaborator

lguohan commented Oct 23, 2018

I found the supervisord activity log also goes to stdout/stderr, therefore you can use docker logs to see them.

https://docs.docker.com/config/containers/logging/configure/#configure-the-delivery-mode-of-log-messages-from-container-to-log-driver

daall added a commit to daall/sonic-buildimage that referenced this issue Mar 19, 2021
…or 202012

sonic-swss
* [nbrmgrd] added function to parse IP address from APP_DB (sonic-net#1672)
* [MUX/PFCWD] Use in_ports for acls instead of seperate ACL table (sonic-net#1670)
* [mux] VS test for neigh, route and fdb (sonic-net#1656)
* [Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (sonic-net#1652)
* spell check fixes (sonic-net#1630)

sonic-utilities
* [reboot]: Stop mux before reboot on dual ToR (sonic-net#1500)
* [config] Disable/enable container monitoring when stopping/starting services (sonic-net#1499)
* Add 'show' and 'clear' command for PG drop (sonic-net#1461)
* [CLI][techsupport] Add NOOP option for commands that did not have that option (sonic-net#1445)
* [202012][reload] Improve reload by using sonic.target (sonic-net#1509)

Signed-off-by: Danny Allen <[email protected]>
daall added a commit that referenced this issue Mar 24, 2021
…or 202012 (#7105)

sonic-swss
* [nbrmgrd] added function to parse IP address from APP_DB (#1672)
* [MUX/PFCWD] Use in_ports for acls instead of seperate ACL table (#1670)
* [mux] VS test for neigh, route and fdb (#1656)
* [Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (#1652)
* spell check fixes (#1630)

sonic-utilities
* [reboot]: Stop mux before reboot on dual ToR (#1500)
* [config] Disable/enable container monitoring when stopping/starting services (#1499)
* Add 'show' and 'clear' command for PG drop (#1461)
* [CLI][techsupport] Add NOOP option for commands that did not have that option (#1445)
* [202012][reload] Improve reload by using sonic.target (#1509)

Signed-off-by: Danny Allen <[email protected]>
jleveque added a commit that referenced this issue Mar 26, 2021
Update sonic-sairedis submodule and also update sonic-swss submodule as there are interdependent changes.

* src/sonic-sairedis 13474d1...bc58b0f (12):
  > Add gbsyncdmgrd; deprecate gbsyncd_startup.py (#809)
  > Remove gbsyncd_start.sh (#808)
  > [gbsyncd] Fix shebang in gbsyncd_startup.py; Make script executable (#807)
  > [saiasiccmp] Add saiasiccmp tool to compare 2 asic views (#791)
  > [configure] Add -Wno-psabi to remove "passing argument changed in GCC 7.1" (#799)
  > Update FlexCounter.cpp, use m_pollInterval in MUTEX lock (#797)
  > [vs] Add special warm boot logic to populate default attributes (#796)
  > [ci]: add vstest (#795)
  > [tests] Add macsec unittest (#782)
  > [debian/control] libsairedis-dev depends on libzmq5-dev (#794)
  > [ci]: use build template (#793)
  > Rename duplicate file name (#773)

* src/sonic-swss 0b0d24c...5adb73e (47):
  > Initialize system port type variable (#1681)
  > [Dynamic Buffer Calc] Enhance the field checking in table handling (#1680)
  > Handle the clear request for 'Q_SHARED_ALL' (#1653)
  > [MuxOrch] FDB ageout safety check (#1674)
  > Deactivate mirror session only when session status is true in updateLagMember (#1666)
  > Revert "[buffermgr] Support maximum port headroom checking (#1607)" (#1675)
  > reduce severity of log to info in case of flush on non-existing member (#1669)
  > Revert "[Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (#1652)" (#1676)
  > [Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (#1652)
  > [acl] Move ACL table constants to acltable.h (#1671)
  > [nbrmgrd] added function to parse IP address from APP_DB (#1672)
  > [MUX/PFCWD] Use in_ports for acls instead of seperate ACL table (#1670)
  > [vog/systemlag] Voq lagid allocator (#1603)
  > Add table descriptions for dynamic buffer calculation to the documents (#1664)
  > [vstest/subintf] Add vs test case to validate processing sequence of APPL DB keys (#1663)
  > Remove vxlanmgrd dependency on orchagent (#1647)
  > Keep attribute order in bulk mode (#1659)
  > [mux] VS test for neigh, route and fdb (#1656)
  > [linksync] Netdev oper status determination using IFF_RUNNING (#1568)
  > [portorch] parse on/off value from autoneg (#1658)
  > [intfsorch] Create subport with the entry contains necessary attributes (#1650)
  > [ci]: Purge swss before install (#1654)
  > Update StateDB with error if state change failed, Update APP_DB in all state chg req (#1662)
  > Added changes to handle dependency check in FdbSyncd and FpmSyncd for warm-boot (#1556)
  > [synchronous mode] Add failure notification for SAI failures in synchronous mode (#1596)
  > [acl] Enable VLAN ID qualifier for ACL rules (#1648)
  > Updated PFCWD to use single ACL table for PFCWD and MUX (#1620)
  > [orchagent] Increase SAI REDIS response timeout to support FW upgrade during init (Mellanox only). (#1637)
  > [vstest/nhg]: use dvs_route fixture to make test_nhg more robust
  > [vstest]: add dvs_route fixture
  > [vstest/subintf] Update vs tests to validate physical port host interface vlan tag attribute (#1634)
  > Remove useless header  in macsecorch (#1628)
  > Add SAI_INGRESS_PRIORITY_GROUP_STAT_DROPPED_PACKETS counter, create new FlexCounter group (#1600)
  > fixed unsupported resource issue (#1641)
  > [test_virtual_chassis]: use wait_for to make test more robust (#1640)
  > spell check fixes (#1630)
  > [bufferorch] Handle NOT IMPLEMENTED status returned during set attr operation (#1639)
  > [ci]: run vstest
  > [test_virtual_chassis]: use wait_for function to improve test robustness
  > [Mux] Neighbor handling based on FDB entry (#1631)
  > [ci]: use build template (#1633)
  > Log level change from ERR to INFO for fetch systemports issue (#1632)
  > Migrate serdes programming to port serdes object (#1611)
  > [tests] Remove legacy saiattributelist.h dependency (#1608)
  > [buffermgr] Support maximum port headroom checking (#1607)
  > Support shared headroom pool on top of dynamic buffer calculation (#1581)
  > Fix the compiling errors in gcc9 (#1621)
raphaelt-nvidia pushed a commit to raphaelt-nvidia/sonic-buildimage that referenced this issue May 23, 2021
Update sonic-sairedis submodule and also update sonic-swss submodule as there are interdependent changes.

* src/sonic-sairedis 13474d1...bc58b0f (12):
  > Add gbsyncdmgrd; deprecate gbsyncd_startup.py (sonic-net#809)
  > Remove gbsyncd_start.sh (sonic-net#808)
  > [gbsyncd] Fix shebang in gbsyncd_startup.py; Make script executable (sonic-net#807)
  > [saiasiccmp] Add saiasiccmp tool to compare 2 asic views (sonic-net#791)
  > [configure] Add -Wno-psabi to remove "passing argument changed in GCC 7.1" (sonic-net#799)
  > Update FlexCounter.cpp, use m_pollInterval in MUTEX lock (sonic-net#797)
  > [vs] Add special warm boot logic to populate default attributes (sonic-net#796)
  > [ci]: add vstest (sonic-net#795)
  > [tests] Add macsec unittest (sonic-net#782)
  > [debian/control] libsairedis-dev depends on libzmq5-dev (sonic-net#794)
  > [ci]: use build template (sonic-net#793)
  > Rename duplicate file name (sonic-net#773)

* src/sonic-swss 0b0d24c...5adb73e (47):
  > Initialize system port type variable (sonic-net#1681)
  > [Dynamic Buffer Calc] Enhance the field checking in table handling (sonic-net#1680)
  > Handle the clear request for 'Q_SHARED_ALL' (sonic-net#1653)
  > [MuxOrch] FDB ageout safety check (sonic-net#1674)
  > Deactivate mirror session only when session status is true in updateLagMember (sonic-net#1666)
  > Revert "[buffermgr] Support maximum port headroom checking (sonic-net#1607)" (sonic-net#1675)
  > reduce severity of log to info in case of flush on non-existing member (sonic-net#1669)
  > Revert "[Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (sonic-net#1652)" (sonic-net#1676)
  > [Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (sonic-net#1652)
  > [acl] Move ACL table constants to acltable.h (sonic-net#1671)
  > [nbrmgrd] added function to parse IP address from APP_DB (sonic-net#1672)
  > [MUX/PFCWD] Use in_ports for acls instead of seperate ACL table (sonic-net#1670)
  > [vog/systemlag] Voq lagid allocator (sonic-net#1603)
  > Add table descriptions for dynamic buffer calculation to the documents (sonic-net#1664)
  > [vstest/subintf] Add vs test case to validate processing sequence of APPL DB keys (sonic-net#1663)
  > Remove vxlanmgrd dependency on orchagent (sonic-net#1647)
  > Keep attribute order in bulk mode (sonic-net#1659)
  > [mux] VS test for neigh, route and fdb (sonic-net#1656)
  > [linksync] Netdev oper status determination using IFF_RUNNING (sonic-net#1568)
  > [portorch] parse on/off value from autoneg (sonic-net#1658)
  > [intfsorch] Create subport with the entry contains necessary attributes (sonic-net#1650)
  > [ci]: Purge swss before install (sonic-net#1654)
  > Update StateDB with error if state change failed, Update APP_DB in all state chg req (sonic-net#1662)
  > Added changes to handle dependency check in FdbSyncd and FpmSyncd for warm-boot (sonic-net#1556)
  > [synchronous mode] Add failure notification for SAI failures in synchronous mode (sonic-net#1596)
  > [acl] Enable VLAN ID qualifier for ACL rules (sonic-net#1648)
  > Updated PFCWD to use single ACL table for PFCWD and MUX (sonic-net#1620)
  > [orchagent] Increase SAI REDIS response timeout to support FW upgrade during init (Mellanox only). (sonic-net#1637)
  > [vstest/nhg]: use dvs_route fixture to make test_nhg more robust
  > [vstest]: add dvs_route fixture
  > [vstest/subintf] Update vs tests to validate physical port host interface vlan tag attribute (sonic-net#1634)
  > Remove useless header  in macsecorch (sonic-net#1628)
  > Add SAI_INGRESS_PRIORITY_GROUP_STAT_DROPPED_PACKETS counter, create new FlexCounter group (sonic-net#1600)
  > fixed unsupported resource issue (sonic-net#1641)
  > [test_virtual_chassis]: use wait_for to make test more robust (sonic-net#1640)
  > spell check fixes (sonic-net#1630)
  > [bufferorch] Handle NOT IMPLEMENTED status returned during set attr operation (sonic-net#1639)
  > [ci]: run vstest
  > [test_virtual_chassis]: use wait_for function to improve test robustness
  > [Mux] Neighbor handling based on FDB entry (sonic-net#1631)
  > [ci]: use build template (sonic-net#1633)
  > Log level change from ERR to INFO for fetch systemports issue (sonic-net#1632)
  > Migrate serdes programming to port serdes object (sonic-net#1611)
  > [tests] Remove legacy saiattributelist.h dependency (sonic-net#1608)
  > [buffermgr] Support maximum port headroom checking (sonic-net#1607)
  > Support shared headroom pool on top of dynamic buffer calculation (sonic-net#1581)
  > Fix the compiling errors in gcc9 (sonic-net#1621)
carl-nokia pushed a commit to carl-nokia/sonic-buildimage that referenced this issue Aug 7, 2021
Update sonic-sairedis submodule and also update sonic-swss submodule as there are interdependent changes.

* src/sonic-sairedis 13474d1...bc58b0f (12):
  > Add gbsyncdmgrd; deprecate gbsyncd_startup.py (sonic-net#809)
  > Remove gbsyncd_start.sh (sonic-net#808)
  > [gbsyncd] Fix shebang in gbsyncd_startup.py; Make script executable (sonic-net#807)
  > [saiasiccmp] Add saiasiccmp tool to compare 2 asic views (sonic-net#791)
  > [configure] Add -Wno-psabi to remove "passing argument changed in GCC 7.1" (sonic-net#799)
  > Update FlexCounter.cpp, use m_pollInterval in MUTEX lock (sonic-net#797)
  > [vs] Add special warm boot logic to populate default attributes (sonic-net#796)
  > [ci]: add vstest (sonic-net#795)
  > [tests] Add macsec unittest (sonic-net#782)
  > [debian/control] libsairedis-dev depends on libzmq5-dev (sonic-net#794)
  > [ci]: use build template (sonic-net#793)
  > Rename duplicate file name (sonic-net#773)

* src/sonic-swss 0b0d24c...5adb73e (47):
  > Initialize system port type variable (sonic-net#1681)
  > [Dynamic Buffer Calc] Enhance the field checking in table handling (sonic-net#1680)
  > Handle the clear request for 'Q_SHARED_ALL' (sonic-net#1653)
  > [MuxOrch] FDB ageout safety check (sonic-net#1674)
  > Deactivate mirror session only when session status is true in updateLagMember (sonic-net#1666)
  > Revert "[buffermgr] Support maximum port headroom checking (sonic-net#1607)" (sonic-net#1675)
  > reduce severity of log to info in case of flush on non-existing member (sonic-net#1669)
  > Revert "[Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (sonic-net#1652)" (sonic-net#1676)
  > [Dynamic buffer calc] Bug fix: Remove PGs from an administratively down port. (sonic-net#1652)
  > [acl] Move ACL table constants to acltable.h (sonic-net#1671)
  > [nbrmgrd] added function to parse IP address from APP_DB (sonic-net#1672)
  > [MUX/PFCWD] Use in_ports for acls instead of seperate ACL table (sonic-net#1670)
  > [vog/systemlag] Voq lagid allocator (sonic-net#1603)
  > Add table descriptions for dynamic buffer calculation to the documents (sonic-net#1664)
  > [vstest/subintf] Add vs test case to validate processing sequence of APPL DB keys (sonic-net#1663)
  > Remove vxlanmgrd dependency on orchagent (sonic-net#1647)
  > Keep attribute order in bulk mode (sonic-net#1659)
  > [mux] VS test for neigh, route and fdb (sonic-net#1656)
  > [linksync] Netdev oper status determination using IFF_RUNNING (sonic-net#1568)
  > [portorch] parse on/off value from autoneg (sonic-net#1658)
  > [intfsorch] Create subport with the entry contains necessary attributes (sonic-net#1650)
  > [ci]: Purge swss before install (sonic-net#1654)
  > Update StateDB with error if state change failed, Update APP_DB in all state chg req (sonic-net#1662)
  > Added changes to handle dependency check in FdbSyncd and FpmSyncd for warm-boot (sonic-net#1556)
  > [synchronous mode] Add failure notification for SAI failures in synchronous mode (sonic-net#1596)
  > [acl] Enable VLAN ID qualifier for ACL rules (sonic-net#1648)
  > Updated PFCWD to use single ACL table for PFCWD and MUX (sonic-net#1620)
  > [orchagent] Increase SAI REDIS response timeout to support FW upgrade during init (Mellanox only). (sonic-net#1637)
  > [vstest/nhg]: use dvs_route fixture to make test_nhg more robust
  > [vstest]: add dvs_route fixture
  > [vstest/subintf] Update vs tests to validate physical port host interface vlan tag attribute (sonic-net#1634)
  > Remove useless header  in macsecorch (sonic-net#1628)
  > Add SAI_INGRESS_PRIORITY_GROUP_STAT_DROPPED_PACKETS counter, create new FlexCounter group (sonic-net#1600)
  > fixed unsupported resource issue (sonic-net#1641)
  > [test_virtual_chassis]: use wait_for to make test more robust (sonic-net#1640)
  > spell check fixes (sonic-net#1630)
  > [bufferorch] Handle NOT IMPLEMENTED status returned during set attr operation (sonic-net#1639)
  > [ci]: run vstest
  > [test_virtual_chassis]: use wait_for function to improve test robustness
  > [Mux] Neighbor handling based on FDB entry (sonic-net#1631)
  > [ci]: use build template (sonic-net#1633)
  > Log level change from ERR to INFO for fetch systemports issue (sonic-net#1632)
  > Migrate serdes programming to port serdes object (sonic-net#1611)
  > [tests] Remove legacy saiattributelist.h dependency (sonic-net#1608)
  > [buffermgr] Support maximum port headroom checking (sonic-net#1607)
  > Support shared headroom pool on top of dynamic buffer calculation (sonic-net#1581)
  > Fix the compiling errors in gcc9 (sonic-net#1621)
theasianpianist pushed a commit to theasianpianist/sonic-buildimage that referenced this issue Feb 5, 2022
…wn port. (sonic-net#1652)

Remove PGs from an administratively down port.
- Introduce a new state: PORT_ADMIN_DOWN which represents the port is administratively down.
- Remove all PGs when the port is shut down and re-add all configured PGs when port is started up
- Only record the new value but don't touch BUFFER_PG_TABLE if the following events come when a port is administratively down, a port's MTU, speed, or cable length is updated, a new PG is added to a port or an existing PG is removed from a port
- Optimize the port event handling flow since refreshPriorityGroupsForPort should be called only once in case more than one fields are updated
- Optimize the Lua plugin which calculates the buffer pool size according

Signed-off-by: Stephen Sun [email protected]

How I verified it
Run regression and vs test
theasianpianist pushed a commit to theasianpianist/sonic-buildimage that referenced this issue Feb 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants