Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hack] try to kill old gobble service #18

Closed
wants to merge 1 commit into from

Conversation

hamima-halim
Copy link
Contributor

Looks like an old version of the gobble service is still running. You can tell this by looking at old journalctl logs and seeing:

Dec 12 17:11:00 ip-172-31-45-145 systemd[1]: gobble.service: Scheduled restart job, restart counter is at 94274.
Dec 12 17:11:00 ip-172-31-45-145 systemd[1]: Stopped gobble.
Dec 12 17:11:00 ip-172-31-45-145 systemd[1]: gobble.service: Consumed 1.036s CPU time.
Dec 12 17:11:00 ip-172-31-45-145 systemd[1]: Started gobble.
// traceback......
Dec 12 17:11:01 ip-172-31-45-145 poetry[456316]:     archives_df = pd.read_csv(GTFS_ARCHIVES_PREFIX / GTFS_ARCHIVES_FILENAME)
// traceback.......
Dec 12 17:11:01 ip-172-31-45-145 systemd[1]: gobble.service: Main process exited, code=exited, status=1/FAILURE
Dec 12 17:11:01 ip-172-31-45-145 systemd[1]: gobble.service: Failed with result 'exit-code'.
Dec 12 17:11:01 ip-172-31-45-145 systemd[1]: gobble.service: Consumed 1.082s CPU time.
Dec 12 17:11:04 ip-172-31-45-145 sshd[456320]: Invalid user postmaster from 43.134.63.194 port 56072
Dec 12 17:11:04 ip-172-31-45-145 sshd[456320]: Received disconnect from 43.134.63.194 port 56072:11: Bye Bye [preauth]
Dec 12 17:11:04 ip-172-31-45-145 sshd[456320]: Disconnected from invalid user postmaster 43.134.63.194 port 56072 [preauth]
Dec 12 17:11:06 ip-172-31-45-145 systemd[1]: gobble.service: Scheduled restart job, restart counter is at 94275.
Dec 12 17:11:06 ip-172-31-45-145 systemd[1]: Stopped gobble.

This service is still referring to old code! I've tried killing the old process via systemctl stop and a couple of other jank commands but to no avail. This pr pushes up some nonsense a la https://serverfault.com/questions/1079789/systemd-limit-the-number-of-times-a-service-can-restart to see if the old service will die if I tweak some systemd settings. This should, in theory, limit the number of restarts the service can have in a day before dying for good.

Copy link
Contributor

@mathcolo mathcolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed this on Slack- OOM situation on the server was (probably) making for clobbered systemd logging. Perhaps this could use a 30s RestartSec just to reduce the high counter- but maybe even that isn't super necessary?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants