Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAP-1781 Cleanup Access Logs more than 3 months old #2408

Merged
merged 6 commits into from
Nov 15, 2024

Conversation

tobyprivett
Copy link
Contributor

@tobyprivett tobyprivett commented Nov 15, 2024

Jira link

https://dsdmoj.atlassian.net/browse/MAP-1781

What?

Adds a daily cronjob to delete stale Access Logs.
DELETE can be a very expensive operation, so we do this in batches of 1000.

Why?

200k Access Logs are created every day.
We use them for debugging production issues, so it should be okay to remove anything older than three months.

@tobyprivett tobyprivett marked this pull request as ready for review November 15, 2024 10:51
@tobyprivett tobyprivett requested a review from a team as a code owner November 15, 2024 10:51
@jimbali
Copy link
Contributor

jimbali commented Nov 15, 2024

I think three months is too short tbh, I've had to debug issues with older moves before

jimbali
jimbali previously approved these changes Nov 15, 2024
AccessLog.where('timestamp < ?', cutoff_date).limit(1000).delete_all
end
end
end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest possibly adding a test for this task but it's up to you

@danbenton-mojdt
Copy link
Contributor

I think three months is too short tbh, I've had to debug issues with older moves before

Agree, minimum 12 months imo

@tobyprivett tobyprivett merged commit f4b8d39 into main Nov 15, 2024
8 checks passed
@tobyprivett tobyprivett deleted the MAP-1781-access-logs branch November 15, 2024 12:19
Copy link

sentry-io bot commented Nov 17, 2024

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ ActiveRecord::ConnectionNotEstablished: connection to server at "172.20.82.180", port 5432 failed: Operation timed out (ActiveRecord::Con... access_logs:cleanup View Issue
  • ‼️ SignalException: SIGTERM (SignalException) access_logs:cleanup View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants