Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve memory usage when purging old logs #15

Merged
merged 1 commit into from
Jun 17, 2024

Conversation

matthewhilton
Copy link
Contributor

@matthewhilton matthewhilton commented Jun 17, 2024

Closes #14

Changes

  • General cleanup (phpdoc, code style)
  • Made scheduled task spawn adhoc task to do purge, so it is more robust and doesn't block other scheduled tasks if it fails.
  • Improve query:
    • Loop and query only 1k records per loop.
    • Only query the id since that's all it uses
    • Pass through 1k records to delete() instead of 1 at a time. This massively improves the speed at which it runs (since it can delete records in chunks of 1k instead of just 1 at a time)

Comparison
Scenario: Testing with ~450k rows in mdl_local_mail_log each with an attachment set.

Before:

  • Started running, but eventually died:
Deleting log entries older than 7 days.
Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /var/www/site/lib/dml/pgsql_native_moodle_database.php on line 1040

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /var/www/site/admin/tool/excimer/classes/sample_set.php on line 84

After:

  • Completes successfully. Takes ~2-3 minutes to process the entire 450k records.
  • Uses ~10mb of memory (according to excimer)

@matthewhilton matthewhilton changed the base branch from MOODLE_35_STABLE to MOODLE_310_STABLE June 17, 2024 01:28
@matthewhilton matthewhilton force-pushed the 14-improve-mem-usage branch 5 times, most recently from 630a4fa to db82b12 Compare June 17, 2024 03:52
@matthewhilton matthewhilton marked this pull request as ready for review June 17, 2024 03:53
@matthewhilton matthewhilton force-pushed the 14-improve-mem-usage branch 2 times, most recently from 69167db to 37e5d75 Compare June 17, 2024 04:04
@matthewhilton matthewhilton changed the title 14 improve mem usage Improve memory usage when purging old logs Jun 17, 2024
@matthewhilton matthewhilton merged commit fcc6018 into MOODLE_310_STABLE Jun 17, 2024
@danmarsden danmarsden deleted the 14-improve-mem-usage branch June 18, 2024 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make purge query more efficient
2 participants