Work In Progress: Add central LRU cache that adjusts to available memory #7426
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bep
force-pushed
the
cache-bound
branch
4 times, most recently
from
5ce19a0 to
c6d29f7
Compare
bep
force-pushed
the
cache-bound
branch
9 times, most recently
from
c2635a7 to
8a38818
Compare
bep
changed the title
bep
force-pushed
the
cache-bound
branch
2 times, most recently
from
eab9cf2 to
fbc75f2
Compare
|
Note to self: Rember/consider |
bep
force-pushed
the
cache-bound
branch
2 times, most recently
from
3209412 to
ac391f9
Compare
bep
changed the title
bep
force-pushed
the
cache-bound
branch
2 times, most recently
from
c00a644 to
e4e1027
Compare
Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage. One failing example would be loading lots of big JSON data files and unmarshal them via `transform.Unmarshal`. This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory. Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the `HUGO_MEMORYLIMIT` environment variable (a float value representing Gigabytes). A natural next step after this would be to use this cache for `.Content`. Fixes gohugoio#7425 Fixes gohugoio#7437 Fixes gohugoio#7436 Fixes gohugoio#7882 Updates gohugoio#7544
|
Closing, superseded by #11894. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage.
One failing example would be loading lots of big JSON data files and unmarshal them via
transform.Unmarshal.This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory.
Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the
HUGO_MEMORYLIMITenvironment variable (a float value representing Gigabytes).A natural next step after this would be to use this cache for
.Content.Fixes #7425
Fixes #7437
Fixes #7436
Updates #7544