Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a better (and consolidated) cache design #5404

Closed
bep opened this issue Nov 5, 2018 · 3 comments · Fixed by #5421
Closed

Add a better (and consolidated) cache design #5404

bep opened this issue Nov 5, 2018 · 3 comments · Fixed by #5421
Labels
Enhancement Outdated
Milestone
v0.52

Comments

@bep
Copy link
Member

bep commented Nov 5, 2018 •
edited
Loading

Several of the things I want to do in Hugo (pages from data sources etc.) needs a cache to ... survive.

We do caching on several levels in Hugo, but the file cache(s) are spread around and you cannot configure much more than the directory they live in.

I suggest we add a concept of named caches with sensible default configuration. Configuration options:

  • Path
  • TTL (time to live), value in seconds >= 0. 0 Effectively means "no cache", -1 means forever.

The default value will, in most cases, be assigned a directory below cacheDir, so we need to do some work in that area, too, if there is no value set for that option and you're running on a CI server (Netlify etc.).

Some defaults could look like this:

[caches]
[caches.image]
path = ":resourceDir/_gen/images"
ttl=-1
[caches]
[caches.getJSON]
path = ":cacheDir/getjson"
ttl=600

/cc @kaushalmodi @regisphilibert
@bep bep added the Proposal label Nov 5, 2018
@kaushalmodi
Copy link
Contributor

Looks good to me, but I don't have enough knowledge in this area to give any constructive criticism.

Copying @paulcmal for comments if he is still interested in this or has time.

Ref: https://discourse.gohugo.io/t/configurable-cache-ttl-for-remote-content/10220

bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
bd60726 
Fixes gohugoio#5404
@bep bep mentioned this issue Nov 8, 2018
Merged
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
7d4a349 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
ca76210 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
aa4fd15 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
7a39691 
Fixes gohugoio#5404
@bep bep added this to the v0.52 milestone Nov 8, 2018
@bep bep added Enhancement and removed Proposal labels Nov 8, 2018
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
6cadf86 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
6626292 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
bf4b990 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
f400ffc 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
83aea1b 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
32aa370 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
d26056a 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
9d2524f 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 8, 2018
@bep
Add a file cache
187e8a4 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 9, 2018
@bep
Add a file cache
5a0c758 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 9, 2018
@bep
Add a file cache
0e1d77e 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 9, 2018
@bep
Add a file cache
b14979d 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 9, 2018
@bep
Add a file cache
cd31ae6 
Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 13, 2018
@bep
Add a consolidated file cache
654980a 
This commits reworks how file caching is performed in Hugo. Now there is only one way, and it can be configured.

This is the default configuration:

```toml
[caches]
[caches.getjson]
dir = ":cacheDir"
maxAge = -1
[caches.getcsv]
dir = ":cacheDir"
maxAge = -1
[caches.images]
dir = ":resourceDir/_gen"
maxAge = -1
[caches.assets]
dir = ":resourceDir/_gen"
maxAge = -1
```

You can override any of these cache setting in your own `config.toml`.

The placeholders explained:

`:cacheDir`: This is the value of the `cacheDir` config option if set (can also be set via OS env variable `HUGO_CACHEDIR`). It will fall back to `/opt/build/cache/hugo_cache/` on Netlify, or a `hugo_cache` directory below the OS temp dir for the others.
`:resourceDir`: This is the value of the `resourceDir` config option.

`maxAge` is the time in seconds before a cache entry will be evicted, -1 means forever and 0 effectively turns that particular cache off.

This means that if you run your builds on Netlify, all caches configured with `:cacheDir` will be saved and restored on the next build. For other CI vendors, please read their documentation. For an CircleCI example, see https://github.com/bep/hugo-sass-test/blob/6c3960a8f4b90e8938228688bc49bdcdd6b2d99e/.circleci/config.yml

Fixes gohugoio#5404
bep added a commit to bep/hugo that referenced this issue Nov 13, 2018
@bep
Add a consolidated file cache
15e5fdb 
This commits reworks how file caching is performed in Hugo. Now there is only one way, and it can be configured.

This is the default configuration:

```toml
[caches]
[caches.getjson]
dir = ":cacheDir"
maxAge = -1
[caches.getcsv]
dir = ":cacheDir"
maxAge = -1
[caches.images]
dir = ":resourceDir/_gen"
maxAge = -1
[caches.assets]
dir = ":resourceDir/_gen"
maxAge = -1
```

You can override any of these cache setting in your own `config.toml`.

The placeholders explained:

`:cacheDir`: This is the value of the `cacheDir` config option if set (can also be set via OS env variable `HUGO_CACHEDIR`). It will fall back to `/opt/build/cache/hugo_cache/` on Netlify, or a `hugo_cache` directory below the OS temp dir for the others.
`:resourceDir`: This is the value of the `resourceDir` config option.

`maxAge` is the time in seconds before a cache entry will be evicted, -1 means forever and 0 effectively turns that particular cache off.

This means that if you run your builds on Netlify, all caches configured with `:cacheDir` will be saved and restored on the next build. For other CI vendors, please read their documentation. For an CircleCI example, see https://github.com/bep/hugo-sass-test/blob/6c3960a8f4b90e8938228688bc49bdcdd6b2d99e/.circleci/config.yml

Fixes gohugoio#5404
@bep bep mentioned this issue Nov 13, 2018
Closed
bep added a commit to bep/hugo that referenced this issue Nov 13, 2018
@bep
Add a consolidated file cache
64b9723 
This commits reworks how file caching is performed in Hugo. Now there is only one way, and it can be configured.

This is the default configuration:

```toml
[caches]
[caches.getjson]
dir = ":cacheDir"
maxAge = -1
[caches.getcsv]
dir = ":cacheDir"
maxAge = -1
[caches.images]
dir = ":resourceDir/_gen"
maxAge = -1
[caches.assets]
dir = ":resourceDir/_gen"
maxAge = -1
```

You can override any of these cache setting in your own `config.toml`.

The placeholders explained:

`:cacheDir`: This is the value of the `cacheDir` config option if set (can also be set via OS env variable `HUGO_CACHEDIR`). It will fall back to `/opt/build/cache/hugo_cache/` on Netlify, or a `hugo_cache` directory below the OS temp dir for the others.
`:resourceDir`: This is the value of the `resourceDir` config option.

`maxAge` is the time in seconds before a cache entry will be evicted, -1 means forever and 0 effectively turns that particular cache off.

This means that if you run your builds on Netlify, all caches configured with `:cacheDir` will be saved and restored on the next build. For other CI vendors, please read their documentation. For an CircleCI example, see https://github.com/bep/hugo-sass-test/blob/6c3960a8f4b90e8938228688bc49bdcdd6b2d99e/.circleci/config.yml

Fixes gohugoio#5404
@bep bep closed this as completed in #5421 Nov 13, 2018
bep added a commit that referenced this issue Nov 13, 2018
@bep
Add a consolidated file cache
f7aeaa6 
This commits reworks how file caching is performed in Hugo. Now there is only one way, and it can be configured.

This is the default configuration:

```toml
[caches]
[caches.getjson]
dir = ":cacheDir"
maxAge = -1
[caches.getcsv]
dir = ":cacheDir"
maxAge = -1
[caches.images]
dir = ":resourceDir/_gen"
maxAge = -1
[caches.assets]
dir = ":resourceDir/_gen"
maxAge = -1
```

You can override any of these cache setting in your own `config.toml`.

The placeholders explained:

`:cacheDir`: This is the value of the `cacheDir` config option if set (can also be set via OS env variable `HUGO_CACHEDIR`). It will fall back to `/opt/build/cache/hugo_cache/` on Netlify, or a `hugo_cache` directory below the OS temp dir for the others.
`:resourceDir`: This is the value of the `resourceDir` config option.

`maxAge` is the time in seconds before a cache entry will be evicted, -1 means forever and 0 effectively turns that particular cache off.

This means that if you run your builds on Netlify, all caches configured with `:cacheDir` will be saved and restored on the next build. For other CI vendors, please read their documentation. For an CircleCI example, see https://github.com/bep/hugo-sass-test/blob/6c3960a8f4b90e8938228688bc49bdcdd6b2d99e/.circleci/config.yml

Fixes #5404
bep added a commit that referenced this issue Nov 13, 2018
@bep
docs: Document the new file cache
abeeff1 
See #5404
@dependencies dependencies bot mentioned this issue Nov 28, 2018
Closed
@thombruce thombruce mentioned this issue Jan 12, 2019
Closed
@paulcmal
Copy link
Contributor

paulcmal commented Mar 6, 2019

Hey, sorry for taking time to answer! And thank you @bep for taking the time to consider and start implementing a proper caching policy :)

Well my comments from the discourse still stand. To summarize, in my opinion:

  • the site-wide config should be able (via CLI params or config-file settings) to ignore cache entirely and to set a maximum caching time for resources (so you can ensure site-wide that no remote resource is more than N-days stale)
  • we should have template-side control of caching policy as well, so some resources (such as a monitoring feed) may explicitely ask not to be cached, while others may explicit for how long they wish to be kept

Would that make sense to you?

@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 20, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Enhancement Outdated
Projects
None yet
Milestone
v0.52
Development

Successfully merging a pull request may close this issue.

Add a file cache bep/hugo
3 participants
@bep @paulcmal @kaushalmodi