Managing minion cache size

[jdonofrio728]

Description of Issue/Question

I'm running into an issue where my /var filesystem is completely filling up by the cache used by the salt-call command. I find myself running commands like below just to get the state to apply:

applications-clear-cache:
  cmd.run:
    - name: salt-call --local saltutil.clear_cache

I am downloading many large files from an artifact repository during the process.

Adding more disk space doesn't seem like the best solution for this. What configuration options do I have to keep the cache from filling up the disk?

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)
applications-installer:
archive.extracted:
- name: /sfs/tmp/osb-installer
- source: http://server:port/path/to/very/large/file
- skip_verify: True
- enforce_toplevel: False

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)
Run

#: salt-call --local state.highstate

To apply a state that downloads enough files to fill up the /var filesystem.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

[vagrant@ets-01 ~]$ salt-call --version
salt-call 2016.11.3 (Carbon)

[Ch3LL]

I believe that is by design to pull down the file and then extract the file form the cache. Are you suggesting a way to bypass the pull down of hte file to the cache to be able to extract the file?

Or are the files staying in the cache after you extract them?

[jdonofrio728]

I'm more referring that the files are staying in cache. Im my case, I have ~5gb available on /var/cache, and while salt-call is running it is pulling dozens of files from an artifact repository form 50mb to ~1gb. After the filesystem fills up, rather than clearing old files from the cache, the state files running last fail to apply. This basically means that I need to maintain space on the filesystem for the file to pull down and stay, and space for its destination.

[sokratisg]

You could setup a periodic schedule, say every week or so, to execute saltutil.clear_cache or even better a cmd.run to check for files bigger than X MB and older than Y days.

[jdonofrio728]

I could, but I don't think that ultimately solves the problem. I would think the salt should be smart enough to manage its own cache, and not fail just because it managed to file it up from previous run's or states.

[EvaSDK]

The minion cache should definitely have some way of being cleaned automatically, just like master's jobs cache. Otherwise the cache grows indefinitely when updating states and it is not sane to have an agent not paying attention to its footprint.

[EvaSDK]

This is actually already filled as #6759 and #34369.

[Ch3LL]

thanks for linking those issues @EvaSDK

one thing to mention is that there is a keep_source argument in some of the file, archive states that will make sure its deleted. Might be an easy solution to allow that to also be a configuration setting. Also to ultimately solve this problem, would be good to add module/runner as that issue points to only clear the fileserver and then add a a config option that will run that on a schedule.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.