Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add cache cleanup option to local cache exporter #1896

Open
MarcelCoding opened this issue Dec 14, 2020 · 8 comments
Open

Add cache cleanup option to local cache exporter #1896

MarcelCoding opened this issue Dec 14, 2020 · 8 comments
Labels
help wanted kind/enhancement

Comments

@MarcelCoding
Copy link

MarcelCoding commented Dec 14, 2020 •
edited
Loading

Implemented an option to remove old versions of a newly created image and images older than a specified time.

Reason/Usecase: docker/build-push-action#252
My temporarily solution: docker/build-push-action#252 (comment)
@MarcelCoding MarcelCoding changed the title Add option for cache cleaning Add cache cleanup option Dec 14, 2020
@tonistiigi
Copy link
Member

BuildKit doesn't have an image store. For cleaning up build cache there is buildctl prune (docker buildx prune)

@crazy-max
Copy link
Member

@tonistiigi It's about the local cache exporter. Atm caches are copied over the existing cache so it keeps growing (#1850).

@tonistiigi tonistiigi changed the title Add cache cleanup option Add cache cleanup option to local cache exporter Dec 16, 2020
@CMonoceros
Copy link

Is there any progress on this option?

@Niek

This comment has been minimized.

Sign in to view
michalinacienciala added a commit to keep-network/keep-ecdsa that referenced this issue Jun 10, 2021
@michalinacienciala
Temp fix for the problem with failing jobs - move cache instead of co…
cb99021 
…pying

We've seen a number of workflows failing with `No space left on device`
error (e.g.
https://github.com/keep-network/keep-ecdsa/actions/runs/924123923).
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduce a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896).

The solution may not keep us from failures forever, as we still may
hit the problems with the full memory if the size of the Docker image
or cache increases (see this FD discussion:
https://www.flowdock.com/app/cardforcoin/tech/threads/Clml1_lI4HYRKrr3N_vcSInDn6A).
@michalinacienciala michalinacienciala mentioned this issue Jun 10, 2021
Merged
nkuba pushed a commit to keep-network/keep-ecdsa that referenced this issue Jun 15, 2021
@michalinacienciala @nkuba
Temp fix for the problem with failing jobs - move cache instead of co…
d848b7f 
…pying

We've seen a number of workflows failing with `No space left on device`
error (e.g.
https://github.com/keep-network/keep-ecdsa/actions/runs/924123923).
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduce a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896).

The solution may not keep us from failures forever, as we still may
hit the problems with the full memory if the size of the Docker image
or cache increases (see this FD discussion:
https://www.flowdock.com/app/cardforcoin/tech/threads/Clml1_lI4HYRKrr3N_vcSInDn6A).
michalinacienciala added a commit to keep-network/keep-core that referenced this issue Jun 16, 2021
@michalinacienciala
Move cache instead of copying
251aa89 
In `keep-ecdsa` we've seen a number of workflows failing with `No space
lefton device` error.
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduced a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896) instead of copying.
Although we havent seen the problems with cache size yet in other
projects than `keep-ecdsa`, we're applying the solution across the
repositories to decrease the likelyhood of encountering problem in the
future.
@michalinacienciala michalinacienciala mentioned this issue Jun 16, 2021
Merged
michalinacienciala added a commit to keep-network/tbtc that referenced this issue Jun 16, 2021
@michalinacienciala
Move cache instead of copying
845994b 
In `keep-ecdsa` we've seen a number of workflows failing with `No space
lefton device` error.
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduced a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896) instead of copying.
Although we havent seen the problems with cache size yet in other
projects than `keep-ecdsa`, we're applying the solution across the
repositories to decrease the likelyhood of encountering problem in the
future.
@michalinacienciala michalinacienciala mentioned this issue Jun 16, 2021
Merged
michalinacienciala added a commit to keep-network/tbtc-dapp that referenced this issue Jun 16, 2021
@michalinacienciala
Move cache instead of copying
d88799b 
In `keep-ecdsa` we've seen a number of workflows failing with `No space
lefton device` error.
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduced a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896) instead of copying.
Although we havent seen the problems with cache size yet in other
projects than `keep-ecdsa`, we're applying the solution across the
repositories to decrease the likelyhood of encountering problem in the
future.
@michalinacienciala michalinacienciala mentioned this issue Jun 16, 2021
Merged
@crazy-max crazy-max mentioned this issue Jul 29, 2021
Open
@lexfrei
Copy link

lexfrei commented Oct 13, 2021

Any news?

mdegat01 added a commit to mdegat01/addon-hedgedoc that referenced this issue Nov 10, 2021
@mdegat01
Clear build cache during CI
5f046fc 
Appears there is an issue where the build cache grows unbounded until it hits Github's limit (docker/build-push-action#252, moby/buildkit#1896). Clear the cache after builds to prevent this
@mdegat01 mdegat01 mentioned this issue Nov 10, 2021
Merged
mdegat01 added a commit to mdegat01/addon-promtail that referenced this issue Nov 10, 2021
@mdegat01
Clear build cache during CI
a81c133 
Appears there is an issue where the build cache grows unbounded until it hits Github's limit (docker/build-push-action#252, moby/buildkit#1896). Clear the cache after builds to prevent this
@mdegat01 mdegat01 mentioned this issue Nov 10, 2021
Merged
mdegat01 added a commit to mdegat01/addon-sharry that referenced this issue Nov 10, 2021
@mdegat01
Clear build cache during CI
2d8fb53 
Appears there is an issue where the build cache grows unbounded until it hits Github's limit (docker/build-push-action#252, moby/buildkit#1896). Clear the cache after builds to prevent this
@mdegat01 mdegat01 mentioned this issue Nov 10, 2021
Merged
mdegat01 added a commit to mdegat01/addon-loki that referenced this issue Nov 10, 2021
@mdegat01
Clear build cache during CI
0d4771d 
Appears there is an issue where the build cache grows unbounded until it hits Github's limit (docker/build-push-action#252, moby/buildkit#1896). Clear the cache after builds to prevent this.
@mdegat01 mdegat01 mentioned this issue Nov 10, 2021
Merged
mdegat01 added a commit to mdegat01/promtail-journal that referenced this issue Nov 10, 2021
@mdegat01
Clear build cache during CI
be0a152 
Appears there is an issue where the build cache grows unbounded until it hits Github's limit (docker/build-push-action#252, moby/buildkit#1896). Clear the cache after builds to prevent this.
@mdegat01 mdegat01 mentioned this issue Nov 10, 2021
Merged
@br3ndonland br3ndonland mentioned this issue Jul 9, 2022
Merged
4 tasks
@crazy-max crazy-max mentioned this issue Sep 14, 2022
Merged
@gotti gotti mentioned this issue Nov 16, 2022
Closed
@jedevc jedevc mentioned this issue Jun 14, 2023
Closed
@beastawakens beastawakens mentioned this issue Jul 20, 2023
Closed
@demartis
Copy link

It's been over 2 years, is there any news about this issue?

@rreckonerr
Copy link

@acidobinario
Copy link

LGTM 👍 💯

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted kind/enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@Niek @tonistiigi @lexfrei @crazy-max @acidobinario @demartis @CMonoceros @rreckonerr @MarcelCoding