Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: caches keep growing #1850

Closed
tonistiigi opened this issue Nov 26, 2020 · 0 comments · Fixed by #1857
Closed

ci: caches keep growing #1850

tonistiigi opened this issue Nov 26, 2020 · 0 comments · Fixed by #1857

Comments

@tonistiigi
Copy link
Member

Because caches are copied over the existing cache so it keeps growing and getting slower and less effective.

michalinacienciala added a commit to keep-network/keep-ecdsa that referenced this issue Jun 10, 2021
…pying

We've seen a number of workflows failing with `No space left on device`
error (e.g.
https://github.com/keep-network/keep-ecdsa/actions/runs/924123923).
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduce a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896).

The solution may not keep us from failures forever, as we still may
hit the problems with the full memory if the size of the Docker image
or cache increases (see this FD discussion:
https://www.flowdock.com/app/cardforcoin/tech/threads/Clml1_lI4HYRKrr3N_vcSInDn6A).
nkuba pushed a commit to keep-network/keep-ecdsa that referenced this issue Jun 15, 2021
…pying

We've seen a number of workflows failing with `No space left on device`
error (e.g.
https://github.com/keep-network/keep-ecdsa/actions/runs/924123923).
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduce a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896).

The solution may not keep us from failures forever, as we still may
hit the problems with the full memory if the size of the Docker image
or cache increases (see this FD discussion:
https://www.flowdock.com/app/cardforcoin/tech/threads/Clml1_lI4HYRKrr3N_vcSInDn6A).
michalinacienciala added a commit to keep-network/keep-core that referenced this issue Jun 16, 2021
In `keep-ecdsa` we've seen a number of workflows failing with `No space
lefton device` error.
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduced a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896) instead of copying.
Although we havent seen the problems with cache size yet in other
projects than `keep-ecdsa`, we're applying the solution across the
repositories to decrease the likelyhood of encountering problem in the
future.
michalinacienciala added a commit to keep-network/tbtc that referenced this issue Jun 16, 2021
In `keep-ecdsa` we've seen a number of workflows failing with `No space
lefton device` error.
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduced a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896) instead of copying.
Although we havent seen the problems with cache size yet in other
projects than `keep-ecdsa`, we're applying the solution across the
repositories to decrease the likelyhood of encountering problem in the
future.
michalinacienciala added a commit to keep-network/tbtc-dapp that referenced this issue Jun 16, 2021
In `keep-ecdsa` we've seen a number of workflows failing with `No space
lefton device` error.
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduced a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896) instead of copying.
Although we havent seen the problems with cache size yet in other
projects than `keep-ecdsa`, we're applying the solution across the
repositories to decrease the likelyhood of encountering problem in the
future.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant