Skip to content
This repository has been archived by the owner on May 22, 2023. It is now read-only.

Commit

Permalink
Temp fix for the problem with failing jobs - move cache instead of co…
Browse files Browse the repository at this point in the history
…pying

We've seen a number of workflows failing with `No space left on device`
error (e.g.
https://github.com/keep-network/keep-ecdsa/actions/runs/924123923).
The problem was a combination of a big cache size and significant memory
allocation after building Docker image.
According to moby/buildkit#1850:
"At the moment caches are copied over the existing cache so it keeps
growing".
As a temporary fix (until issue gets fixed by GH), we introduce a step
that moves the cache (similarily to how it's described in
moby/buildkit#1896).

The solution may not keep us from failures forever, as we still may
hit the problems with the full memory if the size of the Docker image
or cache increases (see this FD discussion:
https://www.flowdock.com/app/cardforcoin/tech/threads/Clml1_lI4HYRKrr3N_vcSInDn6A).
  • Loading branch information
michalinacienciala authored and nkuba committed Jun 11, 2021
1 parent 0183e6b commit d848b7f
Show file tree
Hide file tree
Showing 3 changed files with 34 additions and 4 deletions.
14 changes: 12 additions & 2 deletions .github/workflows/client-celo.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ jobs:
HOST_CHAIN=celo
load: true # load image to local registry to use it in next steps
cache-from: type=local,src=/tmp/.buildx-celo-cache
cache-to: type=local,dest=/tmp/.buildx-celo-cache
cache-to: type=local,dest=/tmp/.buildx-celo-cache-new

- name: Run Go tests
run: |
Expand Down Expand Up @@ -111,4 +111,14 @@ jobs:
HOST_CHAIN=celo
REVISION=${{ github.sha }}
# VERSION= ? TODO: Configure version, sample: 1.7.6
push: ${{ github.event_name == 'workflow_dispatch' }}
push: ${{ github.event_name == 'workflow_dispatch' }}

- # Temp fix - move cache instead of copying (added below step and
# modified value of `cache-to`).
# https://github.com/docker/build-push-action/issues/252
# https://github.com/moby/buildkit/issues/1896
# Without the change some jobs were failing with `no space left on device`
name: Move cache
run: |
rm -rf /tmp/.buildx-celo-cache
mv /tmp/.buildx-celo-cache-new /tmp/.buildx-celo-cache
12 changes: 11 additions & 1 deletion .github/workflows/client-ethereum.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ jobs:
tags: go-build-env-ethereum
load: true # load image to local registry to use it in next steps
cache-from: type=local,src=/tmp/.buildx-ethereum-cache
cache-to: type=local,dest=/tmp/.buildx-ethereum-cache
cache-to: type=local,dest=/tmp/.buildx-ethereum-cache-new

- name: Run Go tests
run: |
Expand Down Expand Up @@ -110,6 +110,16 @@ jobs:
# VERSION= ? TODO: Configure version, sample: 1.7.6
push: ${{ github.event_name == 'workflow_dispatch' }}

- # Temp fix - move cache instead of copying (added below step and
# modified value of `cache-to`).
# https://github.com/docker/build-push-action/issues/252
# https://github.com/moby/buildkit/issues/1896
# Without the change some jobs were failing with `no space left on device`
name: Move cache
run: |
rm -rf /tmp/.buildx-ethereum-cache
mv /tmp/.buildx-ethereum-cache-new /tmp/.buildx-ethereum-cache
- name: Notify CI about completion of the workflow
if: github.event_name == 'workflow_dispatch'
uses: keep-network/notify-workflow-completed@v1
Expand Down
12 changes: 11 additions & 1 deletion .github/workflows/contracts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,17 @@ jobs:
revision=${{ github.sha }}
push: true
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache-new

- # Temp fix - move cache instead of copying (added below step and
# modified value of `cache-to`).
# https://github.com/docker/build-push-action/issues/252
# https://github.com/moby/buildkit/issues/1896
# Without the change some jobs were failing with `no space left on device`
name: Move cache
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
- name: Notify CI about completion of the workflow
uses: keep-network/notify-workflow-completed@v1
Expand Down

0 comments on commit d848b7f

Please sign in to comment.