Skaffold rebuilds all images on every run (GitHub Action) - also the unaffected ones

[NixBiks]

I wish to have a GitHub Action that deploys all services in GKE if the service has been changed (either in manifest, source code or dependency). I wish that the action uses cache for the building.

The action looks like this

name: Deploy to staging
on: 
  push:
    branches:
      - develop
    paths:
      - "k8s-manifest/**/*"
      - "lib/**"
      - "services/**"
      - "skaffold.yaml"

env:
  PROJECT_ID: ${{ secrets.STAGING_PROJECT }}
  GKE_CLUSTER: parallax-cluster
  GKE_ZONE: europe-west3-a
  SKAFFOLD_DEFAULT_REPO: gcr.io/${{ secrets.STAGING_PROJECT }}/parallax
  SKAFFOLD_PROFILE: staging

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository on develop branch
        uses: actions/checkout@v2
        with:
          ref: develop
      - name: Install gcloud
        uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
        with:
          version: '309.0.0'
          service_account_key: ${{ secrets.STAGING_SA_BUILD_AND_DEPLOY }}
          project_id: ${{ secrets.STAGING_PROJECT }}
      - name: Install kustomize, kubectl and skaffold
        uses: daisaru11/setup-cd-tools@v1
        with:
          kubectl: '1.19.2'
          kustomize: '3.8.4'
          skaffold: '1.14.0'
      - name: Configure docker
        run: |
          gcloud --quiet auth configure-docker
      - name: Connect to cluster
        run: |
          gcloud container clusters get-credentials "$GKE_CLUSTER" --zone "$GKE_ZONE"
      - name: Build and deploy to cluster
        run: |
          skaffold run \
          -l skaffold.dev/run-id=LEAVE_UNCHANGED

while my skaffold.yaml looks like this

apiVersion: skaffold/v2alpha2
kind: Config
build:
  local:
    concurrency: 0
  artifacts:
    - image: python-backend
      context: .
      docker:
        dockerfile: services/backend/Dockerfile
    - image: horizon-worker
      context: .
      docker:
        dockerfile: services/horizon_worker/Dockerfile
    - image: react-frontend
      context: services/frontend

profiles:
  - name: staging
    deploy:
      kustomize:
        path: k8s-manifest/staging
  - name: production
    deploy:
      kustomize:
        path: k8s-manifest/production
  - name: local
    deploy:
      kustomize:
        path: k8s-manifest/local
    activation:
      - kubeContext: minikube

Expected behavior

I'd expect the above action to only rebuild affected images

Actual behavior

The above action rebuilds all images - also the ones that already exist in the default repo and which hasn't had any changes. I'd expect that skaffold run would realize that the unaffected images wouldn't be rebuild (works if I do skaffold run from my local machine).

I have considered setting up a self-hosted runner which always will have the cache available but I imagine that my task is very common for the community so I'm probably just missing something essential here to do. What is the best practice?

[noobG]

Dealing with the same thing here

The easy (but bad) solution is just to tag all images with :latest, e.g.:

build:
    tagPolicy:
        sha256: {}

...and then using cacheFrom: image:latest

This will override your past images and loses the commit identifier, but it does make caching work - this is a non-starter for me, so I tried:

1. Manually fetching my images and then using variable substitution to dynamically add the image tag to cacheFrom, e.g.:

docker:
 dockerfile: redacted
 cacheFrom:
    - eu.gcr.io/redacted/image:{{.CACHED_IMAGE_TAG}}

But it's not a templated field, so it can't perform the variable substitution cries in wasted time

2. What I will probably do next: manually add the :latest tag to all my images immediately after running skaffold; that way I can point cacheFrom to image:latest, while keeping the commit tags intact, e.g:

# For each image deployed by skaffold, do:
LAST_IMAGE=$(gcloud container images list-tags eu.gcr.io/**project_id**/**image** --limit=1 --format="get(tags)" | grep '[^;]*$' -o)

gcloud container images add-tag eu.gcr.io/**project_id**/**image**:$LAST_IMAGE eu.gcr.io/**project_id**/**image**:latest

Doesn't feel great but it should work in theory - hopefully someone has a better suggestion?

[StarpTech]

I run skaffold run -v debug in debug mode and found some interesting logs.

2020-10-02T23:16:13.7729379Z time="2020-10-02T23:16:13Z" level=debug msg="Could not import artifact from Docker, building instead (import of missing images disabled)"

The whole issue can be resolved by setting tryImportMissing 😄

apiVersion: skaffold/v2beta8
kind: Config
build:
  tagPolicy:
    sha256: {}
  local:
    concurrency: 0
    tryImportMissing: true

Why isn't it true by default? We should also improve the docs around local caching.

[briandealwis]

tryImportMissing is disabled by default as it can be faster to rebuild than download, especially if you have a slow connection to your registry.

I think this issue will be fixed with #4850 providing you to save and restore your Skaffold cache file. Normally it's saved in ~/.skaffold/cache but it can be set with the --cache-file path/to/cache/file.

[NixBiks]

Great with the tryImportMissing!

How do you suggest to use cache file together with GitHub Actions for instance?

[briandealwis]

It looks like the cache action would work?

[briandealwis]

@mr-bjerre did you have a chance to try the cache-action?

[StarpTech]

Try this. Keep in mind that this doesn't work for local builds. If you delete images from your registry the cache is no longer stable.

- name: Cache skaffold image builds & config
  uses: actions/cache@v2
  with:
    path: ~/.skaffold/
    key: fixed

[NixBiks]

Thanks for the suggestions !

Question before trying it out: I only have to add that additional step suggested by @StarpTech in GitHub actions? I'll leave everything else as stated on my original post? I.e. the skaffold.yaml as well?

[StarpTech]

@mr-bjerre yes. The action must be inserted before calling skaffold.

[NixBiks]

It works !! I'm kinda mind blown by that magic - thanks a lot!

[StarpTech]

keep an eye to #4889

[NixBiks]

I might have closed this a little too early. Now my images are no longer being cached. I'm getting the following message in the post cache step in my Github Action.

Post job cleanup.
Cache hit occurred on the primary key fixed, not saving cache.

My actions file look like this

name: Deploy to staging
on:
  workflow_dispatch:
  push:
    branches:
      - develop
    paths:
      - "k8s-manifest/**/*"
      - "lib/**"
      - "services/**"
      - "skaffold.yaml"

env:
  PROJECT_ID: ${{ secrets.STAGING_PROJECT }}
  GKE_CLUSTER: parallax-cluster
  GKE_ZONE: europe-west3-a
  SKAFFOLD_DEFAULT_REPO: gcr.io/${{ secrets.STAGING_PROJECT }}/parallax
  SKAFFOLD_PROFILE: staging

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository on develop branch
        uses: actions/checkout@v2
        with:
          ref: develop
      - name: Install gcloud
        uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
        with:
          version: "309.0.0"
          service_account_key: ${{ secrets.STAGING_SA_BUILD_AND_DEPLOY }}
          project_id: ${{ secrets.STAGING_PROJECT }}
      - name: Install kustomize, kubectl and skaffold
        uses: daisaru11/setup-cd-tools@v1
        with:
          kubectl: "1.19.2"
          kustomize: "3.8.4"
          skaffold: "1.15.0"
      - name: Cache skaffold image builds & config
        uses: actions/cache@v2
        with:
          path: ~/.skaffold/
          key: fixed
      - name: Configure docker
        run: |
          gcloud --quiet auth configure-docker
      - name: Connect to cluster
        run: |
          gcloud container clusters get-credentials "$GKE_CLUSTER" --zone "$GKE_ZONE"
      - name: Build and deploy to cluster
        run: |
          skaffold run \
          -l skaffold.dev/run-id=LEAVE_UNCHANGED

In skaffold.yaml I do not set tryImportMissing and the tagPolicy is gitCommit.

[briandealwis]

I haven't tried this action, but from the docs it says:

Once you create a cache, you cannot change the contents of an existing cache but you can create a new cache with a new key.

I suspect you need to make your cache key have a unique suffix, like the git commit SHA, and use restore-keys? Untested:

key: fixed-${{ github.sha }}
restore-keys: |
          fixed-${{ github.sha }}
          fixed-

[StarpTech]

fixed was a bad example. If you hit the cache with the same key the cache isn't updated. In order to create a good cache key, we need something like a lock file but we haven't because the artifact is generated after each build.

[StarpTech]

@briandealwis exactly. The cache behind fixed- will be updated with the most recent and fixed-${{ github.sha }} ensure that cache is updated after every run.

[NixBiks]

The suggestion from @briandealwis seems to be working perfectly. Thanks.

[aantn]

I think I'm missing something here, as my ~/.skaffold/cache is only 36kb even after building large images:

$ du -h ~/.skaffold/cache
 36K	/Users/user/.skaffold/cache

Do I need tryImportMissing or some other flag in addition to setting up the GitHub caching as mentioned above?

[briandealwis]

@aantn The ~/.skaffold/cache is a list of <artifact-hash> → <image-hash> pairs. Skaffold hashes the artifact source and looks to see if the hash exists, and if so skips the build and uses the corresponding <image-hash> instead. If that <image-hash> is not found in the remote registry, Skaffold will rebuild the image.

[aantn]

@briandealwis trying to understand. A few questions:

1. What is <artifact_hash> a hash on?
2. If I understand correctly, then this doesn't help cache intermediary layers in the docker file which are re-used by two subsequent builds on different commits. Rather, it helps skip the entire build if and only if <artifact_hash> is identical to a previous build. Is this correct?

What I am trying to accomplish is caching layers in the docker file which didn't change between builds where subsequent layers do change. If I understand correctly, I need to cache Docker internals for that and not ~/.skaffold/cache.

[briandealwis]

The <artifact-hash> is a hash of the files in the artifact. The approach described in this issue is to avoid rebuilding images where the artifact files are unchanged.

For docker layer caching I think you'll need to use the cacheFrom directives.

[afirth]

[removed] wrong issue sorry

[aaron-prindle]

Commenting for triage party, issue is closed with no one currently re-experiencing/commenting on it

[afirth]

Can confirm #4842 (comment) works well. And a pretty stinking smart idea with a write only key

[rajivsavroop]

Hello Folks, need some guidance here.

Trying to learn and test out skaffold cache, and I can see the cache gets restored. But Skaffold still builds a new image

Checking cache...
 - nginx-skaffold: Not found. Building

Workflow - https://github.com/rajivsavroop/GH_skaffold_example/blob/test-skaffold-cache/.github/workflows/example.yaml
@NixBiks @afirth , Hi. Any working example with Skaffold Cache?

I really appreciate any help you can provide.