skaffold ko builder does not detect changes to go:embed files

[jsok]

Expected behavior

Making a change to a file embedded by the embed package should cause skaffold to produce a new image.

Actual behavior

skaffold build instead logs:

Checking cache...
 - app: Found remotely

and no new image is produced.

Information

• Skaffold version: 1.39.2
• Operating system: macOS & Linux
• Installed via: Homebrew
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta28
kind: Config
build:
  artifacts:
    - image: app
      ko:
        dir: .
        main: ./app/cmd
        env:
          - CGO_ENABLED=0
        flags:
          - -buildvcs=false
          - -trimpath
          - -v
        ldflags:
          - -buildid=
          - -extldflags="static"
          - -s
          - -w

Steps to reproduce the behavior

1. clone https://github.com/jsok/skaffold-ko-embed-bug
2. Run a local registry: go run github.com/google/go-containerregistry/cmd/registry@latest -port 1338
3. skaffold build -d localhost:1338

Generating tags...
 - app -> localhost:1338/app:latest
Some taggers failed. Rerun with -vdebug for errors.
Checking cache...
 - app: Not found. Building
Starting build...
Building [app]...
No matching credentials were found, falling back on anonymous
Using base gcr.io/distroless/static:nonroot@sha256:1f580b0a1922c3e54ae15b0758b5747b260bd99d39d40c2edb3e7f6e2452298b for github.com/jsok/skaffold-ko-embed-bug
Using build config app for github.com/jsok/skaffold-ko-embed-bug
Building github.com/jsok/skaffold-ko-embed-bug for linux/amd64
Publishing localhost:1338/app:latest
Published localhost:1338/app@sha256:b772b3a87a43fb1c2708ee9651ce345776ea80fab364972b15a412d87e36f646
Build [app] succeeded

4. Make an edit to hello.txt (the embedded file)
5. skaffold build -d localhost:1338

Generating tags...
 - app -> localhost:1338/app:latest
Some taggers failed. Rerun with -vdebug for errors.
Checking cache...
 - app: Found Remotely

Fixes considered

Adding dependencies to the skaffold.yaml like:

  - image: app
    ko:
      dependencies:
        paths:
        - '**/*.go'
        - hello.txt

Does fix the problem, but this is tedious to maintain for larger projects.

ko natively does not exhibit this behaviour, it picks up embedded file changes and produces new images correctly.

[jsok]

Digging into this a bit, it seems that by default skaffold only consider Go files:

skaffold/pkg/skaffold/build/ko/dependencies.go

Line 37 in 6396524

Paths: []string{"**/*.go"},

The information about embedded files is readily available via go list or via golang.org/x/tools which skaffold already depends on, see https://pkg.go.dev/golang.org/x/[email protected]/go/packages#Package

> go list -json . | jq ."EmbedPatterns"
[
  "hello.txt"
]

So a potential solution here would be to introspect the packages and automatically add embed patterns to the dependencies list?

[halvards]

Thanks for raising this, and for the repro repo.

The dependencies defaults were chosen to be conservative and not fail builds due to patterns that don't match any files. However, if embedded files are not present, then go build fails, so that's an argument for including them by default.

We'll look into this, and also whether it should apply to other builders - e.g., buildpacks when building Go apps.

[halvards]

Also, @jsok, since you're in Sydney - feel free to drop by here and say hi 😄 https://www.meetup.com/devops-sydney/events/287892691/

[timuthy]

@halvards, I just found this Github issue because of a similar problem I faced with static assets. I'd appreciate if you could discuss considering them as well.

[halvards]

Thanks for the comment @timuthy.

To watch static assets you can specify kodata/**/* as a watch pattern. The ko builder also supports syncing static assets (without image rebuilds) in dev mode: https://skaffold.dev/docs/pipeline-stages/builders/ko/#file-sync