Cache mount should have an option to keep data in layer

[ilyadh]

Opting in to use cache mount to speed up go mod download requires all next go build steps to mount it as well.

FROM golang:latest AS base

RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=source=go.mod,target=go.mod \
    --mount=source=go.sum,target=go.sum \
    go mod download
    
FROM base AS build-server

RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=target=.
    go build /server ./cmd/server
    
FROM base AS build-client

RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=target=.
    go build /client ./cmd/client

This also makes it impossible to use buildx --cache-from to reuse a layer with downloaded modules in a CI environment.

I've been able to workaround this issue by manually copying data from cache into layer.

WORKDIR /app
RUN --mount=type=cache,target=/app/go/pkg/mod \
  GOMODCACHE=/app/go/pkg/mod go mod download && \
  mkdir -p /go/pkg/mod &&\
  cp -a /app/go/pkg/mod/. /go/pkg/mod

In practice, it works really well, and makes it possible to both speed up builds in dev and reuse downloaded modules from previous builds in CI.

I think it's confusing that there's no interoperability between cache mounts and layer cache, you can only have one or the other.

An option to copy data from cache into layer can be a bridge between those features.

[profnandaa]

Thanks for raising the issue. This will require some more investigation:

I think it's confusing that there's no interoperability between cache mounts and layer cache, you can only have one or the other.

[jedevc]

@ilyadh this looks like a duplicate of #1512.

An option to copy data from cache into layer can be a bridge between those features.

I don't think this is a viable option - it's generally not a good idea to copy the entire cache into the layer history for the image, since then this will massively inflate the size of the image.

I think it's confusing that there's no interoperability between cache mounts and layer cache, you can only have one or the other.

If I understand correctly, what you're saying is that there's no interoperability because you have to choose an approach of persistent storage so you can use cache mounts or ephemeral storage so you can use layer caches? Both of them do work together, but I see the dilemma. You may be interested in https://github.com/reproducible-containers/buildkit-cache-dance, which is a solution to get the cache mounts and store that in github actions so it can be fetched later.

I'm going to close this since it looks like a duplicate of the above linked issue, but if for some reason it's separate, we can keep discussing and re-open 🎉

[ilyadh]

@jedevc Thank you for your response!

I don't think this is a viable option - it's generally not a good idea to copy the entire cache into the layer history for the image, since then this will massively inflate the size of the image.

Yes, I agree. But as demonstrated with my example, GOMODCACHE is special in a way that it's both the result of running a go mod download command (what I want to keep in a layer) and a cache for the command itself (what I want a cache mount for). It's not a problem for something like apt-get where the result of a command is in a different place than what it needs to run quicker, and where it's possible to use both cache mounts and layer caches.

Also, the size of the image is not a big concern here, since it's all part of a "build" stage in a multi-stage environment. This cache won't be present in the final image anyway.

So all in all, I don't think this is a duplicate issue.
I might have worded it poorly, but the real issue is that you can't use cache mounts and layer caches together when it's the same location that you want to use them for.

GOMODCACHE example of this problem might be the one and only. So I understand, if you want to keep this issue closed anyway.