-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docker --cache-from with BUILDKIT_INLINE_CACHE does not work every second time #1981
Comments
I am expierencing quite a similar behaviour. In my case, I am using 3 multistage Dockerfiles. Strange thing is, the first stages are always cached, in one of the docker files the second stage is also always cached. In the other 2, this cached-non cached behaviour occured. Only difference i could spot is that in the dockerfiles with alternating cache usage, for both stages the same base docker image (From image:tag) is used. |
Seeing the same problem. Asked on stack overflow - https://stackoverflow.com/questions/66224121/docker-cache-from-not-all-layers-being-run-if-no-files-have-changed We have a very simple Dockerfile. To speed up our builds we're using the We're seeing some weird behaviour where if the files have not changed the lines after the COPY line are not being run.
Does not seem to get executed so when the application tries to start node_modules is missing.
We're deploying to Kubernetes from GitHub actions but I can recreate the problem locally. Initial build:
All works - node_modules and build folder are there: Clean up docker as if we starting from scratch like on the build system
Do another build:
Everything is still fine. Clean up docker as if we starting from scratch like on the build system
Do a third build:
Files are missing!
No After inspecting the image I can see that a layer is missing when compared to the working images. |
@cgreening Please provide runnable reproduction steps. |
@tonistiigi Here you go: https://github.com/cgreening/docker-cache-problem Let me know if I can do anything to help. |
I have a very similar problem in my CI using Here are two layers that have the exact same file contents. It seems that only the times are different. I wasn't aware that this should change a layer hash? d19e1c61b0cbe787e9d58d9ea54e2660ab6ae0c6d1fd3b11a410f60154dbe525.tar.gz.txt 2a3db49c74cd0666b6e4d2729cabafa22ea4270a5a0b6a41b92f87ec6f0f1301.tar.gz.txt |
@cgreening Thanks. I can confirm I can repro this in 20.10.2 . Buildkit master(buildx) and 19.03 seem to not have the issue. |
I see this issue on 19.03.11 in docker-in-docker, so I assume that this means it's in the daemon? |
I can reproduce this locally on
Just to confirm you mean buildx using the 19.03 daemon, correct? |
I mean buildx using buildkit release directly with the container/k8s driver. |
The issue reported by @cgreening #1981 (comment) should be fixed with #1993, once vendored to moby codebase. I'm not sure if the other comments in here have the same problem as this was the only reproducer given. |
@tonistiigi thanks. My issue is exactly the same as reported by @cgreening. I'm not familiar with buildkit release cycle and especially how it syncs up with docker engine releases. What would you say is the best way to access the fix sooner? |
It's included in the PRs linked in ^ , including the backport to 20.10 Going to close this. If you find that fix does not apply to your use case open a new ticket with reproduction steps. For the best way to avoid this issue at all is to use |
Actually, it still doesn't work. I have latest docker engine v20.10.7, but my reproducible example still doesn't work |
The build of the container sometimes fails with the following error: failed to compute cache key: failed to walk /var/lib/docker/overlay2/df6ea.../merged/root/aktualizr/build: lstat /var/lib/docker/overlay2/df6ea.../merged/root/aktualizr/build: no such file or directory It seems a cache issue with the sota-builder build stage. For some reason, using --cache-from with BUILDKIT_INLINE_CACHE does not work every time. Indeed, there is a report of this issue in [1] and it is fixed in [2], but it seems our CI it's not running a Buildkit version that has this fix. To workaround this issue, let's disable cached builds for now. [1] moby/buildkit#1981 [2] moby/buildkit#1993 Related-to: TOR-1671 Signed-off-by: Sergio Prado <[email protected]>
To expand on @tonistiigi suggestion of using buildx plugin directly, this is what worked for me in gitlab-ci:
|
Hi @Bi0max , I am experiencing this issue using docker 20.10 on gitlab. Is there a specific version I should be pinning to? |
Hi @alex-treebeard, last time I checked, i had v20.10.7 of docker engine (back then it still did not work). Haven't checked since then. |
My team has been affected by this issue as well, and the original poster's "pull the image each time before building" workaround (as mentioned right at the end of this issue description) seems to be working for us - And I know, that the docs clearly states that an explicit
But without it we experience the same "cache doesn't work every second time" bug as the original poster :/ For reference we're using gitlab CI, with their shared runners and |
Same issue here, using latest docker dind image |
I believe this issue should be reopened. |
I can still reproduce this in |
We're experiencing the same issue as well, please reopen |
Docker version 20.10.15, build fd82621 on Bitbucket Pipelines. Totally reproduces. |
Ok everybody, you probably want to move the discussion to #2274 Sorry I missed it. |
I solved this by building the image with Buildah. It works with the cache reliably and predictably. My code for use in GitLab CI looks like: before_script:
- buildah login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- buildah build
--tag $CI_REGISTRY_IMAGE/app:latest
--cache-from $CI_REGISTRY_IMAGE/app # No tag here
--layers
--cache-to $CI_REGISTRY_IMAGE/app # No tag here
.
- buildah push $CI_REGISTRY_IMAGE/app:latest The cache is stored in the GitLab registry separate from the app image. I spent many hours by researching and testing this. What's also recommended is to have the ".dockerignore" file with Using Kaniko is another good solution but Buildah is closer to default Docker commands (1:1 replacement) and can be easily added into default DinD image. |
Works well, thanks for suggesting. Adding |
I am trying to take advantage of the caching/pulling system of BUILDKIT for Docker for my CI/CD process. But it does not work as expected.
I created a dummy local example (but the same happens also in my CI system - AWS CodePipeline, and for both DockerHub and AWS ECR). You need to have Dockerfile, run_test.py (with any insides) and requirements.txt (with any insides) in a folder.
The Dockerfile:
run_test.py is actually not interesting, but here is the code just in case:
In advance, I export two environment variables:
Then, to test I have the following command. First two commands remove local cache to resemble the CI environment, then build and push.
BE CAREFUL, CODE BELOW REMOVES LOCAL BUILD CACHE:
As expected, the first run just builds everything from scratch:
Then, I slightly adjust
run_test.py
file and the result is again as expected. All the layers until the last step ([7/7] COPY) are downloaded from repository and reused.Now, I change
run_test.py
again and I would expect docker to do the same thing as last time. But I get the following result, where it build everything from scratch:But the strangest thing for me is, when I change
run_test.py
for the third time, it uses cached layers again. And it continues in the same way: fourth time - doesn't use, fifth time - uses, etc...Do I miss something here?
If I pull the image each time before building, then it always uses cache, but it also works in the same way without the BUILDKIT.
The text was updated successfully, but these errors were encountered: