Builds with inline caching do not consistently use cache

[AaronFriel]

Context

When using the github.com/docker/docker/client API to trigger builds with builder version BuilderBuildkit (v2) and the build argument BUILDKIT_INLINE_CACHE=1, builds are cached only every other execution.

Another user using just the Docker CLI (docker build) reported this same behavior in the community Slack.

Expected behavior

Using the repository here: https://github.com/AaronFriel/docker-buildkit-cache-bug

Buildkit caching should consistently utilize remote caches.

Actual behavior

The example script shows this alternating caching behavior of the first stage in a dockerfile:

$ ./example.sh 
Iteration 1

real    0m58.579s
user    0m5.850s
sys     0m1.157s
Iteration 2

real    0m18.505s
user    0m5.636s
sys     0m0.647s
Iteration 3

real    0m57.091s
user    0m5.808s
sys     0m0.659s
Iteration 4

real    0m19.807s
user    0m5.667s
sys     0m1.006s

This alternation is consistent, it is every other build in every trial, for however many iterations, that I've run.

[tonistiigi]

This is likely something in the Moby worker. I tried a similar reproducer in v0.11(buildx container driver) and saw no issues. We should also have plenty of tests for this in buildkit(seems to repro even in quite simple builds).

There is no version info in the report. Would be interesting to know if this is 23 regression or not.

[AaronFriel]

I see this issue on Docker Desktop (v20.x.y) latest. It looks like the same issue as #2274, perhaps?

[damienrj]

If you are using the docker-container driver than it doesn't happen (as I understand it), but there are also downsides to using that compared to the normal drivers. I believe the same issue as #2274. We have been having this cache hit issues for a year plus at this point.

[king-11]

@tonistiigi is there some work being done towards it? If so would likely want to contribute in someway towards making this work

[tonistiigi]

@king-11 I don't have specific ideas atm what part might be the cause, so general debugging is needed to understand better where this goes wrong.

[king-11]

Can you point me somewhere where I should start looking into for the same

[brnpimentel]

the same here