Dockerd / buildkit in a infinite loop and burning cpu

[bpaquet]

Hello,

While trying to use buildkit through docker build, my dockerd daemon seems to go in a infinite loop and the docker build is hanging.

Version: Docker version 19.03.5, build 633a0ea838
Processes (ps axu | grep docker):

root      5822 96.6  0.9 2460340 290756 ?      Ssl  18:29  45:16 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
tc0      26084  0.2  0.2 1080756 72472 ?       Sl   18:51   0:03 docker build --build-arg ...

While looking into the trace, I found lot of occurence of github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*cacheManager).filterResults:173
But I'm not a specialist of this kind of trace, so may be I'm not reading it right.

The trace is here: out.gz (remove the .gz if needed).
Trace extracted with curl --unix-socket /var/run/docker.sock http://./debug/pprof/trace

The CPU usage does not go down when I stop the docker build process (still using 100%).

@tonistiigi I already had an issue around cache and LoadWithParents (#1250, fixed by you, thx), while building the same Dockerfile.

Let me know if you have any idea about the issue and what I can do to dig further.

[tonistiigi]

Do you have a reproducer you can share?

[bpaquet]

Thx a lot for answering :)
I can easily reproduce it on my env, it's systematic.I cannot share the Dockerfile. It's probably due to the fact I'm launching multiple stages in parallel (outside of buildkit, using make -j8, and multiple images in --cache-from). Not a standard use, but it should work.

I did some tests by recompiling docker.
I confirm the code is entering in filterResult function from here, and is never going out.

What can I do to provide more information ? I added logrus.Errorf("FilterResults: %v %v %v", id, m, ck) at the begining of filterResults. I see a loop. The function is called again and again with the same id. I can provide this log.

[tonistiigi]

I can't easily see what could cause the id to repeat there so it must be something with your data. I also don't see if this is from imported or local cache without additional info. Could it be something like a duplicate copy instruction that triggers this?

[bpaquet]

I can't easily see what could cause the id to repeat there so it must be something with your data. I also don't see if this is from imported or local cache without additional info. Could it be something like a duplicate copy instruction that triggers this?

I have that:

COPY --from=assets_builder /app/public/assets /app/public/assets
COPY --from=assets_builder /app/public/webpack /app/public/webpack

And yes, the problem seems to be just after.

[tonistiigi]

That's not exactly what I had in mind but could be related still. Can you try if you can put together a reproducer based on similar snippets to ease the debugging of this issue?

[bpaquet]

I did a lot of tests, I confirm the problem disappear if I remove the second copy.

I'm still trying to have a simple case failing, but did not succeed to write it :(

[tonistiigi]

I think this might be same as #1336 cc @aiordache

[tonistiigi]

@bpaquet Can you check if this is the same as #1336 and #1386 fixes this issue. If not then please post a stacktrace or a reproducer for your issue so it can be looked further.

[bpaquet]

I’m not sure to understand. Seems you have a fix in 0010996. It’s cool ! But is it released / build somewhere? Should I build it myself to test it?

…

On Tue 3 Mar 2020 at 03:55, Tõnis Tiigi ***@***.***> wrote: @bpaquet <https://github.com/bpaquet> Can you check if this is the same as #1336 <#1336> and #1386 <#1386> fixes this issue. If not then please post a stacktrace or a reproducer for your issue so it can be looked further. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1313?email_source=notifications&email_token=AACSJXOD5S4CYSVCNJFDPXLRFRWS3A5CNFSM4KC4W4R2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENR4IPA#issuecomment-593740860>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACSJXLRZPBF5HDPJPVZ4YLRFRWS3ANCNFSM4KC4W4RQ> .

[tonistiigi]

fixed with #1413, report if you still see anything like this