Some records seem to never become reclaimable

[koeberlue]

I am running v0.12.4 of buildkit as a daemon in my kubernetes cluster. I am observing some issues with the garbage collection which I am not sure if it is a configuration problem or if I should raise a "real" issue.

After running for some time, buildkitd will stop working properly, because the disk space ran out.

I have configured the garbage collection as follows:

root = "/var/lib/buildkit"
[worker.oci]
  max-parallelism = 2
  gc = true
  enabled = true
  rootless = true
[[worker.oci.gcpolicy]]
  all = true
  # Note: I have 130GB available exclusively for buildkit
  keepBytes = "85GB"
  keepDuration = 300

The garbage collection seems to work just fine in the beginning. But after some time when I check the disk usage (buildctl du -v), I can see that lots of the entries will report Reclaimable: false. This state will also persist when buildkit has nothing to do for several hours.

So at some point, buildctl du will report something like this, while no build job is running:

Reclaimable:	98.54GB
Total:		98.54GB

Obviously the garbage collection can not clean up anything if there is no reclaimable record. But I was wondering, why they would be stuck in an "unreclaimable" state.

Running ps ax I can verify no build is currently running:

    1 user      0:00 rootlesskit buildkitd --debug --addr unix:///run/user/1000/buildkit/buildkitd.sock --addr tcp://0.0.0.0:1234 --oci-worker-no-process-sandbox
   11 user      0:00 /proc/self/exe buildkitd --debug --addr unix:///run/user/1000/buildkit/buildkitd.sock --addr tcp://0.0.0.0:1234 --oci-worker-no-process-sandbox
   27 user      8:30 buildkitd --debug --addr unix:///run/user/1000/buildkit/buildkitd.sock --addr tcp://0.0.0.0:1234 --oci-worker-no-process-sandbox
44159 user      0:00 sh
50777 user      0:00 ps ax

Does anyone have a hint what could trigger this behavior or what I could do to debug this?

---

Related issue: #4218

buildctl debug info:

BuildKit: github.com/moby/buildkit v0.12.4 833949d0f7908608b00ab6b93b8f92bdb147fcca

[koeberlue]

After some debugging we found the root cause of the issue. Due to a bug with Gitlab's docker registry we disabled sbom and provenance for the attestation:

target "nginx" {
  context    = "."
  dockerfile = "./nginx.dockerfile"
  tags       = ["<redacted>"]
  attest = [ "type=sbom,enabled=false", "type=provenance,enabled=false" ]
}

Note the attest= line. This would cause buildkit to never mark the built layers as Recoverable.

Removing this line resolves the issue for us. I am just wondering if this is a bug in buildkit or if it expected behavior.