Resource-leaked logs-collector-XXXX containers

[mieubrisse]

What's your CLI version?

0.85.29

Description & steps to reproduce

My local Docker-backed Kurtosis is empty:

kurtosis enclave ls
UUID   Name   Status   Creation Time

kurtosis clean -a doesn't clean anything, as expected.

However, there are two zombie logs-collector containers hanging around that don't go away:

These do not correspond to any enclave networks, as there are none (as expected):

$ docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
8662e507c238   bridge    bridge    local
deab0a0cb95e   host      host      local
a8c59496b516   none      null      local

More details in #1832

Desired behavior

Kurtosis should always fully clean up after itself.

The original clean logic was written like so:

1. Use the presence/absence of -a to decide which enclaves ought to remain (if -a is specified, no resources should remain)
2. After finding the enclaves-to-remain, find and kill all Kurtosis resources that aren't in the enclave-to-remain set in the proper order (containers -> volumes -> networks). We DON'T say "look for enclave networks, and then remove resources associated with them" because if the enclave network is removed without all the resources being removed for any reason (e.g. maybe a failure happened), then we permanently leak resources.

In other words, we first grab ALL Kurtosis resources, default them all to "destroy this resource", and then remove the ones that should be preserved. This guarantees that we process every single resource Kurtosis can create.

This algorithm is very important, because it guarantees that even if an enclave is in an inconsistent state (e.g. an error happened, or maybe the user messed with Docker), all the resources for the enclave can still get fully cleaned up.

I suspect that the logs-collector cleanup has not been written this way, and is instead written with, "first try to remove the enclave network and all the containers in it, then try to remove the logs-collector". This means that if a failure happens (e.g. because of Ctrl-C), then the enclave network is lost but the logs-collector hasn't been cleaned up.

What is the severity of this bug?

Critical; I am blocked and Kurtosis is unusable for me because of this bug.

What area of the product does this pertain to?

CLI: the Command Line Interface

[tedim52]

Confirming that the clean logic is as you described it should be (at least on the surface). I confirmed by ensuring that the log collector was one of the containers removed in the destroyContainersInEnclaves step that happens before destroying volumes then destroying the enclave networks.

Going to continue debugging what may be causing the leaked logs collectors.