How to debug stuck VM / hang with no shell access?

[ian-h-chamberlain]

Hello, recently I have been facing an issue while using Colima that I'm kind of at a loss for how to debug. The issue results in a state where I can no longer access any containers and not even the colima VM, and my only recourse is to colima stop --force and start again.

Basically, something in my organization's build process (just a basic Makefile) causes the container and even the VM to lock up whenever I build certain targets with make. I cannot use colima ssh, nor any docker commands — they all seem to just hang forever when I try.

Unfortunately, I can't share the code as it's private to our organization, but I'm hoping someone here might be able to help me get started on debugging this. I've tried adding different kinds of tracing to the build to see if I can figure out what specific command causes it, but so far I've had little luck.

Is there any kind of "backdoor" way into the VM or its filesystem that I might be able to use to help with this? At this point I'm just kind of stuck because I can't build anything using colima and I'm afraid I might have to switch to another solution for this use case, but I have had a great experience using it for other purposes so I'm hoping I can figure out how to work around or fix this. Thanks in advance!

[SuperFola]

Could it be because you are building huge executables and it takes all the CPU time allocated + RAM allocated to the VM, making it unresponsive?

Have you tried giving more RAM and CPU to the VM?

[ian-h-chamberlain]

you could have a look at the system logs and look for docker related events/commands/etc?

Do you mean my host system? colima status, ssh and any docker commands all seem to become unresponsive in this scenario so I guess I'm not sure where else to look.

Regarding high CPU — yes, it is a lot, but I think it's to be expected to some extent while running a build. Normally while idle the container runs from 1—25% but of course compiling code takes significantly more resources. I have seen the percentage creep up well into the 300%+ range without "crashing" like this, so I don't think it's hitting some specific threshold, and in fact sometimes the whole build actually succeeds.

Let me see if there are any other processes running in the container that might be complicating this, and try to pin it down a bit further.

[ian-h-chamberlain]

One more data point — I eliminated some extraneous stuff in the container and reproduced again, this time with very little resource usage!

CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O      BLOCK I/O        PIDS
4779296d1270   flamboyant_haibt   33.33%    21.39MiB / 7.776GiB   0.27%     1.3kB / 0B   1.41MB / 102kB   8
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O      BLOCK I/O        PIDS
4779296d1270   flamboyant_haibt   33.33%    21.39MiB / 7.776GiB   0.27%     1.3kB / 0B   1.41MB / 102kB   8
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O   PIDS
4779296d1270   flamboyant_haibt   --        -- / --             --        --        --          --

[SuperFola]

Do you mean my host system?

I meant in the VM, this is a full blown linux system, there should be some logs

[abiosoft]

Are you emulating x86_64 on M1 or vice-versa?

[ian-h-chamberlain]

No, I'm on an x86_64 host with an x86_64 VM. Since any colima command seems to lock up when this happens, is there some other way (maybe some magic limactl commands) or places on the host I can look for logs if I reproduce again?