[BUG] Idle CRC install causes high CPU usage #707
Comments
|
Please file this issue against OpenShift itself. Related to #607 (comment) These are part of on-going discussions with the OpenShift team, but external pressure/information/assessment is always helpful when having these. |
gbraad
added
kind/question
Minishift is running OpenShift 3.x. OpenShift 4.x operates under a different set of rules and design concepts; the cluster is self-maintaing in the sense that operators control the state of the cluster. |
|
If the state of the cluster is not changing like in a freshly installed idle cluster, then there should be little CPU usage by the operators. Something seems to be spinning needlessly. |
Operators are continously inspecting the state of the cluster. This, in the case of CRC, might not be needed as often, but at the moment there is no 'profile' to determine or adjust this. Please file an issue against OpenShift raising these concerns. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
stale
bot
added
the
status/stale
@cfergeau ^^ hopefully the type/profile designator of a cluster could help with this. This could be part of an enhancement. |
stale
bot
removed
the
status/stale
praveenkumar
added
the
status/pinned
|
@praveenkumar thanks for reopening. I just noticed again that crc seems to be consuming 20-25% of my 8 cores even when it's not doing anything productive (for me). The fan is constantly spinning 🤷 |
|
I would be happy to assist debugging this, but I haven't found a way (like minishift ssh && ps) |
|
Thanks @tsmaeder. You ned to login to the VM using the ~/.crc/machines/crc/crc_id_rsa |
|
Whoaaa...not so fast. When you say "login", you mean ssh to the crc VM? |
|
Also, there is no file named |
|
On Windows you can read ~ as meaning `C:\Users\[username]\`. After start,
the original login key has been replaced with a machine specific key that
has to life somewhere in this folder.
…On Wed, Apr 22, 2020 at 7:33 PM Thomas Mäder ***@***.***> wrote:
Also, there is no file named ~/.crc/machines/crc/crc_id_rsakey. Im on
Windows 10.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#707 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAOZUIVK43QOIFG5I3LEDRN3IXVANCNFSM4I72XCRQ>
.
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
|
|
Sorry, but even when I've started crc, there simply is no such file in ~.crc\machines\crc. I've tried
and that does not work because:
|
|
Permissions for '\\Users\\thomas\\.crc\\machines\\crc\\id_rsa' are too
open.
Please, use a different SSH client or modify the permissions for the file:
…On Thu, Apr 23, 2020 at 2:58 PM Thomas Mäder ***@***.***> wrote:
Sorry, but even when I've started crc, there simply is no such file in
~.crc\machines\crc. I've tried
ssh -i \Users\thomas\.crc\machines\crc\id_rsa ***@***.***
and that does not work because:
Permissions for '\\Users\\thomas\\.crc\\machines\\crc\\id_rsa' are too
open.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#707 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAOZUWYPNHHLOBRO6YJULRN7RK5ANCNFSM4I72XCRQ>
.
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
|
|
So...output from |
|
And running |
|
@gbraad anything else I can provide to move this one forward? |
|
We are in the process to collect deltas on our end |
|
Just reporting that I have a similar problem on latest MacOS version of CRC, where hyperkit is constantly using CPU on a brand new instance, that has been idle for at least an hour:
Logging into the VM: Top processes: |
|
Looks like it might be a hyperkit issue - see this very long thread on the subject: docker/for-mac#3499 |
guillaumerose
added
kind/bug
|
I am having the same issue on a MBP 16" i9 (8 cores). My Hyperkit CPU gets around 300% if doing something and 160% when idle. It doesn't seem to me to be only on Hyperkit side, in the case of my Minikube uses a lot less CPU using Hyperkit. CRC is completely useless for me because my machine gets too slow. |
|
We have the same issue on all devices we use with CRC and Ubuntu 20.04. Idle CRC consumes permanently almost all CPU capacity. Increasing # of assigned cpu cores leads to the additional cores being almost fully utilized as well. Is there any way to alter this behaviour? |
|
On my RHEL test machine, I see one CPU core being used 100%, and the others ~20%. I suspect most of this is coming from kubernetes/OpenShift design though, so I'm not sure how much this can be improved :-/ |
|
Here is the same. Installed crc on a RHEL VM and the fan simply does not stop. No matter what what OS i use, same result. Right now on a iddle system the load inside the CRC VM is: load average: 29.97, 25.60, 25.45. Even if the operators are doing something, is it a case for this kind of load?! The big CPU suckers seem to be kube-apiserver, etcd and kubelet. Any idea on how to troubleshoot this? EDIT: crc version 2.0.1 from Red Hat's site. |
|
macOs 12.4 - the same situation. CRC consumes 200% cpu being idle and only is capable of serving basic hello world. Anything non trivial throws it over the edge and laptop as well as CRC becomes unusable I am wondering how we can prioritize this issue higher, seems like it basically hinders many folks from using CRC at all Is there any alternative to CRC to try out for local development on OpenShift? |
|
What are your system's specs, in prarticular the amount of RAM? |
|
2019 MBP Core i7 6 core, 32 GB ram crc VM has 24 GB memory and 4-6 cores. It doesn't really matter how much cores and memory you give it, it is barely usable with any amount and becomes completely unusable and unstable if you try to do anything non-trivial. For example, web console routinely crashes, internal registry gives EOF in the middle of container downloads, etc. Vanilla configuration, all is out of the box. I can provide any debug info but I think the point here also made by other folks - there's smth seriously wrong with crc here which doesn't even seem to be related to hyperkit or macos (since folks reporting the same issue on Linux based systems) and you don't need a sophisticated scenario - just install it and check the CPU. Then throw a container and see your laptop melting |
|
You don't even need to start a container. It's nuclear all the time. Everything default. Openshift, the same s**t. Someone said it is like this by design. My mind is blown. |
|
I think it is clearly an issue. It can't be unusable by design. This is nuts that latest macbook almost dies immediately after you start empty cluster and surely dies once you run hello world in it. Fans never stop, their console GUI crashes and restarts every couple of minutes. I couldn't get a container with more or less non-trivial image to start since it never was able to finish downloading it from internal registry which gave up midway and threw EOF all the time. Redhat, please take it seriously. It says this is a product for local development. It is surely not in this current form. We are a large company (DataRobot) and we looked at adopting CRC for our internal dev. So far I am struggling to recommend that we move forward |
|
As a data point, I just downloaded crc 2.7.1 on my macbook x86_64/32GB running macOS Big Sur, I kept the default parameters for crc, ran Detailed steps regarding what you are doing would be useful, so that we can try to reproduce and understand what's happening for you. |
|
@as-polyakov Hi Anton - I am following up from our call yesterday. I chatted with the CRC team. Would you be able to provide the steps that you followed? Did you have a lot of the other applications running on your mac at the same time? We have not been able to reproduce this locally on our macs so we would like to understand the steps that you followed. When you reproduce this on your mac, could you check and share the following items with us:
|
|
To add; If needed, we can join a call. |
|
Note; the latest versions of CRC make use of the new Virtualization Framework using our vfkit driver. Issues regarding load are therefore in recent versions not caused by Hyperkit. |
|
@gbraad we are preparing the repro example. High level the situation was this - idle install itself with CRC doing nothing already takes a significant CPU (around 150-200% on my macbook), in agreement with other peoples observations. With Maybe this high level info helps while the team prepares repro example |
|
Seems it's a generic kubernetes problem. Whatever instalation you do, cpu will skyrocket. Be it crc, openshift or k8s. I installed openshift in a powerful server. Load was stupidly high. Someone said this was by design. In a time everyone is talking about saving energy, this is lame and stupid. |
I tried this scenario with 2 images, a 15GiB image filled with 0s, and an image of the same size filled with random data. The image filled with 0s could be uploaded to a crc cluster with its default settings with no huge impact on CPU. Looking at top inside the VM, CPU usage is coming from Apart from the |
|
Same here login inside the VM, those are top 3 ordered by CPU usage. |
General information
crc setupbefore starting it (Yes/No)?: yesCRC version
CRC status
chirino-mbp:~ chirino$ crc status CRC VM: Running OpenShift: Running (v4.2.0-0.nightly-2019-09-26-192831) Disk Usage: 13.15GB of 32.2GB (Inside the CRC VM) Cache Usage: 11.71GB Cache Directory: /Users/chirino/.crc/cacheCRC config
Host Operating System
Steps to reproduce
Expected
Actual
Logs
You can start crc with
crc start --log-level debugto collect logs.Please consider posting this on http://gist.github.com/ and post the link in the issue.
The text was updated successfully, but these errors were encountered: