slow get_cpu_info w/o pyinstaller

[FabioLuporini]

• OS: various mac os's and ubuntu's
• archs: various intel xeons
• P3.6, P3.7
• py-cpuinfo v5.0.0

Hi,

It takes more than a second for get_cpu_info to return w/o pyinstaller. See also here

I haven't profiled the code, but if I bypass that else calling _get_cpu_info_internal directly, it runs much quicker (in at most tens of ms).

I wonder whether we can improve this, at least on Linux/Darwin, where I seem to understand we could skip that else ? Would you accept a tiny patch that reads the OS name such that the if is entered if on Linux/Darwin ?

[workhorsy]

This sounds like a good thing to try. Other platforms shouldn't have to be penalized because of design flaws in Windows (not having proper forking). I'm open to you making a PR for it.

Thanks

[FabioLuporini]

Turns out I was wrong and this:

I haven't profiled the code, but if I bypass that else calling _get_cpu_info_internal directly, it runs much quicker (in at most tens of ms).

is not the culprit

this function seems the one to be causing the slow down. What confused me originally is the fact that this slow down occurs non-deterministically. BTW, I reproduced it on multiple machines and OSs, including MacOS and Ubuntu.

Now, I've been using 5.0.0, but it seems that master is a bit ahead, so I'm not sure if the issue is still there with latest py-cpuinfo, but I wouldn't see why not after glancing at the code

I don't know what's the best course of action at this point. I'll try to dig a bit more if I find the time later in these days.

[workhorsy]

That is okay. Thanks anyway. Maybe you or someone will figure it out later.

BTW. The _get_cpu_info_from_cpuid function spawns a new process and runs the _get_cpu_info_from_cpuid_actual function inside it, then returns the result. This is done because running bytecode on the CPU could crash the whole Python process. The overhead may be from spawning a new process or from inter process communication. I'm not sure though. I don't have the time to do a proper test.

[FabioLuporini]

cpuinfo_v2.pdf

maybe this could help?

[FabioLuporini]

anyway, let me ask a question: I pretty much only need flags and brand. Do you think I could bypass some of the calls performed by get_cpu_info by directly using one of the private routines ?

[workhorsy]

Wow. What did you use to make that PDF?

[workhorsy]

You can call the internal functions directly. But it isn't recommended, because the internal functions are not API stable and may change in the future.

from cpuinfo import *

info = cpuinfo._get_cpu_info_from_registry()
print(info)

Other ones are

# Try the Windows wmic
_get_cpu_info_from_wmic()

# Try the Windows registry
_get_cpu_info_from_registry()

# Try /proc/cpuinfo
_get_cpu_info_from_proc_cpuinfo()

# Try cpufreq-info
_get_cpu_info_from_cpufreq_info()

# Try LSCPU
_get_cpu_info_from_lscpu()

# Try sysctl
_get_cpu_info_from_sysctl()

# Try kstat
_get_cpu_info_from_kstat()

# Try dmesg
_get_cpu_info_from_dmesg()

# Try /var/run/dmesg.boot
_get_cpu_info_from_cat_var_run_dmesg_boot()

# Try lsprop ibm,pa-features
_get_cpu_info_from_ibm_pa_features()

# Try sysinfo
_get_cpu_info_from_sysinfo()

[FabioLuporini]

Wow. What did you use to make that PDF?

Hi! I used Python's CProfile for data collection and gprof2dot to create the pdf. Here's a summary.

You can call the internal functions directly. But it isn't recommended, because the internal functions are not API stable and may change in the future.

yeah, that's why I was a bit hesitant to do it...

BTW, let me add an extra piece to the puzzle: not only is the slow-down not deterministic, but also the output -- the returned dictionary containing the platform information -- may differ from run to run. I recently noticed that some entries may be missing. To reproduce, but you can try running get_cpu_info() a few times on linux and you should be able to see it