Different output between psutil and "free" command

[giampaolo]

From [email protected] on December 02, 2011 14:43:43

What steps will reproduce the problem?  
1. import psutil
2. mem = psutil.phymem_usage()
3. print mem.total, mem.used, mem.free 

What is the expected output?  
This is from "free" command:
total      used        free
2569812    2458704     111108 

What do you see instead?  
This is from psutil:
total  used   free
642453 615393 27060 

What version of psutil are you using? What Python version?  
psutil 0.4.0 and Python 2.7.1 

On what operating system? Is it 32bit or 64bit version?  
Linux Mint Katya 2.6.38-8-generic, 32bit version 

Please provide any additional information below.  
I want to check my physical memory usage using psutil. What I got from psutil 
is different from "free" command. Is my simple script using psutil wrong? Thank 
you in advance.

Original issue: http://code.google.com/p/psutil/issues/detail?id=232

[giampaolo]

From g.rodola on December 02, 2011 06:20:08

psutil results are expressed in bytes while "free" express them in... KB maybe? 
I'm not sure.
To compare results you should use MB instead ("free -m").
Example:

import psutil, os
mem = psutil.phymem_usage()
print "psutil: total=%s used=%s free=%s\n" % (mem.total / 1048576,
                                              mem.used / 1048576,
                                              mem.free / 1048576)
os.system('free -m')


On my linux box I get this output:

psutil: total=3961 used=2757 free=1203

             total       used       free     shared    buffers     cached
Mem:          3961       2757       1203          0        630        645
-/+ buffers/cache:       1481       2479
Swap:         1999         67       1932



Therefore, I'd say they match.
I'm noticing the doc doesn't mention that psutil.phymem_usage() is supposed to 
match free utility.
I'm going to fix that.

[giampaolo]

From [email protected] on December 02, 2011 07:39:28

Hi, thanks for your quick reply.

You said in Documentation that, by default, psutil.phymem_usage() return value 
is in bytes. free -b will return total, used, and free memory in bytes too. 
However, I still  don't get the match result.

--- 1st trial --- 

mem = psutil.phymem_usage()
print 'total:', mem.total, 'used:', mem.used, 'free:', mem.free
os.system('free -b')

This is output on my Linux Mint Katya:

total: 642453 used: 476598 free: 165855

             total       used       free     shared    buffers     cached
Mem:    2631487488 1952526336  678961152          0   55832576 1497534464
-/+ buffers/cache:  399159296 2232328192
Swap:   2012213248          0 2012213248


It will give an almost match result when I multiply each value from psutil by 4096:

--- 2nd trial ---

print 'total:', mem.total*4096, 'used:', mem.used*4096, 'free:', mem.free*4096
os.system('free -b')

Output:

 total: 2631487488 used: 1948819456 free: 682668032

             total       used       free     shared    buffers     cached
Mem:    2631487488 1948946432  682541056          0   56283136 1492238336
-/+ buffers/cache:  400424960 2231062528
Swap:   2012213248          0 2012213248

I continue my trial to "free -k" that returns value in kilo Bytes. It will 
almost match if I multiply value from psutil by 4.

--- 3rd trial ---

print 'total:', mem.total*4, 'used:', mem.used*4, 'free:', mem.free*4
os.system('free -k')

total: 2569812 used: 1903144 free: 666668

             total       used       free     shared    buffers     cached
Mem:       2569812    1903516     666296          0      55840    1451104
-/+ buffers/cache:     396572    2173240
Swap:      1965052          0    1965052

Then, i try "free -m" that returns value in mega Bytes. It will give an almost 
match result if I divide value from psutil by 256

--- 4th trial ---

print 'total:', mem.total/256, 'used:', mem.used/256, 'free:', mem.free/256
os.system('free -m')

total: 2509 used: 1887 free: 621

             total       used       free     shared    buffers     cached
Mem:          2509       1887        621          0         55       1435
-/+ buffers/cache:        397       2112
Swap:         1918          0       1918


Any idea about value 4096, 4, and 256? I have stuck here for the last 5 hours. 
Thank you :)

[giampaolo]

From [email protected] on December 02, 2011 07:46:30

Can you post the contents of /proc/meminfo from your system? This seems really 
odd, I'd like to go direct to the source and see what it looks like before we 
assume its a psutil issue.

[giampaolo]

From [email protected] on December 02, 2011 07:50:56

Okay, here is the contents of my /proc/meminfo:

MemTotal:        2569812 kB
MemFree:          970800 kB
Buffers:           58436 kB
Cached:          1083416 kB
SwapCached:            0 kB
Active:           715356 kB
Inactive:         806328 kB
Active(anon):     373392 kB
Inactive(anon):   156612 kB
Active(file):     341964 kB
Inactive(file):   649716 kB
Unevictable:          16 kB
Mlocked:              16 kB
HighTotal:       1703752 kB
HighFree:         483768 kB
LowTotal:         866060 kB
LowFree:          487032 kB
SwapTotal:       1965052 kB
SwapFree:        1965052 kB
Dirty:                 0 kB
Writeback:             8 kB
AnonPages:        379600 kB
Mapped:           110212 kB
Shmem:            150172 kB
Slab:              37620 kB
SReclaimable:      24928 kB
SUnreclaim:        12692 kB
KernelStack:        2792 kB
PageTables:         8156 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3249956 kB
Committed_AS:    2416732 kB
VmallocTotal:     122880 kB
VmallocUsed:       28612 kB
VmallocChunk:      53448 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       4096 kB
DirectMap4k:       24568 kB
DirectMap4M:      884736 kB

Thank you ^_^

[giampaolo]

From g.rodola on December 02, 2011 08:05:07

What's the exact problem we're talking about? The lack of a perfect match when 
using bytes / kilo bytes?

[giampaolo]

From [email protected] on December 02, 2011 08:35:12

Yes, we're talking about matching psutil.phymem_usage() and free output. Based 
on described Documentation, psutil should match with free output, isn't it? I 
want to use psutil in my server so I double check it with existing tool :)

Thanks.

[giampaolo]

From [email protected] on December 02, 2011 08:41:25

My mistake, I didn't realize we had moved to C code on the Linux platform for 
physical memory information. In that case it's most likely a casting issue. 
I'll have to look up the sysinfo docs to confirm but I think that's the problem.

[giampaolo]

From [email protected] on December 02, 2011 08:44:56

http://stackoverflow.com/questions/4229415/c-sysinfo-returning-bad-values-i686 
http://linux.about.com/library/cmd/blcmdl2_sysinfo.htm The values returned by 
sysinfo are multiples of the "mem_unit" value. Our code is currently not 
calculating the values correctly since we're returning totalram, freeram, 
bufferram directly from sysinfo.

[giampaolo]

From [email protected] on December 02, 2011 08:45:16

Status: Started
Labels: OpSys-Linux

[giampaolo]

From [email protected] on December 02, 2011 08:45:58

Okay, it seems that I have same problem with psutil.virtmem_usage(). Should I 
start a new issue or continue this issue to report psutil.virtmem_usage() problem?

Thanks.

[giampaolo]

From g.rodola on December 02, 2011 08:56:17

psutil and free results already match as shown in my first message using mega 
bytes as unit of measurement.
You're not getting a perfect match when using bytes / kilo bytes because:

- the unit of measurement is more precise
- more importantly: when you call "free" the system allocates/uses an extra 
quantity of memory *for the free command itself*

Therefore there's no way to have "free" and "used" values matching perfectly - 
the only value which keeps being exactly the same is the total memory, which 
isn't subject to any change deriving from the creation of new processes (in 
this case: the python interpreter and the free command).

[giampaolo]

From [email protected] on December 02, 2011 08:56:49

virtmem is a different function and it just reads from the /proc/meminfo file 
so it should match these two lines: 

SwapTotal:       1965052 kB
SwapFree:        1965052 kB

if the virtmem totals don't match /proc/meminfo SwapTotal and SwapFree in 
bytes, then I would open a new issue to track that separately.

[giampaolo]

From [email protected] on December 02, 2011 09:04:51

Ok, I've committed changes in r1231 for get_physmem() in the C code to convert 
the values to unsigned long long byte values for memory. Note that I changed 
the Py_BuildValue to use KKK instead of lll for the 3 values, as it's possible 
on a 64 bit system that memory values in bytes will be > long int size. Tested 
on my machine but ideally we can get it tested on 64 and 32 bit systems with 
various platforms.

Status: FixedInSVN

[giampaolo]

From [email protected] on December 02, 2011 09:10:22

> psutil and free results already match as shown in my first message using 
> mega bytes as unit of measurement.

That may be true on your system, but it won't be the case on all systems due to 
the way sysinf() is implemented currently. We needed to make a code change in 
psutil to make sure the values get calculated correctly for all Linux 
platforms. 

> Therefore there's no way to have "free" and "used" values matching perfectly
> the only value which keeps being exactly the same is the total memory

This is correct, as Giampaolo I wouldn't expect psutil or free to match exactly 
for the used/free values, but they should at least be in the same ballpark.

[giampaolo]

From g.rodola on December 02, 2011 09:15:46

> That may be true on your system, but it won't be the case on all systems 
> due to the way sysinf() is implemented currently

Clear. Good catch.
Can you also update HISTORY?

[giampaolo]

From [email protected] on December 02, 2011 09:17:05

HISTORY file updated

[giampaolo]

From [email protected] on December 02, 2011 10:02:17

> if the virtmem totals don't match /proc/meminfo SwapTotal and SwapFree in bytes,
> then I would open a new issue to track that separately. 

I have to double check psutil.virtmem_usage() first to describe a detail problem.

> This is correct, as Giampaolo I wouldn't expect psutil or free to match exactly
> for the used/free values, but they should at least be in the same ballpark.

I can understand about perfect match. It's reasonable to understand. 

> That may be true on your system, but it won't be the case on all systems due to
> the way sysinf() is implemented currently. We needed to make a code change in
> psutil to make sure the values get calculated correctly for all Linux 
platforms. 

Then, should I commit to SVN to try this little correction to sysinf?

Thanks.

[giampaolo]

From [email protected] on December 02, 2011 10:11:47

Yes, if you want to try the latest SVN revision it should hopefully correct the 
problem for you :)

[giampaolo]

From [email protected] on December 02, 2011 10:19:41

Okey, thank you for you both ^_^

[giampaolo]

From [email protected] on December 07, 2011 05:29:20

Just FYI, I have tried latest SVN revision and it fixed my problem with 
psutil.phymem_usage() output.

Thank you :)

[giampaolo]

From g.rodola on December 07, 2011 05:31:03

Good. This is a high priority issue IMO. Wonder why it worths a new release.

Labels: -Priority-Medium Priority-High

[giampaolo]

From g.rodola on December 14, 2011 09:56:31

Labels: Milestone-0.4.1

[giampaolo]

From g.rodola on December 14, 2011 15:18:36

Status: Fixed

[giampaolo]

From g.rodola on December 14, 2011 15:51:42

This is now fixed in 0.4.1 version.

[giampaolo]

From g.rodola on March 02, 2013 04:05:49

Updated csets after the SVN -> Mercurial migration: r1231 == revision 
4328ec6cf2fe