Module init fails on missing "Buffers" line in /proc/meminfo

[giampaolo]

From [email protected] on October 07, 2011 15:26:30

What steps will reproduce the problem?  
1. Load the module on a system that doesn't have a "Buffers" line in 
/proc/meminfo 

What is the expected output?  


What do you see instead?  
The module doesn't load:
File "psutil/__init__.py", line 84, in <module>
  File "psutil/_pslinux.py", line 122, in phymem_usage
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType' 

What version of psutil are you using? What Python version?  
0.3.0, Python 2.5 

On what operating system? Is it 32bit or 64bit version?  
Debian 6.0, 64bit 

Please provide any additional information below.  
The system is a virtual machine, cat /proc/meminfo is missing the "Buffers" 
line which causes the module init to fail. I propose defaulting to zero for all 
values and not throwing exceptions when a line is missing.

cat /proc/meminfo:
MemTotal:        1048576 kB
MemFree:          865284 kB
Cached:           107968 kB
Active:           132580 kB
Inactive:          24664 kB
Active(anon):      49208 kB
Inactive(anon):       68 kB
Active(file):      83372 kB
Inactive(file):    24596 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
AnonPages:         49276 kB
Shmem:              2632 kB
Slab:              14632 kB
SReclaimable:       9220 kB
SUnreclaim:         5412 kB

free:
             total       used       free     shared    buffers     cached
Mem:       1048576     183296     865280          0          0     107968
-/+ buffers/cache:      75328     973248
Swap:            0          0          0

Original issue: http://code.google.com/p/psutil/issues/detail?id=218

[giampaolo]

From [email protected] on October 07, 2011 10:48:44

Actually, I think two things need to be done:
1. Never raise an exception for these functions (or only raise documented and 
psutil specific exceptions)
2. Don't run any of that code in the __init__.py file, instead just bind 
function names appropriately

Let me know if you want me to help with any of this or if you disagree.

[giampaolo]

From g.rodola on October 13, 2011 11:11:47

Hi, sorry for the delay in responding.

> 1. Never raise an exception for these functions (or only raise
> documented and psutil specific exceptions)

This error represents a bug in psutil therefore raising a non-psutil exception 
makese perfect sense to me.
We do the same in the underlying C code: when a certain C call fails with an 
unexpected error we raise RuntimeError, which most of the times represents a 
something which should be fixed (and signaled here).

> 2. Don't run any of that code in the __init__.py file, 
> instead just bind function names appropriately

How would you do that exactly?
NUM_CPUS, BOOT_TIME and TOTAL_PHYMEM are all global constants and as such we 
are forced to set them at import time.
Note that I don't like that either, but I don't see how to do that otherwise.
If you can't propose a solution in this sense it is fine with me to default 
"Buffers" to zero.

[giampaolo]

From g.rodola on October 13, 2011 11:13:18

Out of curiosity, it is strange that /proc/meminfo doesn't provide Buffers.
Could you please paste the output of "free" command?

[giampaolo]

From g.rodola on October 17, 2011 15:22:32

Ok, in r1166 we are now retrieving total, free and buffers memory info in C 
instead of parsing /proc/meminfo in python.
This should be more portable and fix your issue.
Can you confirm?

Status: Started
Labels: -Priority-Medium Priority-High OpSys-Linux Milestone-0.3.1 Crash

[giampaolo]

From [email protected] on October 18, 2011 10:13:27

Thank you for your response.

> > 1. Never raise an exception for these functions (or only raise
> > documented and psutil specific exceptions)
>
> This error represents a bug in psutil therefore raising a non-psutil exception
> makese perfect sense to me.
> We do the same in the underlying C code: when a certain C call fails with an
> unexpected error we raise RuntimeError, which most of the times represents a
> something which should be fixed (and signaled here).

Yes, in this case it was to be expected because psutil didn't anticipate this 
line to be missing but there are other instances where psutil handles the case 
of a missing line with an (undocumented) "RuntimeError" exception.

> > 2. Don't run any of that code in the __init__.py file, 
> > instead just bind function names appropriately
>
> How would you do that exactly?
> NUM_CPUS, BOOT_TIME and TOTAL_PHYMEM are all global constants and as such we are
> forced to set them at import time
> Note that I don't like that either, but I don't see how to do that otherwise.
> If you can't propose a solution in this sense it is fine with me to default
> "Buffers" to zero.

I'd suggest getting these values from functions when first needed. The 
functions can cache their results in a static variable so there would be (close 
to) no performance impact.

> Out of curiosity, it is strange that /proc/meminfo doesn't provide Buffers.
> Could you please paste the output of "free" command?

Yes, strange indeed, I have never seen this before. The output of "free" is in 
the initial post.

Unfortunately I don't have access to this machine anymore and so far I haven't 
seen this line missing on any other installation but to me this sounds like a 
good replacement for the function in question.

[giampaolo]

From g.rodola on October 18, 2011 11:10:45

> Yes, in this case it was to be expected because psutil didn't anticipate this line 
> to be missing but there are other instances where psutil handles the case of a 
> missing line with an (undocumented) "RuntimeError" exception.

Other cases are exactly the same: they represent a bug in psutil.
If we document this we should also document all other possible 
errors/exceptions we might have because of a bug inside psutil (TypeError for 
bad args passed to a function, OSError(errno) for an underlying C call which 
has failed, etc...), which obviously we can't do.

> I'd suggest getting these values from functions when first needed.

They are needed at import time because they are provided in form of *global 
constants* of psutil/__init__.py module and they are part of the public API: 
https://code.google.com/p/psutil/source/browse/tags/release-0.3.0/psutil/__init__.py#82
 As such, we are forced to call those functions at import time, no matter what.
This was probably a bad design choice but NUM_CPUS, BOOT_TIME and TOTAL_PHYMEM 
are there since 0.1.0 version, so we can't get rid of them.
The best we can do is try to make them as robust as possible as we did now or 
as I did in r1165 with NUM_CPUS which is currently relying on 3 different 
methods for figuring out the number of cpus.
In case we *can't* follow this strategy we can decide to return a fake value 
(0) for those functions raising RuntimeError, but I'd like to see an issue 
filed first, if you know what I mean.

> Unfortunately I don't have access to this machine anymore

Urgh! :(
Well, according to its manual, sysinfo() is supposed to be there since kernel 
2.3.23, so let's assume this is fixed for now. =)
Thanks for signaling btw, this was a high priority one.

Status: FixedInSVN

[giampaolo]

From g.rodola on October 21, 2011 16:44:17

Labels: -Milestone-0.3.1

[giampaolo]

From g.rodola on October 21, 2011 16:45:27

Labels: Milestone-0.4.0

[giampaolo]

From g.rodola on October 28, 2011 20:44:14

Status: Fixed

[giampaolo]

From g.rodola on March 02, 2013 04:04:24

Updated csets after the SVN -> Mercurial migration: r1165 == revision 
37f787a2c3a8 r1166 == revision 729068fe5f84