disk_usage() percent doesn't take reserved blocks into account (-5%)

[ccztux]

Hi!

As described here: NagiosEnterprises/ncpa#102 the disk usage differs between df command and psutil output.

df -a
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/vg_mgmt1-lv_root
                      51475068 20684108  28169520  43% /

python
Python 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.disk_usage('/')
sdiskusage(total=52710469632, used=21180530688, free=28845584384, percent=40.200000000000003)

Is there an explanation about this?

[giampaolo]

psutil metrics are expressed in bytes, while df should show kb. As for the
percentage there's a comment in the source code explaining why it's
different but right now I am on the metro and can't check it.
On May 31, 2016 11:16 AM, "ccztux" [email protected] wrote:

Hi!

As described here: NagiosEnterprises/ncpa#102
NagiosEnterprises/ncpa#102 the disk usage
differs between df command and psutil output.

df -a
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_mgmt1-lv_root
51475068 20684108 28169520 43% /

python
Python 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import psutil
psutil.disk_usage('/')
sdiskusage(total=52710469632, used=21180530688, free=28845584384, percent=40.200000000000003)

Is there an explanation about this?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#829, or mute the thread
https://github.com/notifications/unsubscribe/AAplLMYlAxFmVeCc0xRHzyTXBP8llp2gks5qG_xRgaJpZM4IqVGg
.

[eschava]

# NB: the percentage is -5% than what shown by df due to
# reserved blocks that we are currently not considering:
# http://goo.gl/sWGbH

[jomann09]

Mostly interested in the % since that doesn't give the proper percentage when it doesn't account for the reserved blocks. Do you know of any good way to account for this? Is it always 5% off?

[giampaolo]

So here's my findings. Linux (not sure about other UNIXes) by default reserves 5% of total disk space for the root user so that if, say, the user home directory fills the disk, the root user and the system will still have 5% of space to write into /var/log, not crash in general and also to avoid disk fragmentation:
http://unix.stackexchange.com/a/7964/168884
psutil currently does not take the reserved block into account so that is why % usage is 5% smaller:

psutil/psutil/_psposix.py

Line 144 in c1779c1

# NB: the percentage is -5% than what shown by df due to

tune2fs utility can be used to find the exact value of the reserved block, which is what psutil would need in order to calculate the correct percent value:

~/svn/psutil {master}$ sudo tune2fs  -l /dev/sda6 | grep "Reserved block" 
Reserved block count:     4607763

Parsing tune2fs output is a no-no thought: I just don't want to rely on it. I tried to take a look at tune2fs source code which is available here: http://git.kernel.org/cgit/fs/ext2/e2fsprogs.git. It determines this information via a ext2fs_r_blocks_count() function defined in ./lib/ext2fs/blknum.c but it's not clear to me how the struct it uses is filled. Relying on tune2fs lib is also a no-no as it would require installing the C headers separately (sudo apt-get install e2fslibs-dev on ubuntu).

As for df utility. df is also able to print the right percentage information so it must take the reserved blocks value from somewhere (or "fake it", let's hope not). I tried to strace it (strace df) but I didn't notice anything interesting. The strace output is pretty short, suggesting df doesn't do anything particularly complex but unfortunately it seems the information doesn't come from the /proc fs. I will try to take a look at df source code later.

Another open question is what to do with other UNIX platforms. Is the reserved block something which exists in Linux only? If not how can we determine its value on other UNIX platforms (aka is tune2fs util available?)?

[jomann09]

Maybe I am doing my math wrong but is this not why it's not taking them into account? I'm not normally a Python developer, but this seems to make sense.

In the pstuil code you currently have: used = (st.f_blocks - st.f_bfree) * st.f_frsize

From the Python docs for 2.7.11 here I see that F_BFREE is actually the amount of free blocks total, not the amount that can be used by non-superusers. So when you're doing total blocks - available = used it's giving you the amount available for root, not the amount available for the user, you'd have to use F_BAVAIL to get that... is this something that could be triggered with a flag?

Now I am not sure how reliable the F_BAVAIL is but I am assuming it's grabbing it directly from the C library, statvfs.

As a side note, I would think you could use the two values to find the value of the actual space reserved too without having to add anything to pusitls.

[giampaolo]

Yes, you are right: statvfs() already gives all the necessary bits. Thanks for figuring this out.
But what is wrong is not how psutil calculates the used space: it's the percentage which is wrong.
If you compare df -h and psutil you'll notice total, used and free/avail match, while the percentage doesn't (df reports a higher percentage) so we're doing something wrong in in there.

[giampaolo]

df source code: https://searchcode.com/codesearch/view/17947198/
Aliases for these confusing metrics are created at line 551, the percentage should be calculated later on at line 610.

[giampaolo]

OK, I fixed this here: 3dea30d
It was kinda complicated to get all the math right so I made sure to properly comment every value for future reference. Also, the returned values match df -B 1 perfectly (tests here: d5e56ec).

[jomann09]

Awesome this looks great, thanks for acting on this so quickly! I'm glad it wasn't as complicated as originally thought!

[giampaolo]

Yeah I'm glad too. It's not complicated because you have all the necessary pieces already in place but the math to calculate useful metrics out of those pieces is kinda hard to get right (it took me all day), suggesting a wrapper on top of os.statvfs is good to have.

[ccztux]

Thank you for fixing this!