disk_usage raises UnicodeEncodeError when called with unicode string

[giampaolo]

From [email protected] on August 09, 2013 11:17:46

>>> from __future__ import unicode_literals
>>> import os
>>> from psutil import disk_usage
>>> name = "ƒőő"
>>> os.mkdir(name)
>>> disk_usage(name)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File 
"/Users/kparmar/.virtualenvs/tmp/lib/python2.7/site-packages/psutil/__init__.py",
 line 1196, in disk_usage
    return _psplatform.get_disk_usage(path)
  File 
"/Users/kparmar/.virtualenvs/tmp/lib/python2.7/site-packages/psutil/_psposix.py",
 line 99, in get_disk_usage
    st = os.statvfs(path)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: 
ordinal not in range(128) 

What version of psutil are you using? What Python version?  
psutil v1.0.1
python v2.7.3 

On what operating system? Is it 32bit or 64bit version?  
OS X 10.8.4
Win 7 Enterprise 64-bit

Original issue: http://code.google.com/p/psutil/issues/detail?id=416

[giampaolo]

From g.rodola on August 09, 2013 03:04:36

Tested this on Python 3.3 and everything works fine while on Python 2.7 I can 
work around the error by using name.encode('utf8').
It looks like the difference in behavior between mkdir and statvfs is caused by 
how they treat the paths given as input:

- mkdir: 
http://hg.python.org/cpython/file/0152152b09d0/Modules/posixmodule.c#l2651 - 
statvfs: 
http://hg.python.org/cpython/file/0152152b09d0/Modules/posixmodule.c#l7449 This 
looks like a Python bug to me (statvfs should use PyArg_ParseTuple(args, "et")).
I will file an issue on the Python bug tracker.
As for what concerns psutil I don't think we should take any action.

[giampaolo]

From [email protected] on August 09, 2013 03:39:28

I tried using name.encode('utf8') on Windows, but that did not work -

$ cat test.py
# coding: utf-8

from __future__ import unicode_literals
import os
from psutil import disk_usage
name = "ƒőő"
os.mkdir(name)
disk_usage(name.encode('utf8'))


$ python test.py
Traceback (most recent call last):
  File "test.py", line 8, in <module>
    disk_usage(name.encode('utf-8'))
  File "c:\users\sync\sync\vendor\psutil\psutil\__init__.py", line 1020, in disk
_usage
    return _psplatform.get_disk_usage(path)
  File "c:\users\sync\sync\vendor\psutil\psutil\_psmswindows.py", line 98, in ge
t_disk_usage
    raise OSError(errno.ENOENT, "No such file or directory: '%s'" % path)
OSError: [Errno 2] No such file or directory: 'ƒőő'

[giampaolo]

From g.rodola on August 09, 2013 04:43:31

Ah OK! Well, I think we can at least fix that on Windows.

Labels: -Priority-High Priority-Medium

[giampaolo]

From g.rodola on August 09, 2013 05:11:31

Windows fixed in revision 9d233da8d725 .
os.statvfs() bug is now tracked at: http://bugs.python.org/issue18695 .
At this point I think psutil should expose its own fixed version of statvfs() 
on Python 2 and keep using the one provided by the os module on Python 3.

[giampaolo]

From g.rodola on August 09, 2013 06:26:44

I added a pure python workaround for POSIX as of revision c8451ddcf746 .
Re-exposing statvfs() from scratch was too much work.
I consider this fixed.

Status: FixedInHG
Labels: Milestone-1.1.0

[giampaolo]

From [email protected] on August 22, 2013 00:34:56

Shameless Plug: I open sourced a project that we have used quite successfully 
to test our application for Unicode characters. You may find it useful to test 
all psutil APIs. Its available at https://pypi.python.org/pypi/rotunicode .

[giampaolo]

From g.rodola on September 28, 2013 03:06:18

Status: Fixed

[giampaolo]

From g.rodola on September 28, 2013 10:32:46

Closing out as fixed. Release 1.1.0 is now available for download.