Input truncated at first NUL byte

[jeffrey-easyesi]

When the byte-string passed to detect or feed has a NUL (\x00) byte inside it, none of the following bytes are actually fed to the detector. This leads to binary files often being detected as ASCII with high confidence:

>>> cchardet.detect(open('/bin/sh', 'rb').read())
{'confidence': 1.0, 'encoding': 'ASCII'}

Possibly relevant: https://github.com/cython/cython/wiki/FAQ#how-to-pass-string-buffers-that-may-contain-0-bytes-to-cython

[PyYoshi]

Hi @jeffrey-easyesi .
Thank you for reporting!

I'll check this issue.

Thanks.

[PyYoshi]

Hi @jeffrey-easyesi.
Please tell me the version of Python and cChardet you are using.

I tried it with cChardet v2.0.0:

>>> import sys, cchardet
>>> sys.version_info
sys.version_info(major=3, minor=6, micro=0, releaselevel='final', serial=0)
>>> cchardet.detect(open('/bin/sh', 'rb').read())
{'encoding': None, 'confidence': None}
>>> cchardet.detect(open('/usr/bin/vim', 'rb').read())
{'encoding': None, 'confidence': None}

[jeffrey-easyesi]

Python 3.5.2, cChardet v2.0.0

I shouldn't have used /bin/sh as the only example since that's going to look different on different OSes. Try an explicit string like b'ABC\x00\x80\x81'

>>> import sys, cchardet
>>> sys.version_info
sys.version_info(major=3, minor=5, micro=2, releaselevel='final', serial=0)
>>> cchardet.__version__
'2.0.0'
>>> cchardet.detect(b'ABC\x00\x80\x81')
{'confidence': 1.0, 'encoding': 'ASCII'}