pull request #778 breaks streaming inputs

[fgregg]

PR #778 introduced a seek that is not compatible with stdin and other inputs that do not provide seek.

[jpmckinney]

Did 1.10.1 fix it? Can you provide a way to reproduce?

[jpmckinney]

Aha you edited - I’ll add a test.

[fgregg]

if you are mucking about in there, i'm pretty sure that you will need a seek or other way of resetting the file in this block.

agate/agate/table/from_csv.py

Lines 64 to 67 in 2898d17

	if sniff_limit is None:
	# Reads to the end of the tile, but avoid reading the file twice.
	handle = StringIO(f.read())
	kwargs['dialect'] = csv.Sniffer().sniff(handle.getvalue())

[fgregg]

not an agate test but, here's a csvkit repro.

echo "a" | csvsort

[jpmckinney]

Yeah, It can read from stdin csvsort < examples/dummy.csv, but not from pipes. Not sure how to test pipes in pytest, so I'll add some sample commands to CI.

 if sniff_limit is None: 
     # Reads to the end of the tile, but avoid reading the file twice. 
     handle = StringIO(f.read()) 
     kwargs['dialect'] = csv.Sniffer().sniff(handle.getvalue())

We're passing handle.getvalue() which is an str. The position of the original StringIO doesn't move. Right?

getvalue() can be called consecutively with the same result.

[fgregg]

got it. got confused with the overwriting of handle

[jpmckinney]

I started #780

The docstring says sniff_limit reads bytes, which is true for stdin (now), but it actually reads characters for the original behavior. I don't know if it's worth forcing reading characters in both cases. I'm currently doing something like:

• https://til.simonwillison.net/python/io-bufferedreader
  • simonw/sqlite-utils@99ff0a2
  • https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/cli.py

Would need something like:

• https://stackoverflow.com/a/14292897/244258
• https://github.com/racitup/magicrescue-vmdk/blob/master/vmdk.py
• https://github.com/Lesmiscore/wapysm/blob/master/wapysm/parser/text/bufferedreader.py

I can also check if anyone wrote a peekable TextIOWrapper

I should also see what happens if sys.stdin.buffer is passed in. Edit: buffer is bytes and the old behavior would error on bytes. So, nothing to do there.

Edit: Add test for sniff limit that ends within UTF8 character. Edit2: sys.stdin.buffer.peek() always reads the entire input, up to 65536 bytes (on my machine at least, and in CPython at least). So, a test would be difficult, but I checked the logic with:

import io
io.BufferedReader(io.BytesIO('ʤ,ʤ,c\n1,2,3'.encode()), 4).peek().decode('utf-8')
# UnicodeDecodeError
io.BufferedReader(io.BytesIO('ʤ,ʤ,c\n1,2,3'.encode()), 4).peek().decode('utf-8', 'ignore')
# 'ʤ,'

[jpmckinney]

Thank you for reporting! Should be fixed in 1.10.2.