S3Boto3StorageFile always returns bytes iterator (not string)

[icpetcu]

When I run this code:

import csv
storage = S3Boto3Storage()
with storage.open('test.csv', 'r') as f:
    reader = csv.reader(f)
    print (list(reader))

I get the following message:

Error: iterator should return strings, not bytes (did you open the file in text mode?)

The problem is that the _file property of S3Boto3StorageFile is always instantiated in 'w+b' mode in order to write the contents from the s3boto3 Object. Therefore, when I read from that file, the iterator always returns bytes, and not string.

I've worked around this problem by using the codecs library:

import codecs
import csv
storage = S3Boto3Storage()
with storage.open('test.csv') as f:
    reader = csv.reader(codecs.iterdecode(f, 'utf-8'))
    print (list(reader))

But I still think this issue should be resolved, since the S3Boto3Storage _open() function has a mode parameter.

[jcushman]

+1, I was bit by this as well. With regular file storage, storage.open('test.csv', 'r').read() returns a unicode str, while storage.open('test.csv', 'rb').read() returns bytestring. With S3Boto3Storage both return bytestring. S3Boto3Storage should follow the same function signature as file storage.

[jschneier]

I think we can just pass mode=self._mode to SpooledTemporaryFile.

[nnseva]

Hi everybody,

I've fixed the issue and made a PR (see above). Using S3Boto3Storage is now transparent, compatible with the Django provided FileStorage and doesn't require additional code. Please push package originator to include a PR into the next release

[nicusX]

Any chance the PR is going to be merged soon?
This is really a blocking issue for many use-cases

[chescales]

Also blocking for us, any updates? Thanks!

[jschneier]

Fixing in #827.

[aquadila]

I'm experiencing this very same issue with the GoogleCloudStorage backend. Any chance of a fix there too?

Should I open a new issue perhaps?

[jschneier]

Can you open a PR?