Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

put(recursive=True) gives IsADirectoryError when uploading to root of a bucket #491

Closed
jherman3 opened this issue Jun 8, 2021 · 0 comments · Fixed by #496
Closed

put(recursive=True) gives IsADirectoryError when uploading to root of a bucket #491

jherman3 opened this issue Jun 8, 2021 · 0 comments · Fixed by #496

Comments

@jherman3
Copy link

jherman3 commented Jun 8, 2021

import s3fs

s3 = s3fs.S3FileSystem(client_kwargs={"endpoint_url": "http://my-minio-server.url"})
# test_bug is a folder with a few files in it. 
s3.put("/path/to/test_bug", "s3://bug", recursive=True)
print("Done")

This gives

Output:
Traceback (most recent call last):
  File "s3fs_bug.py", line 6, in <module>
    s3.put("/path/to/test_bug", "s3://bug", recursive=True)
  File "/snip/lib/python3.8/site-packages/fsspec/asyn.py", line 72, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/snip/lib/python3.8/site-packages/fsspec/asyn.py", line 53, in sync
    raise result[0]
  File "/snip/lib/python3.8/site-packages/fsspec/asyn.py", line 20, in _runner
    result[0] = await coro
  File "/snip/lib/python3.8/site-packages/fsspec/asyn.py", line 283, in _put
    return await _throttled_gather(
  File "/snip/lib/python3.8/site-packages/fsspec/asyn.py", line 144, in _throttled_gather
    results.extend(await asyncio.gather(*chunk, **gather_kwargs))
  File "/snip/lib/python3.8/site-packages/s3fs/core.py", line 851, in _put_file
    with open(lpath, "rb") as f0:
IsADirectoryError: [Errno 21] Is a directory: '/path/to/test_bug'

I noticed the following while investigating this:

  • This issue is the same regardless of whether s3://bug exists.
  • This issue is the same with any combination of trailing slashes on lpath and rpath
  • The problem does not occur when copying to a subfolder within the bucket: if you replace s3://bug with s3://bug/foo then the problem goes away.
  • This didn't happen on an older version (0.4.2 on python 3.6)

Environment:

  • Dask version: s3fs 2021.5.0, the latest available in conda
  • Python version: 3.8.10
  • Operating System: Ubuntu
  • Install method (conda, pip, source): conda
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant