s3 client hangs when uploading an object during streaming download of another object #1235
Comments
|
great info thanks! ya almost certainly an aiohttp/asyncio bug. have you tried uvloop and/or different versions of python/asyncio? |
|
oh I see you're latest aiohttp/python nm. |
|
perhaps try running with debug logging for some hints. may need to add extra logging to aiohttp |
|
Some progress on this. I managed to reproduce locally using localstack (docker version), which greatly improved the debug experience. I had tried before with minio but I couldn't reproduce there. I'm just doing and running the script below. I started removing aiobotocore parts of the code to try to reduce the problem as much as possible, arrived at this following minimal example: import asyncio
import aiohttp
from aiobotocore.response import StreamingBody
async def run():
for _ in range(10):
async with aiohttp.ClientSession() as session:
response = await session.get(
"http://localhost:4566/bucket/ae4bbfcdc47f450aa8557abefeba4a5ct",
)
i = 0
async for chunk in StreamingBody(response, response.headers["content-length"]):
i += 1
if i >= 900:
print("Streamed, time to upload")
# It hangs awaiting this
await session.put(
"http://localhost:4566/bucket/output/some_file",
data=b""
)
print("Uploaded")
asyncio.run(run())Removing That's it for now, I'll try to get some more time to continue debugging later/tomorrow. P.S: I also tried with uvloop and some other python/aiohttp versions. Nothing changed, the problem exists with all combinations I tried. |
|
Well, I did something wrong earlier, but this can be definitely be reproduced without any aiobotocore. It just depends on the chunk size and aiobotocore's default of 1024 makes it happen pretty often in this example, not sure why. Anyway, the following reproduces only with import asyncio
import aiohttp
async def run():
for _ in range(10):
async with aiohttp.ClientSession() as session:
response = await session.get(
"http://localhost:4566/bucket/ae4bbfcdc47f450aa8557abefeba4a5ct",
)
i = 0
async for chunk in response.content.iter_chunked(1024):
i += 1
print(f"chunk number {i}")
if i >= 900:
print("Streamed, time to upload")
# It hangs awaiting this
await session.put(
"http://localhost:4566/bucket/output/some_file",
data=b""
)
print("Uploaded")
asyncio.run(run())I guess I should move this issue to aiohttp's repo. Unless you want to keep this around here as well? |
Yes, please. And thank you for the detailed analysis and follow-up! |
|
Linking the aiohttp issue in case someone needs to follow the trace. Thanks for your help getting this triaged! ❤️ And sorry for the original report here, it took a while to be able to reproduce without aiobotocore involvement. |
FYI, you can just transfer the issue next time, from the sidebar on the right. |
|
rockstar status! no apologies needed, this was an awesome one to follow. Definitely will be following that bug |
Describe the bug
We've got some code that downloads a large archive file from S3, extracts it and uploads each file to another key in the same bucket, all using iterators to work on small chunks of data. We noticed that the process got stuck some times (maybe ~30% of the time) while doing so. No timeout or exceptions were raised but the process never finished. After debugging for a while we got to this minimum reproducing example:
Some important observations:
This is pure speculation based on the previous observations: it feels like there's some race condition when returning the downloading connection to the pool and it gets reused by one of the uploads.
Checklist
pip checkpasses without errorspip freezeresultspip freeze results
Local:
Environments:
Local:
EC2 instance:
The text was updated successfully, but these errors were encountered: