ServerDisconnectedError on subsequent requests only on Py 3.8

[Gargauth]

🐞 Describe the bug

Aiohttp throws aiohttp.ServerDisconnectedError on 3.8.1 on subsequent requests within scope of single ClientSession, but only on certain domains. There is no such issue on older Python 3.7.6 so it shouldn't be issue on remote server.

💡 To Reproduce

1. Code snippet

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, 'https://www.fit-pro.cz/bcaa-mega-caps-1100-olimp-blistr-30-kapsli')
        print(html)
        # await asyncio.sleep(0.0001)
        html = await fetch(session, 'https://www.fit-pro.cz/bcaa-mega-caps-1100-olimp-blistr-30-kapsli')
        print(html)

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

2. Run -> Throws error on second request
3. Uncomment line await asyncio.sleep(0.0001) (this is just for experimentation, to demonstrate that somehow this issue doesn't occur when there's sleep).
4. Run -> Doesn't throw error on second request

💡 Expected behavior

Both requests should complete successfully sequentially - just like on Python 3.7.

📋 Logs/tracebacks

Traceback (most recent call last):
  File "src/aiohttp-example.py", line 18, in <module>
    loop.run_until_complete(main())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 612, in run_until_complete
    return future.result()
  File "src/aiohttp-example.py", line 13, in main
    html = await fetch(session, 'https://www.fit-pro.cz/bcaa-mega-caps-1100-olimp-blistr-30-kapsli')
  File "src/aiohttp-example.py", line 5, in fetch
    async with session.get(url) as response:
  File "/home/ondra/Documents/asyncio-playground/.venv/lib/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/home/ondra/Documents/asyncio-playground/.venv/lib/python3.8/site-packages/aiohttp/client.py", line 504, in _request
    await resp.start(conn)
  File "/home/ondra/Documents/asyncio-playground/.venv/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 847, in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
  File "/home/ondra/Documents/asyncio-playground/.venv/lib/python3.8/site-packages/aiohttp/streams.py", line 591, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError

📋 Your version of the Python

$ python --version
Python 3.8.1

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.6.2
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: [email protected]
License: Apache 2
Location: /home/ondra/Documents/py37/.venv/lib/python3.7/site-packages
Requires: async-timeout, chardet, attrs, multidict, yarl
Required-by: py37

$ python -m pip show multidict
Name: multidict
Version: 4.7.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/ondra/Documents/py37/.venv/lib/python3.7/site-packages
Requires: 
Required-by: yarl, aiohttp

$ python -m pip show yarl
Name: yarl
Version: 1.4.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/ondra/Documents/py37/.venv/lib/python3.7/site-packages
Requires: idna, multidict
Required-by: aiohttp

📋 Additional context

aiohttp client. Python 3.8.1 (pyenv), venv installed via poetry.

[webknjaz]

but only on certain domains.

Looks like this is a good starting point for debugging. Could you please specify a few URLs where this explodes and (separately) a few URLs where it works. Also, is your Python interpreter coming from some public OS distro or did you compile it?

[Gargauth]

I tried google.com for example, where it worked on 3.8.

As for python interpreter, pyenv, as far as I know, compiles from source. But I tried it on Manjaro/Archlinux's Python 3.8.1 on which it also failed.

[webknjaz]

Is this happening to both HTTP and HTTPS URLs?

[Gargauth]

Yes, it happens on both https and http.

[gligneul]

I've found the same problem in python 3.8.0 from pyenv.
The problem happens when I try to connect to a server in localhost created with aiohttp.

[Gargauth]

Why this got tagged with reproducer:missing? I provided code snippet, which I believe highlights the issue clearly enough. Was it not possible to reproduce?
I just tried the same code snippet again, this time on WinPython 3.7.6 and then 3.8.1 and could reproduce the issue.
On Python 3.7.6 the code snippet provided in first post, will print returned response twice. On Python 3.8.1 it will print response only for the first time, followed by aiohttp.client_exceptions.ServerDisconnectedError

[Emilv2]

I cannot reproduce this issue with the code snippet you provided running with python 3.8.1 on Archlinux, but I do have a similar issue as @gligneul with a test server on localhost.

[webknjaz]

@Gargauth I don't remember why exactly but I probably marked is as missing because it looked like it's not enough. Now that I confirmed that it's reproducible under 3.8-dev from pyenv (on my Gentoo laptop), I can safely mark it as present.

[webknjaz]

@Gargauth please provide HTTP URL with this issue happening. Changing your current reproducer URLs to plain HTTP does not explode. This is needed to confirm if it's related to TLS and it's what's unstable about your reproducer ATM.

By the way, the best way to share a reproducer is to contribute a reliably failing test and mark it as @pytest.mark.xfail per https://pganssle-talks.github.io/xfail-lightning. This would allow somebody to start fixing the behavior later and have some indicator if it's actually fixed.

[WH-2099]

I got this problem on Python 3.8.4 (Windows) too.
And it's interesting that I tried to add a await asyncio.sleep(0) which solve the problem.

[ljavalysos]

I am seeing the same error, but on Python 3.6 (Ubuntu 20.04.1 LTS). The first request would succeed, the second would consistently and immediately fail.

Per @WH-2099, await asyncio.sleep(0) fixed the issue.

Using versions:
Python 3.6.13
aiohttp==3.7.4

[Dreamsorcerer]

Does anybody have any other URLs that this happens on? Trying a few URLs, only the one in the reproducer triggered the issue for me.

I see the error with the original code, but await asyncio.sleep(0) doesn't fix it for me. 0.001 as per the original example works though. This suggests to me that it may be some kind of timing issue with the server. Maybe it disconnects because it's too fast, or maybe some command is getting sent out of order and the server is aborting...

[WH-2099]

In fact I spent a lot of time trying to solve the same problem when I intuitively found these three issues.
And I was able to find effective solutions to alleviate (not completely solve) all of them.

1. Unable to use HTTP proxies #4719
2. Received "Response payload is not completed" when reading response #4581
3. ServerDisconnectedError on subsequent requests only on Py 3.8 #4549

In general, I think these issues are clearly related to the change in the Python 3.8 asyncio module to change the default event loop.https://docs.python.org/3/library/asyncio-platforms.html

And the most important feature that manifests itself is premature disconnection from the underlying connection to the server.
Personally, I would suggest that these three issues be investigated in connection, and I am available these days to assist if needed.
(Please excuse my rudimentary English)

[Dreamsorcerer]

In general, I think these issues are clearly related to the change in the Python 3.8 asyncio module to change the default event loop.https://docs.python.org/3/library/asyncio-platforms.html

Not sure if it changes your theory, but I reproduced the example for this on Linux, which is not using the ProactorEventLoop. I did not try testing in Python 3.7 though.

[gorbunovav]

I am experiencing this with Python 3.9.6 running inside Docker container (official Python image, tag 3.9.6-buster) running on Ubuntu 20.04 LTS (launched in WSL2).

Requests are being sent to:

• https://quasar.yandex.ru/
• https://passport.yandex.com/
• https://mobileproxy.passport.yandex.net/

It is possible to overcome it with asyncio.sleep(0.1), but it works only if I put it before the second request, not after the first one (they are made in a for loop and there is some user input between iterations, so they are not exactly sequential in terms of time)

And it doesn't happen with the requests lib.

[Dreamsorcerer]

* https://quasar.yandex.ru/

* https://passport.yandex.com/

* https://mobileproxy.passport.yandex.net/

I can't reproduce with any of those URLs, still only the original URL.

Also, I've just realised I was reproducing it in an lxc container, but I can't reproduce the issue on my host.

One additional datapoint is that the issue also seems to only appear if reading from the response (i.e. no error when removing resp.text()).

[WH-2099]

@Dreamsorcerer Thanks for the reminder, I'm mainly using the windows platform and am not very familiar with linux, so I was not thinking well.

It looks like the event loop is not the culprit.

However, I still intuitively think that these issues may be caused by the same problem at the bottom.

[gorbunovav]

ServerDisconnectedError.zip

@Dreamsorcerer here is the piece of code which reproduces the issue for me consistently both inside the docker container (Python 3.9.6), on the Ubuntu (WSL, Python 3.8.10) and on the Windows host (Python 3.9.6).

It uses pipenv to manage dependencies, so you want to:

1. install pipenv
2. launch the shell: pipenv shell
3. install dependencies: pipenv install
4. launch get_token.py

It will be asking you for the username and password - you can enter any random data.
After several retries it will ask you to fill captcha - open the url provided and then again you can enter random data instead of the captcha answer and the password.

After that it tries to login again and gives me ServerDisconnectedError.
Uncommenting sleep lines "fixes" the issue.

Strange that ServerDisconnectedError doesn't appear for login attempts before the captcha request (but if I remember correctly it fails in case of the correct login without the captcha).
Also, it seems it doesn't appear if you don't open the captcha url (the server will reply with another error, that the captcha was not opened).

[ingvar-lynn]

python 3.10.6
aiohttp 3.8.1

Same problem. With await asyncio.sleep(0.001) it lasts for ~4k requests, without - for ~1k.

[iamalbert]

python 3.10.6 (docker image python:3.10-slim)
aiohttp 3.8.1

Same problem here. about 5%-10% chances the second request would raise ServerDisconnectedError. Adding await asyncio.sleep(0.001) after each request "solves" the problem.

[gluhar2006]

Maybe it will be useful
Same problem:

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        resp = await response.text()
        print(resp)


async def main():
    urls = [...]
    async with aiohttp.ClientSession() as session:
        fetchers = [fetch(session, url) for url in urls]
        await asyncio.gather(*fetchers)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

problem 'solved'

import aiohttp
import asyncio


async def fetch(session, url):
    async with session.get(url) as response:
        await asyncio.sleep(0.001)
        resp = await response.text()
        print(resp)


async def fetch_batch(urls):
    async with aiohttp.ClientSession() as session:
        for url in urls:
            await fetch(session, url)


async def main():
    urls_batches = [[...], [...]]
    fetchers = [fetch_batch(urls) for urls in urls_batches]
    await asyncio.gather(*fetchers)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

[Bu3a3a]

Hi!

python 3.9.18, aiohttp 3.6.2 -> problem is reproduced (using @Gargauth 's script)
python 3.9.18, aiohttp 3.8.5 -> problem is not reproduced

Can someone explain please, was it fixed in the latest aiohttp versions? Where and how? Just curious :)

Edit:
I'm running tests in docker using python:3.9-alpine3.17 image

[basavyr]

I can confirm that I am facing this issue on macOS Sonoma 14.0 with Python3.10.13 (installed via pyenv).
aiohttp==3.8.6 and asyncio==3.4.3 packages are using within my codebase.

The disconnects were random, but they seem to stop after I added the short sleep

async def get_data(session: aiohttp.ClientSession, api_url: str) -> dict:
    async with session.get(api_url) as response:
        await asyncio.sleep(0.001)
        resp_as_json = await response.json()
        return resp_as_json

[b-phi]

I can confirm that I am facing this issue on macOS Sonoma 14.0 with Python3.10.13 (installed via pyenv). aiohttp==3.8.6 and asyncio==3.4.3 packages are using within my codebase.

The disconnects were random, but they seem to stop after I added the short sleep

async def get_data(session: aiohttp.ClientSession, api_url: str) -> dict:
    async with session.get(api_url) as response:
        await asyncio.sleep(0.001)
        resp_as_json = await response.json()
        return resp_as_json

Adding asyncio.sleep(0) was enough to greatly improve stability for me on Python 3.10, version 3.9.1

async with session.get(url) as response:
    await asyncio.sleep(0)
    upload(response.content)

[smart7324]

Same issue for me on home assistant 2024.1.0 with aiohttp 3.9.1. Sometimes I randomly get a client error: "Server disconnected".

[webknjaz]

asyncio==3.4.3 packages

asyncio is a part of the standard library that used to be needed before Python 3.4. Sounds like you're shadowing imports of what should actually be used.

[webknjaz]

@bdraco here's someone mentioning HA, FYI

[smart7324]

I am having this issue in HA for 1.5 years now. If someone else is having this issue with home assistant and needs more reliability in triggering web commands, I found an alternative using curl:

In the past I used rest_commands, which are using aiohttp, like this:

rest_command:
  abc:
    url: "http://127.0.0.1/api/path?title={{ title }}"
    method: GET
    timeout: 15

... and then in an automation rest_command.abc to call that service.

Here is a very reliable alternative with shell_commands and curl, where you also can define retry options:

shell_command:
  abc: "curl 'http://127.0.0.1/api/path?title={{ title }}' --request GET --location --max-time 15 --retry 1 --retry-delay 5 --fail"

To trigger that service in an automation use shell_command.abc. This works like a charm and it also reports errors in HA error log with curl error codes, so also timeouts and other errors will be logged.

Curl has been very reliable for me in past projects, so I hope this will finally solve the issue for me.

[bdraco]

It seems that #7363 is likely to solve that issue

[smart7324]

Ok, fingers crossed!

But anyways, I prefer using curl as I am very used to it and finally found an easy way to use it from HA and also parametrize it the same way like rest_commands

[jefer94]

I got this error with http://www.oxfordlearnersdictionaries.com/us/wordlists/oxford3000-5000

[Dreamsorcerer]

I can't reproduce anymore, so I suspect the issue was caused by a questionable server closing the connection immediately after saying it would keep it alive. New versions automatically retry, thus working around the problem.