Unable to use HTTP proxies #4719

Th0rgal · 2020-04-26T20:36:57Z

🐞 Describe the bug
Hi, I hope I'm not bothering you with a bug that is actually a problem of use on my part but it's been several days that I've been trying by all means to use proxies with AioHTTP but I can't manage to do so. My goal is to make a small client for my app and it must absolutely have an option to be anonymous (through a proxy) because it is very privacy oriented. For my tests I'm just trying to get the contents of the page of one of my other projects (it's called hackedserver, but don't worry, it's a teaser name and it doesn't have much to do with security). When I'm not using a proxy it works great, no worries. But when I use one, I make a big error message.

💡 To Reproduce
You can just try my demo code:

import aiohttp
import asyncio

async def test():
    async with aiohttp.ClientSession() as session:
        async with session.request('GET', "https://hackedserver.info/", proxy = "http://80.187.140.26:8080") as response:
                print("resp:" + str(response))

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(test())
    loop.run_forever()

I am using Python 3.8 with AIoHTTP 3.6.2 (last version if I'm not wrong)

💡 Expected behavior
Here is what I get without a proxy:

resp:<ClientResponse(https://hackedserver.info/) [200 OK]>
<CIMultiDictProxy('Server': 'nginx', 'Date': 'Sun, 26 Apr 2020 20:30:20 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Last-Modified': 'Thu, 05 Mar 2020 19:00:23 GMT', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'ETag': 'W/"5e614c47-3967"', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Content-Encoding': 'gzip')>

📋 Logs/tracebacks

Traceback (most recent call last):
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\proactor_events.py", line 395, in _loop_writing
    self._write_fut = self._loop._proactor.send(self._sock, data)
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 525, in send
    self._register_with_iocp(conn)
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\windows_events.py", line 714, in _register_with_iocp
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 87] Paramètre incorrect

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".\test.py", line 11, in <module>
    loop.run_until_complete(test())
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File ".\test.py", line 6, in test
    async with session.request('GET', "https://hackedserver.info/", proxy = "http://80.187.140.26:8080") as response:
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\client.py", line 480, in _request
    conn = await self._connector.connect(
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 855, in _create_connection
    _, proto = await self._create_proxy_connection(
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 1093, in _create_proxy_connection
    transport, proto = await self._wrap_create_connection(
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python38-32\lib\site-packages\aiohttp\connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host hackedserver.info:443 ssl:default [Paramètre incorrect]

It says that a parameter is incorrect (I translate because it is written in french). I don't understand what parameter could be incorrect.

📋 Your version of the Python

Python 3.8.2

📋 Your version of the aiohttp/yarl/multidict distributions

python -m pip show aiohttp 3.6.2
multidict 4.7.5
yarl 1.4.2

📋 Additional context
I was using aiohttp (client) on windows 10 (64 bits). I can also try on nixos 20.03 if needed (on my laptop).

The text was updated successfully, but these errors were encountered:

webknjaz · 2020-04-26T21:36:29Z

Please check:

if you run aiohttp with Cython extensions or without them (pure-python)
if running through 64-bit Python helps (your log suggests that you use a 32-bit version)
if using Linux help (with or without C-extensions)
if your proxy supports connections to HTTPS resources and whether it works with some other tool like curl or your browsers

Th0rgal · 2020-04-26T22:04:54Z

I'm running a newly installed python with default settings; I believe this is pure python
I reinstalled it in 64 bits, same issue:

Traceback (most recent call last):
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 395, in _loop_writing
    self._write_fut = self._loop._proactor.send(self._sock, data)
  File "C:\Program Files\Python38\lib\asyncio\windows_events.py", line 525, in send
    self._register_with_iocp(conn)
  File "C:\Program Files\Python38\lib\asyncio\windows_events.py", line 714, in _register_with_iocp
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 87] Paramètre incorrect

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".\test.py", line 11, in <module>
    loop.run_until_complete(test())
  File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File ".\test.py", line 6, in test
    async with session.request('GET', "https://hackedserver.info/", proxy = "http://80.187.140.26:8080") as response:
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\client.py", line 480, in _request
    conn = await self._connector.connect(
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\connector.py", line 855, in _create_connection
    _, proto = await self._create_proxy_connection(
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\connector.py", line 1093, in _create_proxy_connection
    transport, proto = await self._wrap_create_connection(
  File "C:\Users\thoma\AppData\Roaming\Python\Python38\site-packages\aiohttp\connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host hackedserver.info:443 ssl:default [Paramètre incorrect]

Linux help: I don't think so. How to check?
I'll try with a chrome extension.

EDIT:
So I tried with foxy proxy and this proxy works well with hackedserver.info (and this is with HTTPS). I can't use google though (privacy error).

webknjaz · 2020-04-26T22:18:15Z

(1) & (3): you need to pay attention to your pip install to find out what you install.

Th0rgal · 2020-04-26T22:23:25Z

(1) & (3): you need to pay attention to your pip install to find out what you install.

C:\Windows\system32>pip3 list
Package           Version
----------------- ---------
aiohttp           3.6.2
altgraph          0.17
appdirs           1.4.3
astroid           2.3.3
async-timeout     3.0.1
attrs             19.3.0
black             19.10b0
chardet           3.0.4
click             7.1.1
colorama          0.4.3
entrypoints       0.3
flake8            3.7.9
future            0.18.2
idna              2.9
isort             4.3.21
lazy-object-proxy 1.4.3
mccabe            0.6.1
multidict         4.7.5
pathspec          0.8.0
pefile            2019.4.18
pip               20.0.2
pycodestyle       2.5.0
pyflakes          2.1.1
PyInstaller       3.6
pylint            2.4.4
pywin32-ctypes    0.2.0
regex             2020.4.4
setuptools        41.2.0
six               1.14.0
toml              0.10.0
typed-ast         1.4.1
wrapt             1.11.2
yarl              1.4.2

Is that what you need?

webknjaz · 2020-04-26T23:05:16Z

No, it's only visible in the log of the pip install command

Th0rgal · 2020-04-26T23:08:52Z

No, it's only visible in the log of the pip install command

I'm probably dumb, but when I type pip install I get this:

C:\Windows\system32>pip3 install
ERROR: You must give at least one requirement to install (see "pip help install")

sorry

webknjaz · 2020-04-27T10:29:16Z

@Th0rgal I mean pip install aiohttp, of course :)

Th0rgal · 2020-04-27T12:49:38Z

@Th0rgal I mean pip install aiohttp, of course :)

Oh okay, I'm sorry. Here is the result:

C:\Windows\system32>pip3 install aiohttp             
Collecting aiohttp        
Using cached aiohttp-3.6.2-py3-none-any.whl (441 kB)
Requirement already satisfied: yarl<2.0,>=1.0 in c:\users\thoma\appdata\roaming\python\python38\site-packages (from aiohttp) (1.4.2)
Requirement already satisfied: attrs>=17.3.0 in c:\program files\python38\lib\site-packages (from aiohttp) (19.3.0)
Requirement already satisfied: multidict<5.0,>=4.5 in c:\users\thoma\appdata\roaming\python\python38\site-packages (from aiohttp) (4.7.5)
Requirement already satisfied: async-timeout<4.0,>=3.0 in c:\users\thoma\appdata\roaming\python\python38\site-packages (from aiohttp) (3.0.1) 
Requirement already satisfied: chardet<4.0,>=2.0 in c:\users\thoma\appdata\roaming\python\python38\site-packages (from aiohttp) (3.0.4)  
Requirement already satisfied: idna>=2.0 in c:\users\thoma\appdata\roaming\python\python38\site-packages (from yarl<2.0,>=1.0->aiohttp) (2.9)   
Installing collected packages: aiohttp                                      
Successfully installed aiohttp-3.6.2

webknjaz · 2020-04-27T13:04:55Z

Using cached aiohttp-3.6.2-py3-none-any.whl (441 kB)

So it uses a pure-python wheel, no C-extensions. Which is sort of expected (https://pypi.org/project/aiohttp/3.6.2/#files doesn't have wheels for Python 3.8). You could force it to build from source tho.

Try out Linux, it'll probably pick up a manylinux wheel unless you use Python 3.8 there too.

Th0rgal · 2020-04-27T13:11:22Z

Alright. So this is working on my linux environment:

✘ thomas@horus  ~  python ./test.py
resp:<ClientResponse(https://hackedserver.info/) [200 OK]>
<CIMultiDictProxy('Server': 'nginx', 'Date': 'Mon, 27 Apr 2020 13:08:43 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Last-Modified': 'Thu, 05 Mar 2020 19:00:23 GMT', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'Etag': 'W/"5e614c47-3967"', 'Strict-Transport-Security': 'max-age=63072000; includeSubDomains; preload', 'Content-Encoding': 'gzip')>

Linux: Nixos 20.03 (kernel 5.5.17)
Python 3.7.6
aiohttp 3.6.2
multidict 4.7.4
yarl 1.4.2

webknjaz · 2020-04-27T13:15:00Z

@Th0rgal py3.7 probably picked up the version with C-extensions so the logic there is probably different. Also, this may be some limitation of the asyncio loop on Windows

webknjaz · 2020-04-27T13:17:33Z

This may be a bug in CPython: https://superuser.com/a/1474491/182499

Th0rgal · 2020-04-27T13:17:53Z

Do you think I could get it to work on windows using Python 3.7.6? Should I specify something to use C-extensions?

webknjaz · 2020-04-27T13:22:32Z

Should I specify something to use C-extensions?

You can force pip use sdist but then you'd need the build toolchain on your machine.

Do you think I could get it to work on windows using Python 3.7.6?

Maybe, I'm not sure. It looks like a Proactor bug: fastapi/fastapi#759.

Th0rgal · 2020-04-27T13:38:14Z

Alright so I installed Python 3.7.6 on windows and this is now working fine. Since this is not really a aiohttp bug, do you want me to close the issue? Thanks for your great help!

webknjaz · 2020-04-27T13:40:38Z

Let's keep it for now. I'm still not sure if we can have some workaround for this in aiohttp.

WH-2099 · 2020-04-29T12:39:24Z

I got the same problem, too. I think it's just like the issue #2245.
And the solution mentioned could help.

Changed in version 3.8: On Windows, ProactorEventLoop is now the default event loop.
https://docs.python.org/3/library/asyncio-platforms.html

ProactorEventLoop is default in Python 3.8 on Windows which breaks proxy support for https urls.

Solution:
if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

dustiwe · 2021-03-26T13:50:57Z

I got the same problem, too. I think it's just like the issue #2245.
And the solution mentioned could help.
Changed in version 3.8: On Windows, ProactorEventLoop is now the default event loop.
https://docs.python.org/3/library/asyncio-platforms.html

ProactorEventLoop is default in Python 3.8 on Windows which breaks proxy support for https urls.
Solution:
if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

THANK YOU !!!!!!!!!!!❤❤❤

isidentical · 2021-05-07T11:41:11Z

This seems like the issue on #4536, and I wonder whether this needs to be solved on the upstream or not?

munding · 2021-07-28T05:39:27Z

Why does this problem arise？what‘s ProactorEventLoop problem？

Dreamsorcerer · 2024-08-23T18:23:55Z

Is this still reproducible?

djhenya · 2024-08-26T11:48:46Z

Is this still reproducible?

Y

Dreamsorcerer · 2024-08-26T19:21:17Z

Is this still reproducible?

Y

Can someone provide me with an up-to-date reproducer? The original doesn't seem to connect to anything anymore.

djhenya · 2024-08-28T09:53:22Z

upd: Not reproducible with aiohttp==3.10.4

Th0rgal added the bug label Apr 26, 2020

isidentical mentioned this issue Jun 7, 2021

async: always use selector loop policy fsspec/filesystem_spec#658

Merged

WH-2099 mentioned this issue Aug 14, 2021

ServerDisconnectedError on subsequent requests only on Py 3.8 #4549

Closed

Dreamsorcerer added the needs-info Issue is lacking sufficient information and will be closed if not provided label Aug 23, 2024

Dreamsorcerer closed this as completed Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to use HTTP proxies #4719

Unable to use HTTP proxies #4719

Th0rgal commented Apr 26, 2020 •

edited by webknjaz

Loading

webknjaz commented Apr 26, 2020

Th0rgal commented Apr 26, 2020 •

edited

Loading

webknjaz commented Apr 26, 2020

Th0rgal commented Apr 26, 2020 •

edited

Loading

webknjaz commented Apr 26, 2020

Th0rgal commented Apr 26, 2020 •

edited

Loading

webknjaz commented Apr 27, 2020

Th0rgal commented Apr 27, 2020 •

edited

Loading

webknjaz commented Apr 27, 2020

Th0rgal commented Apr 27, 2020

webknjaz commented Apr 27, 2020

webknjaz commented Apr 27, 2020

Th0rgal commented Apr 27, 2020

webknjaz commented Apr 27, 2020

Th0rgal commented Apr 27, 2020

webknjaz commented Apr 27, 2020

WH-2099 commented Apr 29, 2020 •

edited

Loading

dustiwe commented Mar 26, 2021

isidentical commented May 7, 2021

munding commented Jul 28, 2021

Dreamsorcerer commented Aug 23, 2024

djhenya commented Aug 26, 2024

Dreamsorcerer commented Aug 26, 2024

djhenya commented Aug 28, 2024

Unable to use HTTP proxies #4719

Unable to use HTTP proxies #4719

Comments

Th0rgal commented Apr 26, 2020 • edited by webknjaz Loading

webknjaz commented Apr 26, 2020

Th0rgal commented Apr 26, 2020 • edited Loading

webknjaz commented Apr 26, 2020

Th0rgal commented Apr 26, 2020 • edited Loading

webknjaz commented Apr 26, 2020

Th0rgal commented Apr 26, 2020 • edited Loading

webknjaz commented Apr 27, 2020

Th0rgal commented Apr 27, 2020 • edited Loading

webknjaz commented Apr 27, 2020

Th0rgal commented Apr 27, 2020

webknjaz commented Apr 27, 2020

webknjaz commented Apr 27, 2020

Th0rgal commented Apr 27, 2020

webknjaz commented Apr 27, 2020

Th0rgal commented Apr 27, 2020

webknjaz commented Apr 27, 2020

WH-2099 commented Apr 29, 2020 • edited Loading

dustiwe commented Mar 26, 2021

isidentical commented May 7, 2021

munding commented Jul 28, 2021

Dreamsorcerer commented Aug 23, 2024

djhenya commented Aug 26, 2024

Dreamsorcerer commented Aug 26, 2024

djhenya commented Aug 28, 2024

Th0rgal commented Apr 26, 2020 •

edited by webknjaz

Loading

Th0rgal commented Apr 26, 2020 •

edited

Loading

Th0rgal commented Apr 26, 2020 •

edited

Loading

Th0rgal commented Apr 26, 2020 •

edited

Loading

Th0rgal commented Apr 27, 2020 •

edited

Loading

WH-2099 commented Apr 29, 2020 •

edited

Loading