EOF occurred in violation of protocol starting Python3.10 on large requests

[3asm]

Bug report

Bug description:

We have identified a regression starting with Python 3.10 when making a large request over TLS.

import requests
requests.post('https://google.com', data=b'A'*1000000)

urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:2426)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))

The same request works fine on Python 3.9.

CPython versions tested on:

3.10, 3.11, 3.12

Operating systems tested on:

Linux

Linked PRs

• gh-110467: Fix EOF occurred in violation of protocol starting Python3.10 on large requests #115273

[hugovk]

This issue tracker is for CPython. This looks like a bug in either https://github.com/psf/requests or https://github.com/urllib3/urllib3, please report to one of their trackers.

[3asm]

My mistake. Here is the same issue without urllib3 or requests:

import http.client
http.client.HTTPSConnection("www.google.com").request('GET', '/', body=b'A'*1000000)

No issue on Python3.9, crashes on 3.10 onward with the following statck trace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1077, in _send_output
    self.send(chunk)
  File "/usr/local/lib/python3.10/http/client.py", line 999, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python3.10/ssl.py", line 1270, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/lib/python3.10/ssl.py", line 1239, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2426)

[hugovk]

Thanks, reopening as reproducible using the stdlib.

[pquentin]

Is the same version of OpenSSL used with Python 3.9 and Python 3.10?

[pquentin]

Looking a bit more closely, sending a body using a GET request is not supported by the HTTP standard. www.google.com is just forcefully closing the socket at some point because the request cannot be valid. The cutoff appears to be between 100KB and 1MB. While it could be interesting to look at the difference between Python 3.9 and Python 3.10 (and the version of OpenSSL used), nothing seems wrong here.

In urllib3, sending multiple gigabytes of data over OpenSSL is part of our test suite, and it does work well on Linux. (We do have an issue with that test being slow on macOS.)

[pquentin]

The Python 3.10 release notes also mention changes around ssl.OP_IGNORE_UNEXPECTED_EOF that seem relevant.

[3asm]

To help understand the difference between Python3.9 and Python 3.10, SSLKEYLOGFILE can be used to decrypt the TLS traffic. I am using requests because it is convenient to use.

import os
import requests
os.environ["SSLKEYLOGFILE"] = "secrets.log"
req = requests.get('https://google.com')
req = requests.post('https://google.com', data=b'A'*100000)

The secrets.log can then be opened Wireshark, in Edit > Preferences > Protocols > TLS and set in (Pre)-Master-Secret log filename.

From the traffic, the TLS does break on Python 3.10.

[narayanacharya6]

Is there a way to solve this other than moving to a version of Python other than 3.10?

[keepworking]

import http.client
import requests

try :
    print("START send http.client.HTTPSConnection")
    http.client.HTTPSConnection("google.com").request('POST', '/', body=b'A'*1000000)
    print("END send http.client.HTTPSConnection")
except Exception as e:
    print(e)

try :
    print("START send requests.post")
    requests.post('https://google.com/', data=b'A'*1000000)
    print("END send requests.post")
except Exception as e:
    print(e)

result :

orange@32thread-server:~/project/pythonssleof$ python3.9 main.py
START send http.client.HTTPSConnection
[Errno 32] Broken pipe
START send requests.post
END send requests.post
orange@32thread-server:~/project/pythonssleof$ python3.10 main.py
START send http.client.HTTPSConnection
EOF occurred in violation of protocol (_ssl.c:2426)
START send requests.post
HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2426)')))
orange@32thread-server:~/project/pythonssleof$

Hello,

It seems that it is rather the right situation for an error to occur in the above test case situation.

Python 3.9 also encountered errors when sending large amounts of data to Google with HTTPS Connection. However, 'requests' does not cause an error. Python 3.10 is error in both transmission methods. This is not an error that occurred in python3.10, but simply seems to have failed to deal with exceptions in the implementation of requests or urlib3 as the type of exception changed.

In this regard, I think the problem is that requests in python3.9 version did not return the exception. Even in python 3.9, the data sent by the user through requests would have been interrupted without being fully transmitted. Users should also be aware that their data has not been sent successfully. This is vulnerable to tls truncation attack.

[ankitmaloo]

Now that this won't be picked up, is there a way we can circumvent it?

Like they have done here: websocket-client/websocket-client#942

I encountered the error while using Google API python client as it uses http.Client under the hood.

[keepworking]

I think the original python 3.9 error format changed from 3.10 is the cause of the issue now. I need to investigate why the error type changed from 3.10. I'll check if there's a good reason for the change or if it's simply a side effect by the upgrade and upload the investigation back here.

[3asm]

Some further info as had to look into this and find a workaround. The issue is a combination of peculiar behavior of the backend and the client. For instance, in our case, we are seeing the issue with GCP's GKE LB, which others are facing with other non-python clients. See issue.

We are however seeing that for instance using requests on Py3.9 works, while it fails on newer versions. I can also see a difference in the settings of the TLS connections when dumping traffic between the two.

[Sameera2001Perera]

Is there a way to solve this other than moving to another python version?

[ankitmaloo]

The python versions below 3.10 do not show this error, they do not solve it. Versions above 3.10 do not solve it either. You can do something like this as a work around.

For the record, this code will always give an error in 3.10

import requests

try :
    print("START send http.client.HTTPSConnection")
    http.client.HTTPSConnection("google.com").request('POST', '/', body=b'A'*1000000)
    print("END send http.client.HTTPSConnection")
except Exception as e:
    print(e)

try :
    print("START send requests.post")
    requests.post('https://google.com/', data=b'A'*1000000)
    print("END send requests.post")
except Exception as e:
    print(e)

You change it in this way (copying from another function i wrote so you may have to change a few things here and there.)

fh = io.BytesIO()
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
    try:
         status, done = downloader.next_chunk()
     except HttpError as error:
           if error.resp.status in [404]:
               print('The file does not exist.')
                break
           elif error.resp.status in [500, 502, 503, 504]:
                if num_retries > 0:
                  num_retries -= 1
                  continue
               else:
                   print('A server error occurred and the download failed.')
                   break
               else:
                  print('An error occurred and the download failed.')
                  break
fh.seek(0)

[keepworking]

@encukou, According to #115627, this issue is no longer a cpython issue. I think this is also possible to close.