files.pythonhosted.org: Painfully slow download with Deutsche Telekom (Fastly peering issue)

[klada]

When downloading files from pythonhosted.org the download speeds are extremely slow in my environment. Even on a 100 MBit/s FTTH connection I hardly get more than 45 KB/s.

wget https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-17 20:46:30--  https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamens »files.pythonhosted.org (files.pythonhosted.org)« … 151.101.1.63, 151.101.65.63, 151.101.129.63, ...
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.1.63|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
Wird in »»virtualenv-16.2.0-py2.py3-none-any.whl.2«« gespeichert.

virtualenv-16.2.0-py2.py3-none-any.whl.2   16%[============>                                                                        ] 308,62K  26,5KB/s    eta 64s    ^

curl -L --output /dev/null --silent --show-error --write-out 'lookup:        %{time_namelookup}\nconnect:       %{time_connect}\nappconnect:    %{time_appconnect}\npretransfer:   %{time_pretransfer}\nredirect:      %{time_redirect}\nstarttransfer: %{time_starttransfer}\ntotal:         %{time_total}\n' 'https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl'
lookup:        0,124598
connect:       0,178949
appconnect:    0,307898
pretransfer:   0,308016
redirect:      0,000000
starttransfer: 0,369662
total:         25,305663

Probably many other people from Germany are facing the same issue, as I have seen this behavior on multiple connections from Deutsche Telekom (largest ISP in Germany). Probably a Fastly/AS54113 peering issue.

My Platform

• Debian Stretch, Linux 4.9 (tried different platforms)
• Connection type:
  • FTTH connection with 100 MBit/s
  • IPv4 internet connection with single NAT (public IPv4 address)
  • Provider: Deutsche Telekom
  • Firewall: OPNsense 19.1
  • Local network: 1 Gbit/s
• Location: Germany
• Misc: I can reproduce the slow download speed on a different connection (DSL, also from Deutsche Telekom) in a totally different area of Germany

DNS Resolution

dig pypi.org A

; <<>> DiG 9.10.3-P4-Debian <<>> pypi.org A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5794
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pypi.org.                      IN      A

;; ANSWER SECTION:
pypi.org.               54      IN      A       151.101.0.223
pypi.org.               54      IN      A       151.101.128.223
pypi.org.               54      IN      A       151.101.192.223
pypi.org.               54      IN      A       151.101.64.223

;; Query time: 54 msec
;; SERVER: 192.168.20.254#53(192.168.20.254)
;; WHEN: Sun Mar 17 20:36:49 CET 2019
;; MSG SIZE  rcvd: 101

dig pypi.org AAAA

; <<>> DiG 9.10.3-P4-Debian <<>> pypi.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25763
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pypi.org.                      IN      AAAA

;; ANSWER SECTION:
pypi.org.               31      IN      AAAA    2a04:4e42:600::223
pypi.org.               31      IN      AAAA    2a04:4e42:200::223
pypi.org.               31      IN      AAAA    2a04:4e42::223
pypi.org.               31      IN      AAAA    2a04:4e42:400::223

;; Query time: 30 msec
;; SERVER: 192.168.20.254#53(192.168.20.254)
;; WHEN: Sun Mar 17 20:37:04 CET 2019
;; MSG SIZE  rcvd: 149

dig files.pythonhosted.org A

; <<>> DiG 9.10.3-P4-Debian <<>> files.pythonhosted.org A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15600
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;files.pythonhosted.org.                IN      A

;; ANSWER SECTION:
files.pythonhosted.org. 21      IN      CNAME   dualstack.r.ssl.global.fastly.net.
dualstack.r.ssl.global.fastly.net. 29 IN A      151.101.1.63
dualstack.r.ssl.global.fastly.net. 29 IN A      151.101.65.63
dualstack.r.ssl.global.fastly.net. 29 IN A      151.101.129.63
dualstack.r.ssl.global.fastly.net. 29 IN A      151.101.193.63

;; Query time: 105 msec
;; SERVER: 192.168.20.254#53(192.168.20.254)
;; WHEN: Sun Mar 17 20:37:15 CET 2019
;; MSG SIZE  rcvd: 162

dig files.pythonhosted.org AAAA

; <<>> DiG 9.10.3-P4-Debian <<>> files.pythonhosted.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42755
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;files.pythonhosted.org.                IN      AAAA

;; ANSWER SECTION:
files.pythonhosted.org. 9       IN      CNAME   dualstack.r.ssl.global.fastly.net.
dualstack.r.ssl.global.fastly.net. 29 IN AAAA   2a04:4e42::319
dualstack.r.ssl.global.fastly.net. 29 IN AAAA   2a04:4e42:200::319
dualstack.r.ssl.global.fastly.net. 29 IN AAAA   2a04:4e42:400::319
dualstack.r.ssl.global.fastly.net. 29 IN AAAA   2a04:4e42:600::319

;; Query time: 41 msec
;; SERVER: 192.168.20.254#53(192.168.20.254)
;; WHEN: Sun Mar 17 20:37:27 CET 2019
;; MSG SIZE  rcvd: 210

Traceroutes

IPv4

traceroute pypi.org

traceroute to pypi.org (151.101.64.223), 30 hops max, 60 byte packets
 1  gate.middleearth.lan (192.168.20.254)  0.329 ms  0.371 ms  0.396 ms
 2  62.155.247.115 (62.155.247.115)  5.003 ms  5.010 ms  5.025 ms
 3  hh-ea8-i.HH.DE.NET.DTAG.DE (62.154.32.218)  16.655 ms  16.678 ms  16.684 ms
 4  hh-ea8-i.HH.DE.NET.DTAG.DE (62.154.32.218)  16.643 ms  16.638 ms  16.679 ms
 5  80.150.168.162 (80.150.168.162)  26.662 ms  26.655 ms  26.680 ms
 6  hbg-bb1-link.telia.net (213.155.135.80)  64.556 ms  62.692 ms hbg-bb4-link.telia.net (62.115.141.110)  24.717 ms
 7  ffm-bb4-link.telia.net (62.115.138.172)  25.049 ms  25.778 ms  26.220 ms
 8  ffm-b12-link.telia.net (62.115.142.47)  24.847 ms  24.554 ms ffm-b12-link.telia.net (62.115.142.5)  25.049 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

traceroute files.pythonhosted.org

traceroute to files.pythonhosted.org (151.101.113.63), 30 hops max, 60 byte packets
 1  gate.middleearth.lan (192.168.20.254)  0.806 ms  0.547 ms  0.491 ms
 2  62.155.247.115 (62.155.247.115)  15.483 ms  15.484 ms  15.474 ms
 3  hh-ea8-i.HH.DE.NET.DTAG.DE (62.154.32.126)  17.342 ms  17.398 ms  17.410 ms
 4  hh-ea8-i.HH.DE.NET.DTAG.DE (62.154.32.126)  13.265 ms  17.316 ms  17.362 ms
 5  80.150.168.162 (80.150.168.162)  28.840 ms  29.128 ms  29.025 ms
 6  hbg-bb1-link.telia.net (62.115.141.112)  29.497 ms hbg-bb4-link.telia.net (213.155.135.84)  29.721 ms hbg-bb4-link.telia.net (62.115.141.110)  29.706 ms
 7  ffm-bb3-link.telia.net (62.115.123.76)  26.486 ms ffm-bb4-link.telia.net (62.115.138.172)  28.574 ms ffm-bb3-link.telia.net (62.115.123.76)  27.441 ms
 8  ffm-b12-link.telia.net (62.115.142.47)  27.515 ms ffm-b12-link.telia.net (62.115.142.5)  27.709 ms  28.199 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

IPv4

curl -vvv -I --ipv4 https://pypi.org/pypi/pip/json

*   Trying 151.101.192.223...
* TCP_NODELAY set
* Connected to pypi.org (151.101.192.223) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=Delaware; serialNumber=3359300; C=US; ST=New Hampshire; L=Wolfeboro; O=Python Software Foundation; CN=www.python.org
*  start date: Sep 18 00:00:00 2018 GMT
*  expire date: Oct 14 12:00:00 2020 GMT
*  subjectAltName: host "pypi.org" matched cert's "pypi.org"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55fe32052e80)
> HEAD /pypi/pip/json HTTP/1.1
> Host: pypi.org
> User-Agent: curl/7.52.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
HTTP/2 200 
< access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
< access-control-allow-methods: GET
access-control-allow-methods: GET
< access-control-allow-origin: *
access-control-allow-origin: *
< access-control-expose-headers: X-PyPI-Last-Serial
access-control-expose-headers: X-PyPI-Last-Serial
< access-control-max-age: 86400
access-control-max-age: 86400
< cache-control: max-age=900, public
cache-control: max-age=900, public
< content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ *.fastly-insights.com sentry.io https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com https://cdn.ravenjs.com; style-src 'self' fonts.googleapis.com; worker-src *.fastly-insights.com
content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ *.fastly-insights.com sentry.io https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com https://cdn.ravenjs.com; style-src 'self' fonts.googleapis.com; worker-src *.fastly-insights.com
< content-type: application/json
content-type: application/json
< etag: "/Z73FUFl8uukw1tLUt+8OA"
etag: "/Z73FUFl8uukw1tLUt+8OA"
< referrer-policy: origin-when-cross-origin
referrer-policy: origin-when-cross-origin
< server: nginx/1.13.9
server: nginx/1.13.9
< x-pypi-last-serial: 4846670
x-pypi-last-serial: 4846670
< accept-ranges: bytes
accept-ranges: bytes
< date: Sun, 17 Mar 2019 19:40:46 GMT
date: Sun, 17 Mar 2019 19:40:46 GMT
< x-served-by: cache-iad2137-IAD, cache-hhn1546-HHN
x-served-by: cache-iad2137-IAD, cache-hhn1546-HHN
< x-cache: HIT, HIT
x-cache: HIT, HIT
< x-cache-hits: 1, 1
x-cache-hits: 1, 1
< x-timer: S1552851646.446026,VS0,VE1
x-timer: S1552851646.446026,VS0,VE1
< vary: Accept-Encoding, Accept-Encoding
vary: Accept-Encoding, Accept-Encoding
< strict-transport-security: max-age=31536000; includeSubDomains; preload
strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: deny
x-frame-options: deny
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-permitted-cross-domain-policies: none
x-permitted-cross-domain-policies: none
< content-length: 68340
content-length: 68340

< 
* Curl_http_done: called premature == 0
* Connection #0 to host pypi.org left intact

curl -vvv -I --ipv4 https://files.pythonhosted.org/packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.0.1.tar.gz

curl -vvv -I --ipv4 https://files.pythonhosted.org/packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.0.1.tar.gz
*   Trying 151.101.1.63...
* TCP_NODELAY set
* Connected to files.pythonhosted.org (151.101.1.63) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc; CN=r.ssl.fastly.net
*  start date: Sep  4 19:42:10 2018 GMT
*  expire date: Apr 14 16:28:35 2019 GMT
*  subjectAltName: host "files.pythonhosted.org" matched cert's "*.pythonhosted.org"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign CloudSSL CA - SHA256 - G3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5590097ede80)
> HEAD /packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.0.1.tar.gz HTTP/1.1
> Host: files.pythonhosted.org
> User-Agent: curl/7.52.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
HTTP/2 200 
< x-amz-id-2: JtyvxRbz2pOO22Vu9KcST0ol5wXA+5vE+02gS8wdFJECeSCUGe88zBDcwi77/ZGHlb9iJbzGtFg=
x-amz-id-2: JtyvxRbz2pOO22Vu9KcST0ol5wXA+5vE+02gS8wdFJECeSCUGe88zBDcwi77/ZGHlb9iJbzGtFg=
< x-amz-request-id: 78DDFB1C0B79BAC9
x-amz-request-id: 78DDFB1C0B79BAC9
< last-modified: Thu, 19 Apr 2018 18:56:10 GMT
last-modified: Thu, 19 Apr 2018 18:56:10 GMT
< etag: "83a177756e2c801d0b3a6f7b0d4f3f7e"
etag: "83a177756e2c801d0b3a6f7b0d4f3f7e"
< x-amz-version-id: 1N.JvpbFmUQle7Hn0tnUuxQqUTsn5iEu
x-amz-version-id: 1N.JvpbFmUQle7Hn0tnUuxQqUTsn5iEu
< content-type: binary/octet-stream
content-type: binary/octet-stream
< server: AmazonS3
server: AmazonS3
< cache-control: max-age=365000000, immutable, public
cache-control: max-age=365000000, immutable, public
< accept-ranges: bytes
accept-ranges: bytes
< date: Sun, 17 Mar 2019 19:41:01 GMT
date: Sun, 17 Mar 2019 19:41:01 GMT
< age: 2333031
age: 2333031
< x-served-by: cache-sea1046-SEA, cache-fra19168-FRA
x-served-by: cache-sea1046-SEA, cache-fra19168-FRA
< x-cache: HIT, HIT
x-cache: HIT, HIT
< x-cache-hits: 7, 1
x-cache-hits: 7, 1
< x-timer: S1552851661.436121,VS0,VE15
x-timer: S1552851661.436121,VS0,VE15
< strict-transport-security: max-age=31536000; includeSubDomains; preload
strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: deny
x-frame-options: deny
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-permitted-cross-domain-policies: none
x-permitted-cross-domain-policies: none
< x-robots-header: noindex
x-robots-header: noindex
< content-length: 1246072
content-length: 1246072

< 
* Curl_http_done: called premature == 0
* Connection #0 to host files.pythonhosted.org left intact

[brainwane]

Thanks for the report. Are you still experiencing this? (I see https://status.python.org/ didn't mention any incidents but it could have missed something.)

[klada]

I have been experiencing these issues ever since I changed my ISP. My new ISP (Deutsche Telekom) has a different connection to Fastly and it looks like they have no direct peering agreement (which BTW is funny, because Deutsche Telekom Capital Partners invests in Fastly).

The issue also depends on the time of day. In the morning connection speeds are fine, in the evening (between 17:00 and 0:00 CEST) they are horrible. I can be 100% sure that this is not related to my actual internet connection, because when I enable VPN on my internet connection I can download at full speed from files.pythonhosted.org. I have talked to other people with Deutsche Telekom as their ISP and they can confirm that they also have the exact same issue with files.pythonhosted.org (or Fastly in general).

As you can see in the traceroute output above I am using Telia as a carrier, but Level3 is also affected (confirmed by other people using Deutsche Telekom).

From what other people tell me it sounds like IPv6 connectivity in the evening is still mostly okay (cannot test myself), so it looks like pretty much only IPv4 is affected. Please file a ticket at your CDN provider, they should make sure that their traffic can actually reach people.

Here are some more wget speed tests from different times of day:

21:03

Actually 80 KB/s is not too bad, it can get worse...

wget -4 https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-18 21:03:58--  https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamens »files.pythonhosted.org (files.pythonhosted.org)« … 151.101.113.63
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.113.63|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
Wird in »»virtualenv-16.2.0-py2.py3-none-any.whl.1«« gespeichert.

virtualenv-16.2.0-py2.py3-none-any.whl.1  100%[====================================================================================>]   1,83M  80,1KB/s    in 24s

2019-03-18 21:04:23 (78,4 KB/s) - »»virtualenv-16.2.0-py2.py3-none-any.whl.1«« gespeichert [1921889/1921889]

08:31

Full speed, as this is in the morning.

wget -4 https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-19 08:31:23--  https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamens »files.pythonhosted.org (files.pythonhosted.org)« … 151.101.113.63
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.113.63|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
Wird in »»virtualenv-16.2.0-py2.py3-none-any.whl.1«« gespeichert.

virtualenv-16.2.0-py2.py3-none-any.wh 100%[=========================================================================>]   1,83M  7,55MB/s    in 0,2s

2019-03-19 08:31:23 (7,55 MB/s) - »»virtualenv-16.2.0-py2.py3-none-any.whl.1«« gespeichert [1921889/1921889]

17:32

It's starting to slow down again...

wget -4 https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-19 17:32:37--  https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamens »files.pythonhosted.org (files.pythonhosted.org)« … 151.101.13.63
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.13.63|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
Wird in »»virtualenv-16.2.0-py2.py3-none-any.whl.4«« gespeichert.

virtualenv-16.2.0-py2.py3-none-any.whl.4                  100%[===================================================================================================================================>]   1,83M  98,9KB/s    in 18s

2019-03-19 17:32:56 (102 KB/s) - »»virtualenv-16.2.0-py2.py3-none-any.whl.4«« gespeichert [1921889/1921889]

19:43

We are at 45 KB/s again...

wget -4 https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-20 19:42:52--  https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamens »files.pythonhosted.org (files.pythonhosted.org)« … 151.101.13.63
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.13.63|:443 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
Wird in »»virtualenv-16.2.0-py2.py3-none-any.whl«« gespeichert.

virtualenv-16.2.0-py2.py3-none-any.whl    100%[====================================================================================>]   1,83M  63,0KB/s    in 41s     

2019-03-20 19:43:33 (45,8 KB/s) - »»virtualenv-16.2.0-py2.py3-none-any.whl«« gespeichert [1921889/1921889]

19:45 (with VPN)

This is from the same computer, but with a VPN connection. Through the VPN I get full speed due to different routing/peering.

wget -4 https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
--2019-03-20 19:45:44--  https://files.pythonhosted.org/packages/6a/d1/e0d142ce7b8a5c76adbfad01d853bca84c7c0240e35577498e20bc2ade7d/virtualenv-16.2.0-py2.py3-none-any.whl
Auflösen des Hostnamen »files.pythonhosted.org (files.pythonhosted.org)«... 151.101.113.63
Verbindungsaufbau zu files.pythonhosted.org (files.pythonhosted.org)|151.101.113.63|:443... verbunden.
HTTP-Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 1921889 (1,8M) [binary/octet-stream]
In »»virtualenv-16.2.0-py2.py3-none-any.whl«« speichern.

100%[===================================================================================================================================================================================================================>] 1.921.889   9,92MB/s   in 0,2s   

2019-03-20 19:45:45 (9,92 MB/s) - »»virtualenv-16.2.0-py2.py3-none-any.whl«« gespeichert [1921889/1921889]

[ewdurbin]

Thank you for the detailed report. I'm submitting this information to our CDN provider to see if they have any insight.

[ewdurbin]

@klada it's probably worth contacting someone at your ISP at the same time to attack this from both sides.

[ewdurbin]

Our CDN provider has noted this as a known issue that they are working to resolve, but is unable to provide a timeline for resolution.

[klada]

I'll also file a complaint at my ISP. I need some time for this though, as I have to collect more data and comparisons to VPN speed through the same line. Usually their argument with issues like these is that it's up to the content provider to make sure the traffic reaches their customers. And that's where peering agreements come into play (sadly).

Are there any mirrors which are not provided by Fastly that German people can use meanwhile? All of the mirrors I have found seem to be dead.

[ewdurbin]

Due to the potential cost of additional bandwidth we are unable to provide an official public mirror as PyPI runs on donated services and infrastructure.

Additionally the size of a full mirror of PyPI has grown to over 4TB, so not many are able to operate a full mirror at this point.

[locdoan12121997]

I met the same problem. Hope to know a workaround.

[floer32]

One workaround is to reuse after your initial download... @locdoan12121997 @klada

So not just pip cache dir but, considering the Wheelhouse pattern for example

• utilities - neither are >1.0 version, so it is your call whether to depend on them directly or to take them as inspiration...
  • https://github.com/level12/wheelhouse
  • https://github.com/datawire/wheelhouse
• reading/blogs
  • https://www.brianthicks.com/post/2014/10/16/speed-up-pip-install-with-wheel/
  • https://medium.com/@amimahloof/how-to-package-a-python-project-with-all-of-its-dependencies-for-offline-install-7eb240b27418
  • not any one post, but if you are using docker, there are some good ways to put the wheelhouse step in the base stage(s) of a multistage build, or bake into a base image etc

some people commit their wheelhouse to their repo, some use a git submodule, some keep it out of source but do save it into their docker setup and so on. some others do a copy on s3 (see: pypicloud; you could synchronize just what you depend on, rather than full pypi)

these won't fix your problem first time around of course.

but after first download, you'll have your issue fixed as long as your dependencies don't change. then even if something is added/changed, and that hasn't been added to your wheelhouse or base image yet ... you have a small delta, so relative wait time is slower even with the internet latency issues you are experiencing

[klada]

Fastly has solved the issue through direct peering with Deutsche Telekom. Connection speeds to files.pythonhosted.org are now fine again.

[Safari77]

Fastly has solved the issue through direct peering with Deutsche Telekom. Connection speeds to files.pythonhosted.org are now fine again.

Fastly has solved nothing.
Telekom: 50 KiB/s, Digital Ocean Frankfurt: 95 MiB/s.
If I use Mullvad VPN with Telekom, I get 3 MiB/s.

traceroute to files.pythonhosted.org (151.101.37.63), 40 hops max, 60 byte packets
 1  _gateway (192.168.178.1)  0.457 ms  0.552 ms  0.667 ms
 2  p3e9bf683.dip0.t-ipconnect.de (62.155.246.131)  8.457 ms  8.463 ms  8.484 ms
 3  62.154.15.242 (62.154.15.242)  10.871 ms  11.355 ms  11.356 ms
 4  62.157.249.186 (62.157.249.186)  11.965 ms  11.967 ms  12.002 ms
 5  ae-2.r21.frnkge13.de.bb.gin.ntt.net (129.250.6.41)  12.275 ms  12.321 ms  12.312 ms
 6  ae-9.r25.amstnl02.nl.bb.gin.ntt.net (129.250.3.77)  22.710 ms  21.897 ms  21.910 ms
 7  ae-8.r03.amstnl02.nl.bb.gin.ntt.net (129.250.2.109)  18.243 ms  18.179 ms  18.230 ms
 8  81.20.67.250 (81.20.67.250)  17.893 ms  17.666 ms  17.752 ms
 9  151.101.37.63 (151.101.37.63)  18.404 ms  18.498 ms  18.497 ms

DO:

traceroute to files.pythonhosted.org (151.101.113.63), 30 hops max, 60 byte packets
 1  46.101.128.254 (46.101.128.254)  2.772 ms  2.789 ms  2.760 ms
 2  138.197.250.180 (138.197.250.180)  1.008 ms  1.005 ms  1.401 ms
 3  138.197.250.157 (138.197.250.157)  0.768 ms  0.781 ms  0.802 ms
 4  fra1.decixfra.fastly.net (80.81.195.54)  0.968 ms  1.020 ms  1.042 ms
 5  151.101.113.63 (151.101.113.63)  0.785 ms  0.810 ms  0.825 ms

[supine]

@Safari77 what resolver are you using?

[Safari77]

I was using unbound with dnscrypt-proxy, just now it resolved files.pythonhosted.org to 151.101.241.63 (50 KiB/s).
Without forwarding configured in unbound, I got 151.101.13.63 (9 MiB/s).
🤓 👍

[SergeyHein]

@klada , thank you for pointing to Deutsche Telekom!
I've similar situation, downloading via Deutsche Telekom at 2-4 Mpbs .
1und1 full speed .
I have same max 100Mbs download rate with 1und1 and Deutsche Telekom tarifs .

Did you find a workaroud?