Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Response code wrong after a redirect with a proxy #168

Open
gyfis opened this issue Jun 1, 2020 · 0 comments
Open

Response code wrong after a redirect with a proxy #168

gyfis opened this issue Jun 1, 2020 · 0 comments

Comments

@gyfis
Copy link

gyfis commented Jun 1, 2020

Hi, we've stumbled upon an inconsistency with the response_code attribute after performing a request with a redirect (with followlocation: true) using a proxy. We're using a Squid proxy.

I've setup a simple redirect to a non-existing page using https://cutt.ly (https://cutt.ly/ayL05Ek -> https://testingcurlfollowlocation.com)
The expected response_code is 503, and the response code returned from Ethon (possibly Curl) is 301 for the redirect. I suspect the issue is that the proxy returns the final error.

Here's the code reproducing the issue (we normally use Typhoeus but I've been able to replicate in Ethon as well, so I'm posting the issue here):

> easy = Ethon::Easy.new
> easy.http_request("https://cutt.ly/ayL05Ek", :get, { followlocation: true, proxy: proxy })
=> nil
> easy.perform
=> :recv_error
> ap easy.mirror.options
{
        :httpauth_avail => 0,
            :total_time => 1.161504,
    :starttransfer_time => 0.927618,
       :appconnect_time => 0.412944,
      :pretransfer_time => 0.412976,
          :connect_time => 0.235225,
       :namelookup_time => 6.7e-05,
         :redirect_time => 0.927769,
         :effective_url => "https://testingcurlfollowlocation.com/",
            :primary_ip => "34.66.44.2",
         :response_code => 301,
          :request_size => 259,
        :redirect_count => 1,
           :return_code => :recv_error,
      :response_headers => "HTTP/1.1 200 Connection established\r\n\r\nHTTP/1.1 301 Moved Permanently\r\nDate: Mon, 01 Jun 2020 13:46:54 GMT\r\nContent-Type: text/html; charset=UTF-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nSet-Cookie: __cfduid=d3de837e177f3242281c7e11384afd1d91591019213; expires=Wed, 01-Jul-20 13:46:53 GMT; path=/; domain=.cutt.ly; HttpOnly; SameSite=Lax\r\nSet-Cookie: PHPSESSID=d625u7idkujbh9k0dsrhtms77r; path=/; secure\r\nExpires: Thu, 19 Nov 1981 08:52:00 GMT\r\nCache-Control: no-cache, no-store, must-revalidate\r\nPragma: no-cache\r\nLocation: https://testingcurlfollowlocation.com\r\nVary: Accept-Encoding\r\nCF-Cache-Status: DYNAMIC\r\ncf-request-id: 0311bbabf20000a3332e202200000001\r\nExpect-CT: max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"\r\nServer: cloudflare\r\nCF-RAY: 59c962265ff4a333-ORD\r\n\r\nHTTP/1.1 503 Service Unavailable\r\nServer: squid/4.11\r\nMime-Version: 1.0\r\nDate: Mon, 01 Jun 2020 13:46:54 GMT\r\nContent-Type: text/html;charset=utf-8\r\nContent-Length: 3754\r\nX-Squid-Error: ERR_DNS_FAIL 0\r\nVary: Accept-Language\r\nContent-Language: en\r\n\r\n",
         :response_body => "",
            :debug_info => #<Ethon::Easy::DebugInfo:0x0000562570ba9d48 @messages=[]>
}
=> nil

The response headers contain the expected response code:

> ap easy.mirror.options[:response_headers].split("\r\n"), index: false
[
    "HTTP/1.1 200 Connection established",
    "",
    "HTTP/1.1 301 Moved Permanently",
    "Date: Mon, 01 Jun 2020 13:46:54 GMT",
    "Content-Type: text/html; charset=UTF-8",
    "Transfer-Encoding: chunked",
    "Connection: keep-alive",
    "Set-Cookie: __cfduid=d3de837e177f3242281c7e11384afd1d91591019213; expires=Wed, 01-Jul-20 13:46:53 GMT; path=/; domain=.cutt.ly; HttpOnly; SameSite=Lax",
    "Set-Cookie: PHPSESSID=d625u7idkujbh9k0dsrhtms77r; path=/; secure",
    "Expires: Thu, 19 Nov 1981 08:52:00 GMT",
    "Cache-Control: no-cache, no-store, must-revalidate",
    "Pragma: no-cache",
    "Location: https://testingcurlfollowlocation.com",
    "Vary: Accept-Encoding",
    "CF-Cache-Status: DYNAMIC",
    "cf-request-id: 0311bbabf20000a3332e202200000001",
    "Expect-CT: max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
    "Server: cloudflare",
    "CF-RAY: 59c962265ff4a333-ORD",
    "",
    "HTTP/1.1 503 Service Unavailable",
    "Server: squid/4.11",
    "Mime-Version: 1.0",
    "Date: Mon, 01 Jun 2020 13:46:54 GMT",
    "Content-Type: text/html;charset=utf-8",
    "Content-Length: 3754",
    "X-Squid-Error: ERR_DNS_FAIL 0",
    "Vary: Accept-Language",
    "Content-Language: en"
]
=> nil

Here's how the response headers look when I use curl directly:

curl -IL -x <proxy> "https://cutt.ly/ayL05Ek"
HTTP/1.1 200 Connection established

HTTP/2 301 
date: Mon, 01 Jun 2020 13:38:07 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=da20ea7978135e5d2fbf5db3ef74dc9bc1591018686; expires=Wed, 01-Jul-20 13:38:06 GMT; path=/; domain=.cutt.ly; HttpOnly; SameSite=Lax
set-cookie: PHPSESSID=84vt6lt9mn2an2virt75695if1; path=/; secure
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: no-cache, no-store, must-revalidate
pragma: no-cache
location: https://testingcurlfollowlocation.com
cf-cache-status: DYNAMIC
cf-request-id: 0311b3a15b0000e20acc2ac200000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 59c955489e38e20a-ORD

HTTP/1.1 503 Service Unavailable
Server: squid/4.11
Mime-Version: 1.0
Date: Mon, 01 Jun 2020 13:38:07 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 3789
X-Squid-Error: ERR_DNS_FAIL 0
Vary: Accept-Language
Content-Language: en

curl: (56) Received HTTP code 503 from proxy after CONNECT

Can you advise on how to modify the request to receive the 503 response_code from Ethon (and possibly Typhoeus as well)? We'd like to refrain from parsing the headers manually.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant