How can I use Proxy to send HTTP requests?

[prehawk1999]

I'm using ngx.thread and lua-resty-http to send multiple request, and I want them all go through a proxy, what is the best way to do that?

[kaporzhu]

@prehawk1999 did u get a solution? I need this feature too.

[kaporzhu]

@prehawk1999 原来是自己人 有没有找到解决方案

[prehawk1999]

@kaporzhu Use low level api: http:connect and http:request， use http:connect to connect proxy, and http:request to perform actual request.

[kaporzhu]

@prehawk1999 that's exactly the way I tried this afternoon. It seems I need to check it again. Thanks for ur help.

[iriyak]

@kaporzhu If the origin server expects only HTTPS few extra steps need to be considered. See RFC2817 https://tools.ietf.org/html/rfc2817. Just 2 cents.

[kaporzhu]

@iriyak thanks for your info :=) http is enough for my app.

[iriyak]

@kaporzhu Good to know that HTTP was sufficient in your use case. :) It wasn't to my use case that required HTTPS and I am going to post a new issue to cope with.

[jasonbigl]

@prehawk1999 can you please post a simple code to show how to perform the request by proxy?
I have the same issue here, and I have tried many times without success, thank you very much 😄

Use low level api: http:connect and http:request， use http:connect to connect proxy, and http:request to perform actual request.

[iriyak]

@MrLeex

The origin server can be connected via HTTP or you need to establish HTTPS connection?

[jasonbigl]

@iriyak
Hi,

Thank you very much for your reply, the HTTP would be enough for my app, but I'm in a private network, and all HTTP request need to be made by a proxy like: http://proxy.oa.com:8080, how can I do that?

Thank you once more for your time

[iriyak]

@MrLeex Right, give me some while to prepare.

For your information, key points to consider for going through a non-transparent forwarding proxy include:

1. The target destination to establish TCP connection becomes the proxy, not the origin
2. "When making a request to a proxy, other than a CONNECT or server-wide OPTIONS request (as detailed below), a client MUST send the target URI in absolute-form as the request-target." RFC7230 Section 5.3.2
3. You may not want a cached resource stored by the forwarding proxy

More to come later.

[jasonbigl]

@iriyak Hi, I really appreciate your help, the absolute-form URI fixed my issue, so just set the path to the absolute URI will works great.

To help other people why may have encountered this issue, here is how you can make request via proxy with lua-resty-http:

1. assume my proxy is http://127.0.0.1:8080, first, connect with the proxy using :connect, like:
   local ok, err = httpc:connect("127.0.0.1", 8080)
2. set the path to the absolute URI instead of relative ones using the :request, like:

local res, err = httpc:request{
    method = "GET",
    path = "http://www.google.com/",
    headers = {
              ["Host"] = "www.google.com"
    }
}

But there is still two questions:

1. request to https failed
2. can't follow the 302 temporary move

[prehawk1999]

@MrLeex When dealing with https proxy, or redirect, details should be manipulate by yourself.

            local res, err = instance.httpc:request(
                  {
                      version = 1.1,
                      method = "CONNECT",
                      headers = req_headers,
                      path = tostring(url_host) .. ":" .. tostring(url_port),
                  }
            )

For example, when proxys https, you should send CONNECT request , and then do httpc:ssl_handshake(nil, url_host, true). The https session will be stored when you use keepalive.

Redirect is a much more simple case, as you should only deal with 3xx requests and resend.

[jasonbigl]

@prehawk1999 thanks for your help, gonna try it

[iriyak]

@MrLeex Yes, RFC2817 Section 5.2 describes the formal specification for establishing a tunneling with CONNECT.

Now that multiple HTTP sessions must be handled in your app it is highly recommended to check res, err and add some error handling logic. While my debugging I have experienced a connection timed out caused by set_timeout(), 400/500/504 by the proxy while/after establishing a tunnel.

Just 2 cents

[vkill]

Example: https://gist.github.com/vkill/0a35fc705707f93939f1eee6ddcd4a67