Set proxy per request

[alamothe]

Hello,
We ported our service to use Vert.x awhile ago. While using Vert.x we still used Apache HTTP client without problems. However, now I tried to port also this functionality to Vert.x and I hit a problem.

It seems that Vert.x HttpClient does not allow to set a proxy per request, only on the client level. We use 1000+ proxies simultaneously to scrape websites.

Is there any workaround? How feasible is to create an instance per request?

[vietj]

do you mean a forward proxy in front of a request ?

…

On Tue, Feb 9, 2021 at 7:57 AM alamothe ***@***.***> wrote: Hello, We ported our service to use Vert.x awhile ago. While using Vert.x we still used Apache HTTP client without problems. However, now I tried to port also this functionality to Vert.x and I hit a problem. It seems that Vert.x HttpClient does not allow to set a proxy per request, only on the client level. We use 1000+ proxies simultaneously to scrape websites. Is there any workaround? How feasible is to create an instance per request? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3795>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABXDCTEBXHUKIGYMH75LRDS6DMFBANCNFSM4XKNOTRA> .

[alamothe]

Yes. I found the docs here: https://vertx.io/docs/vertx-core/java/#_using_a_proxy_for_httphttps_connections

But it has to be set before creating a client.

How heavy it is to create a client for each request?

[vietj]

it is quite lightweight. keep in mind to close the client. we will likely work toward having a setting per HTTP request, however for proxy this has consequence on the connection pooling.

…

On Tue, Feb 9, 2021 at 9:22 AM alamothe ***@***.***> wrote: Yes. I found the docs here: https://vertx.io/docs/vertx-core/java/#_using_a_proxy_for_httphttps_connections But it has to be set before creating a client. How heavy it is to create a client for each request? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3795 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABXDCW6GDJESENZZMDEAIDS6DWGHANCNFSM4XKNOTRA> .

[vietj]

note also that Vert.x 4.1 should allow to connect to an HTTP server without a connection pool which might make this use case easier.

[vietj]

@alamothe I think the point here is pooling because the client might cache a client connection that was established using a specific proxy. In your scenario do you intend to connect to the same server using different proxies ? or will a remote server always use the same proxy ?

[alamothe]

@vietj Thanks so much for working on this!

In our case, we will connect to the same server using different proxies. It is essential that the connection is not reused if proxies are different even though the destination is the same. Even if just proxy authentication is different, it should not reuse the connection.

Can you please tell me if the implemented solution satisfies this?

Is there any way to attach a "token" to a connection, like in Apache HTTP client?

[vietj]

you can now have a ProxyOptions per RequestOptions

I don't understand what you mean by token

[alamothe]

In Apache HTTP client, you can add a custom "token" to a request (connection), and it will reuse only connections with the sam token.

In our scenario (same server, different proxies), will Vert.x HTTP client try to reuse the connection if it was initiated with a different proxy to the same server?

[vietj]

I think to have several tokens you should instead use one instance of HTTP client.

…

On Fri, Jul 2, 2021 at 11:34 AM alamothe ***@***.***> wrote: In Apache HTTP client, you can add a custom "token" to a request (connection), and it will reuse only connections with the sam token. In our scenario (same server, different proxies), will Vert.x HTTP client try to reuse the connection if it was initiated with a different proxy to the same server? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.