Support for proxy hosts black listing in ProxyOptions

[haithkris]

Hi, we have this issue when we want to ignore the proxy and when it comes to call certain components of our internal network. Since vertx ignores JVM proxy arguments like http.nonProxyHosts there is no real workaround.

The proxy configuration is set at the WebClient level. ProxyOptions object should have an equivalent to http.nonProxyHosts.

Vertx web version: 3.5.1
Vertx core version: 3.5.1

[vietj]

is that something you can contribute ?

[haithkris]

I would if had proper time and proper knowledge on how vertx works in the internal. But I was hoping you guys can pull it through.

[vietj]

ok, i'm keeping this as feature request

[marcottedan]

Actually I believe it's more "whitelisting" than "blacklisting" since the noProxy feature of the JVM (and docker network has the same feature) is that everything goes to the proxy except the noProxy hosts.

Edit for references:

• https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html
• https://docs.docker.com/network/proxy/#configure-the-docker-client

[KowalczykBartek]

Hi, I crafted something today,
of course there is a lot of to do in this PR still (only HttpClientImpl covered), but I wanted to ask what do you think about API,
cc @haithkris

new ProxyOptions()
  .setType(ProxyType.HTTP)
  .exludeHost("localhost") //
  .exludeHost("localhost2") //
  .setHost("google.com")
  .setPort(proxy.getPort());

[haithkris]

looks good for me! if we can make accept a list of string or something like that it could be better, but just having the behavior is already good! thanks

[vietj]

the setter should be setExcludedHosts(List<String>) and there should be a method addExcludedHost(String) that adds element to the list.

Look at how other options handle this.

[KowalczykBartek]

What with NetClient ? should ProxyOptions include option to exclude SocketAddress ?

NetClient connect(SocketAddress remoteAddress, Handler<AsyncResult<NetSocket>> connectHandler);

[vietj]

that's a concern I had indeed, but ProxyOptions don't support @VertxGen so I think the best would be to have the excluded host to begin with / instead and we should parse them accordingly when we build the proxy

[KowalczykBartek]

so I think the best would be to have the excluded host to begin with / instead and we should parse them accordingly when we build the proxy

can you explain it a little bit ?

[vietj]

well actually I don't think we need to do that because one can still create 2 net clients if he wants to use domain sockets.

so just a plain list of excluded hosts is fine for me, we don't need to address the port.

[KowalczykBartek]

hmm, so, I guess that I can ask you for review,
If someone wants to test it "in action", I played as follow:

CountDownLatch a = new CountDownLatch(1);

Vertx v = Vertx.vertx();

ProxyOptions proxyOptions = new ProxyOptions();
proxyOptions.setHost("targethost");
proxyOptions.setPort(8080);
proxyOptions.addExcludedHost("exludedhost1");
proxyOptions.addExcludedHost("exludedhost2");

HttpClient client = v.createHttpClient(new HttpClientOptions()
        .setProxyOptions(proxyOptions));

client.getNow(8080,"exludedhost1","/", request -> {
    System.out.println(request);
    a.countDown();
});

a.await();

and /etc/hosts

127.0.0.1 targethost
127.0.0.1 exludedhost1
127.0.0.1 exludedhost2

P.S As I played with this proxy feature, I noticed that it doesn't work with HTTPS traffic, ProxyChannelProvider do not understand https, generates HTTP request, then this proxied http response is handled by handler(SslHandler) that expects to have SSL record.
this is request that fails

ProxyOptions proxyOptions = new ProxyOptions();
proxyOptions.setHost("targethost");
proxyOptions.setPort(8080);
proxyOptions.addExcludedHost("exludedhost1");
proxyOptions.addExcludedHost("exludedhost2");

HttpClient client = v.createHttpClient(new HttpClientOptions()
        .setSsl(true)
        .setTrustAll(true)
        .setVerifyHost(false)
        .setProxyOptions(proxyOptions));

did I do something wrong ? maybe I can fix this in this branch as well ?

[vietj]

@KowalczykBartek it is possible to test it using configuration of the Vert.x AddressResolver, look at HttpTLSTest which uses it

[vietj]

@KowalczykBartek it's not clear what you mean with https. The HttpClient supports HTTPS proxy using the connect method, and the traffic will be encrypted after the proxy issues an CONNECT request to the server. This is explained here https://vertx.io/docs/vertx-core/java/#_using_a_proxy_for_http_https_connections

[vietj]

I think what you are missing is the ChannelConnect#doConnect method that here will make the decision to use or not a proxy:

    if (options.getProxyOptions() == null || !ssl && options.getProxyOptions().getType()== ProxyType.HTTP ) {
      channelProvider = ChannelProvider.INSTANCE;
    } else {
      channelProvider = ProxyChannelProvider.INSTANCE;
    }

[vietj]

the boolean usesProxy is only used for non https proxies that will rewrite the URL for the proxy as a full uri

[KowalczykBartek]

ok, so it was my fault, I didn't know about this CONNECT request to create tunnel in case of HTTPS request :/ sorry :)
I need to read about that, and understand those tests, are a little bit tricky

NetSocket clientSocket = result.result();
serverSocket.write("HTTP/1.0 200 Connection established\n\n");
serverSocket.closeHandler(v -> clientSocket.close());
clientSocket.closeHandler(v -> serverSocket.close());
Pump.pump(serverSocket, clientSocket).start();
Pump.pump(clientSocket, serverSocket).start();

[vietj]

I'm wondering if we really need this feature, i.e if you don't need proxying then just create another HttpClient / NetClient that does not use a proxy and it should work in a simpler manner

[KowalczykBartek]

That is question to @haithkris , i just wanted to make some code :)

[vietj]

oh right, if you are looking for contributions, we can guide you and assign you tasks

[marcottedan]

The goal of the feature seems to handle the Vertx Proxy settings like the JVM does: https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html

[vietj]

@marcottedan the JVM has a global handling (static), in vertx you can create (lightweight) many different clients with different configurations, so it's fine to create one HttpClient with proxy settings and one HttpClient without proxies and let the application chose the right client

[KowalczykBartek]

@vietj do you mean some stuff not present/listed in Issues vert.x tab ?

[vietj]

@KowalczykBartek I got your mail and will reply to you soon, sorry for the delay

[laurentvaills]

Hi guys.
Any news on this feature ? It seems unfortunately inactive since Sept 2018.
Cheers.

[vietj]

we should review it and merge it I think

[laurentvaills]

Thanks @vietj .

[cescoffier]

@vietj any update on this one?

[vietj]

it is not yet scheduled to be implemented by the team

[gurusreekanth]

Hi Julien,

Is there any plans to port this fix to 3.9.x releases?

Thank you.

[vietj]

there are no plans, we only support 3.9.x for bug fixes, sorry

…

On Tue, Feb 8, 2022 at 6:22 AM Gurusreekanth ***@***.***> wrote: Hi Julien, Is there any plans to port this fix to 3.9.x releases? Thank you. — Reply to this email directly, view it on GitHub <#2600 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABXDCQX2J5N76PYTEJGGD3U2CSA3ANCNFSM4FRINTHA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>