JSOUP not sending Proxy-Authorization header

[Serhii-iLchenko]

Hello, I've been trying to parse page through proxy, but cannot connect to proxy server.
Receiveing error: java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.0 407 Proxy Authentication Required"

Here is my code:

Connection connection = Jsoup.connect("https://google.com/");
connection.proxy("x.x.x.x", 1111);
String proxyCredentials = new String(Base64.encodeBase64("username:password".getBytes()));
connection.header("Proxy-Authorization", "Basic " + proxyCredentials);
connection.method(Method.GET);
connection.ignoreContentType(true);
try {
	Response response = connection.execute();
	Document document = response.parse();
	String text = document.body().wholeText();
	System.out.println(text);
} catch (IOException e) {
	e.printStackTrace();
}

And if I use java.net.Authenticator, the connection works OK.

Authenticator.setDefault(new Authenticator() {
	@Override
	public PasswordAuthentication getPasswordAuthentication() {
		return new PasswordAuthentication("username", "password".toCharArray());
	}
});
Connection connection = Jsoup.connect("https://google.com/");
connection.proxy("x.x.x.x", 1111);
connection.method(Method.GET);
connection.ignoreContentType(true);
try {
	Response response = connection.execute();
	Document document = response.parse();
	String text = document.body().wholeText();
	System.out.println(text);
} catch (IOException e) {
	e.printStackTrace();
}

I've read numerous SO threads, the only solution that I've found is using Authenticator.
What am I doing wrong? Please help.

I heve axamined the process with wireshark, and there were no "Proxy-Authorization" header sent at all.
I've also tried setting "Proxy-Authorization" as data and cookies, no result.

While I can use Authenticaor, I think that passing proxy username and password with header should be more convinient.

[jhy]

If the Authenticator method works for you, what's the problem with using that?

From my understanding of how this works, the underlying Java Http handler has special case handling for proxy authentication, and if you're not using the Authenticator, it gets pulled out of the request, which is why you're not seeing it in wireshark.

[Serhii-iLchenko]

Sending proxy username and proxy password through header is kinda connection-specific.
While setting global Authenticator is JVM-specific. If I use several connections with diffrent proxies that requre diffrent usernames and passwords, global authenticator will be much more error-prone.

[jhy]

Got it. In your getPasswordAuthentication method, you can return null if you don't want to provide a password. The method is called on every hit, so you can check the host or URL to decide if you want to provide a password. See https://docs.oracle.com/javase/7/docs/api/java/net/Authenticator.html#getRequestingHost()

[Serhii-iLchenko]

Yeah, this is what I'm done so far. Thank you anyway.

[rublin]

Hi!
For me, even Authenticator doesn't work. I really don't understand how to use the proxy with basic authentication here.

[benignoms]

This only occurs when the connection is with HTTPS. When you use multithreading there is no way to use the Default Authenticator if you need different users. Please help us. How to be able to link the credentials for each connection using JSOUP?

[varra4u]

I think still it doesn't have a way to provide credentials apart from using Authenticator (https://docs.oracle.com/javase/7/docs/api/java/net/Authenticator.html#getRequestingHost()), can someone confirm pls?

[jhy]

I have added direct support for per-request authentication in Jsoup.connect().auth() in #2046. Would be greatly appreciated if folks can test this in a local snapshot version and provide feedback (on the PR - #2046)