IllegalArgumentException: Unexpected char ... in header value: ... at com.squareup.okhttp.Headers$Builder.checkNameAndValue (Headers.java:295)

[thest1]

I get crash reports like that:

java.lang.IllegalArgumentException: Unexpected char 0x43a at 101 in header value: Mozilla/5.0 (Linux; U; Android 4.1.2; ru-ru; PMP7170B3G_DUO Build/JZO54K) AppleWebKit/534.30 (KHTML, как Gecko) Version/4.0 МобильныйSafari/534.30
    at com.squareup.okhttp.Headers$Builder.checkNameAndValue (Headers.java:295)
    at com.squareup.okhttp.Headers$Builder.set (Headers.java:275)
    at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.setRequestProperty (HttpURLConnectionImpl.java:526)
    <some 3rd library code I don't have access to>

Seems to happen due to non-ASCII chars in header value.
Similar to #1998. But #1998 is about response header and my smacktrace is about request header.

Thank you!

[swankjesse]

Any chance you can get the 3rd party library to fix? It's difficult to fix this in OkHttp because the request header contains invalid data.

[thest1]

No problem, I can ask library developers try/catch that.

Just wanted you to be aware about the issue.

Imagine some other developer. His project works fine. He adds

 URL.setURLStreamHandlerFactory(new OkUrlFactory(okHttpClient));

After that his project starts crashing. He can't add try/catch becasue it happens in some 3rd party library. He can't fix the wrong header becasue it's in some 3rd party library. He's not hapy with "your header is incorrect" answer because the same header worked just fine for a long time with HttpUrlConnection.

Probably not a big deal, I'm not sure. Just wanted issue to be created, may be helpful in further development :-) Thank you!

[swankjesse]

Yeah, agreed. We decided to make it more strict than it was before on the expectation that most people weren't doing this deliberately.

[cakoose]

When someone passes you a Java String, rejecting non-ASCII (as you do now) is definitely a good idea.

However, the HTTP specification allows more than just ASCII (roughly, any octet except control characters). What happens when the response contains a header with non-ASCII? Do you just get rid of the non-ASCII parts? It might be worth having a byte[] interface to let clients receive non-ASCII header values uncorrupted.

[pbertsch]

I had that issue a few minutes ago.
What I did was adding a Base64 encoded string as Header.

I used Android and the Base64 class with Base64.DEFAULT.

When I change it to Base64.URL_SAFE, its working. Problem is, the server
does not understand it anymore.

Patrick

https://www.streetlife.com.mx
powered by Alpha Wave Systems
On Dec 11, 2015 14:46, "Kannan Goundan" [email protected] wrote:

When someone passes you a Java String, rejecting non-ASCII (as you do
now) is definitely a good idea.

However, the HTTP specification allows more than just ASCII (roughly, any
octet except control characters). What happens when the response contains a
header with non-ASCII? Do you just get rid of the non-ASCII parts? It might
be worth having a byte[] interface to let clients receive non-ASCII
header values uncorrupted.

—
Reply to this email directly or view it on GitHub
#2016 (comment).

[cakoose]

@pbertsch: What encoding does the server say you should use? Is there public documentation for the server you're sending requests to? (Aside: OkHttp should be fine with both DEFAULT and URL_SAFE -- both use only printable ASCII characters.)

[pbertsch]

I used only NO_WRAP and it seems to work at the moment. I will try to write
a test case with the Base64.DEFAULT flag only.

Patrick

https://www.streetlife.com.mx
powered by Alpha Wave Systems
On Dec 11, 2015 15:16, "Kannan Goundan" [email protected] wrote:

@pbertsch https://github.com/pbertsch: What encoding does the server
say you should use? Is there public documentation for the server you're
sending requests to? (Aside: OkHttp should be fine with both DEFAULT and
URL_SAFE -- both use only printable ASCII characters.)

—
Reply to this email directly or view it on GitHub
#2016 (comment).

[cakoose]

Ah, that's probably it. Newlines aren't allowed as header values.

[carlosEleao]

Hi, this restriction about just ASCII chars is very strange, we support chinese users etc... So almost all of our request fail when we try to send some chinese value using the header. This works fine in the HttpRequest apache. Why this restriction?

[DASAR-zz]

@gudomau This restriction comes from official HTTP spec.

[cakoose]

Non-ASCII characters are allowed according to both the original and updated HTTP 1.1 specs.

The updated spec recommends against using non-ASCII characters for newly-defined headers, but clearly they're allowed:

Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOULD limit their field values to US-ASCII octets. A recipient SHOULD treat other octets in field content (obs-text) as opaque data.

Because of the way UTF-8 is encoded, all codepoints above 127 are composed of bytes whose values are 128 or greater, which means they should pass through untouched. The only characters you have to be careful with are ASCII control characters and, in certain situations, spaces, quotes, and backslashes.

[carlosEleao]

How can I do that since all headers are validated by the checkNameAndValue ?

[peterArconsis]

The HTTP Spec actually uses the word SHOULD, so isn't this restriction in the http client too harsh?

[Panthro]

so you're saying that "ç" and "é á ó" are "invalid data in headers"?

Sorry but I respectfully disagree, the RFC does not say that, is the client being too strict.

In my scenario, what's happening is that I use pagination with the Link header, but when my users are querying my API with special chars eg "ç ã é" this error is being thrown =|

[Panthro]

Alright, I can see the issue is fixed in new releases as mentioned #1998 sorry for my comment in an outdated thread.

[henryleu]

Whatever I use whether android.util.Base64.DEFAULT or URL_SAFE, as flag for Base64.encodeToString(...), it throws IllegalArgumentException.

        byte[] sigBytes = hmac.doFinal(reqStr.getBytes());
        String sig = Base64.encodeToString(sigBytes, Base64.URL_SAFE);
...

        Request.Builder builder = request.newBuilder()
                .header("x-guoer-ts", signature.ts)
                .header("x-guoer-nonce", signature.nonce)
                .header("x-guoer-sig", signature.sig);


java.lang.IllegalArgumentException: Unexpected char 0x0a at 44 in x-guoer-sig value: IkObQzVpEFRv06bEwc5-CnSUO4rqYX_9Gdqs4qEJ_vI=


	at okhttp3.Headers$Builder.checkNameAndValue(Headers.java:320)
	at okhttp3.Headers$Builder.set(Headers.java:300)
	at okhttp3.Request$Builder.header(Request.java:164)

[henryleu]

I regret that I used base64 which is always a trouble. I should use hex for it is stable and deterministic ascii. the code is running in production in iOS, it is hard to change now.

[cakoose]

Character 0x0a is newline. I'm guessing sig contains a newline at the end. The docs say you can pass in Base64.NO_WRAP to omit newlines.

For OkHttp: it might be good to quote strings in error messages so the mistake is more clear. Java string literal quoting or JSON quoting seem reasonable.

[swankjesse]

Also consider using ByteString.base64() which might be a bit easier.

[castaldichristian]

Hi, just switched from Volley and this is the only issue we found, we understand what the standard suggests, however we have cookies we need to send in the header that contain Russian characters.

Is there a way we can bypass the checkNameAndValue method?

[cakoose]

@kriskast: What value are you passing to OkHttp and what error are you seeing, exactly?

[castaldichristian]

Caused by: java.lang.IllegalArgumentException: Unexpected char 0x416 at 9 in User-Agent value: M.R.User=Ж

builder.addHeader(entry.getKey(), "M.R.User=Ж");

This is only a small extract of it as the cookie string is longer and has more Russian characters.

[swankjesse]

Thats not valid HTTP. You might need to base64 encode it client-side and decode it server-side. ByteString from Okio can help with that.

[castaldichristian]

Thanks for your response, to Clarify

builder.addHeader("Cookie", "M.R.User=Ж;"); where the value is part of a cookie that is generated by .NET and we have no access on how it is produced. Also we don't have access to the server to be able to decode.

What do you mean by not valid HTTP?

[swankjesse]

What happens when you use this cookie in a browser like Chrome or Safari?

[swankjesse]

Also – make sure you're using the latest OkHttp. We recently fixed things to be consistent with major web browsers.

[cakoose]

Just to be clear, HTTP is a byte-oriented protocol so I'm assuming you want "Ж" to be encoded as UTF-8, which would be a two-byte sequence: 0xd0, 0x96.

Loophole: use Headers.of(...), which doesn't seem to do any validation, though this is probably dangerous to rely on. (This only works because OkHttp's HTTP 1 and 2 codecs encode/decode header values as UTF-8, which seems problematic in general but oh well.)

[cakoose]

@swankjesse: Excluding control characters, UTF-8-encoded text is valid (in some sense) as an HTTP header value. See: #2016 (comment)

[castaldichristian]

Just checked on Chrome, looks like UserTitle=женский from the server is being encoded into UserTitle=%D0%B6%D0%B5%D0%BD%D1%81%D0%BA%D0%B8%D0%B9 on chrome, and if I send both of them back to the server both are accepted, would you know how I can convert it to the one in chrome so I send that?

[castaldichristian]

Correction, if I send UserTitle=%D0%B6%D0%B5%D0%BD%D1%81%D0%BA%D0%B8%D0%B9 to the server it seems it is not being decoded back, looks like chrome only displays it like that but actually sends UserTitle=женский in the background.

@cakoose 's method is the best I found so far of using Headers.of(...) which skips validation

Thanks everyone

[smart1024]

@pbertsch good job

[vishna]

Headers.of() is your friend if you want non-ASCII characters in your headers... at least for know, unless someone decides to add ASCII validation there.

[swankjesse]

@vishna yikes. #4283

[rohitkum28]

Hi,

I am trying to send "tête-à-tête.pdf" in my header and I am getting the same exception. But, I am able to send the same header from either Postman or using HttpsURLConnection. So, I don't understand why this restriction in okhttp when HTTP protocol supports this.

[swankjesse]

Try this: https://square.github.io/okhttp/4.x/okhttp/okhttp3/-headers/-builder/add-unsafe-non-ascii/

[rohitkum28]

Try this: https://square.github.io/okhttp/4.x/okhttp/okhttp3/-headers/-builder/add-unsafe-non-ascii/

Thanks! This works. I am suprised why was this not mentioned in this thread before.

[yschimke]

@rohitkum28 If you ask and self answer on stackoverflow, I'll bump up both. 25 points right there :)

[rohitkum28]

@rohitkum28 If you ask and self answer on stackoverflow, I'll bump up both. 25 points right there :)

I did ask the question on StackOverflow and now that I know the answer I added it on Stackoverflow.

[castaldichristian]

Try this: https://square.github.io/okhttp/4.x/okhttp/okhttp3/-headers/-builder/add-unsafe-non-ascii/

Thanks! This works. I am suprised why was this not mentioned in this thread before.

thats because it wasn't possible at the time with the current version

[vvelmurugan]

Try this: https://square.github.io/okhttp/4.x/okhttp/okhttp3/-headers/-builder/add-unsafe-non-ascii/

Thanks! This works. I am suprised why was this not mentioned in this thread before.

thats because it wasn't possible at the time with the current version

is this method is renamed as addLenient() now?