net/http: add field Cookie.Quoted bool

[gazerro]

For the RFC 6265, the double-quotes are part of the cookie value but the functions and methods in the standard library that operates on cookies treat them as if they were not part of it.

The SetCookie function does not allow to send a cookie, that conforms to the spec, with a double-quoted value and the (*Request).Cookie method strips the quotes from the value despite the double-quotes are part of it.

The syntax in the RFC 6265 is

cookie-pair   = cookie-name "=" cookie-value
...
cookie-value  = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )

but it has been implemented in the standard library as

cookie-pair   = cookie-name "=" ( cookie-value / ( DQUOTE cookie-value DQUOTE ) )
...
cookie-value  = *cookie-octet

The author of the RFC 6265 has confirmed in https://lists.w3.org/Archives/Public/ietf-http-wg/2017JanMar/0229.html that this was the intent.

The draft https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-rfc6265bis-02 added this note to the spec

Per the grammar above, the cookie-value MAY be wrapped in DQUOTE
characters.  Note that in this case, the initial and trailing DQUOTE
characters are not stripped.  They are part of the cookie-value, and
will be included in Cookie headers sent to the server.

and in the appendix reports this discussion https://issues.apache.org/jira/browse/HTTPCLIENT-1006.

[vdobler]

For the RFC 6265, the double-quotes are part of the cookie value but the functions and methods in the standard library that operates on cookies treat them as if they were not part of it.

This is wrong. The optional double-quotes around a cookie are not part of the value. The standard library is correct.

[gazerro]

What version of Go are you using (go version)?

$ go version
go version go1.16.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/marco/Library/Caches/go-build"
GOENV="/Users/marco/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/marco/go/pkg/mod"
GOOS="darwin"
GOPATH="/Users/marco/go"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/z9/xrln_qks56jbzxjbhs04fpm80000gn/T/go-build950868117=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Call an HTTP server that returns cookies with double-quoted values using http.Client.

https://play.golang.org/p/Ka8G_NWPu0n

What did you expect to see?

I expect to see

Cookie: [a=; b=v; c=""; d="v"]

The client sends the cookies to the server preserving the values ​​without stripping the double-quotes.

I also tested it, using this server, with all major browsers (Chrome, Firefox, Safari, Edge, Opera) on MacOS, Windows, iOS and Android with the latest versions and outdated versions (as Internet Explorer 6 on Windows XP, Safari 4 on MacOS Snow Leopard, iPhone 4, default browser on Android 4.4.2) and all browsers display

Cookie: [a=; b=v; c=""; d="v"]

What did you see instead?

Cookie: [a=; b=v; c=; d=v]

The client sends the cookies stripping the double-quotes.

[networkimprov]

cc @neild @empijei @fraenkel

[fraenkel]

I believe there is some confusion over what the Cookie.Value represents. My reading is that its the cookie-octets. The double quotes are inserted as necessary (spaces and commas). There is no way to force the value to always be double quoted.

[neild]

RFC 6265 is quite clear that double-quotes are part of the cookie-value when present. The standards-track RFC 6265-bis emphasizes this point, but the grammar in RFC 6265 is unambiguous.

Empirically, ("net/http".Cookie).Value contains the unquoted cookie-octets from the cookie-value. The (*Cookie.String) function adds surrounding DQUOTE characters if the value contains a space or comma and strips any existing DQUOTE characters from the Value. (Interestingly, the only time (*Cookie).String returns a quoted value is when the value is invalid under RFC 6265, which forbids spaces and commas in values.) The (*Request).Cookie function removes DQUOTE characters from cookies.

[gazerro]

After some investigation, with this message I will explain the source of the problem

• the Cookie.Value field does not represent the cookie value

and its consequences

• a Cookie.Value value cannot be used to represent cookies with values in the form DQUOTE *cookie-octet DQUOTE
• no implementation of the http.CookieJar interface can be standard compliant
• the http.Client type does not conform the RFC 6265 when storing cookies

Also, I will propose some solutions.

Source of the problem

The Cookie.Value field does not represent the cookie value (cookie-value in the grammar of the RFC 6265) but *cookie-octect (as @fraenkel and @neild also noted, although not necessarily as a problem).

Consequences

Consequently, a Cookie.Value value, apart from non standard compliant cookies, cannot be used to represent cookies with values in the form DQUOTE *cookie-octet DQUOTE.

A type that implements the http.CookieJar interface, such as the cookiejar.Jar type, cannot be used to store cookies with DQUOTE because http.CookieJar methods receive and return cookies represented with http.Cookie values. The cookiejar.Jar type is documented as an in-memory RFC 6265-compliant http.CookieJar, but as I pointed out, no implementation of the http.CookieJar interface can conform to the standard.

http.Client uses the http.CookieJar interface to "insert relevant cookies into every outbound Request and is updated with the cookie values of every inbound Response" as required for user agents by RFC 6265. It also requires that the value be stored as is but this is not the case because a http.CookieJar cannot store cookies with value in the form DQUOTE *cookie-octet DQUOTE.

So, if a http.Client with a Jar is used, a cookie received as

Set-Cookie: name="value"

is sent to the server as

Cookie: name=value

instead of

Cookie: name="value"

Note that all the major browsers, latest and older versions, do not alter the cookie value sent to the server.

Solutions

I propose three alternative solutions

a) Standard compliant cookies received with a Set-Cookie header with a double-quoted value are discarded, as allowed by RFC 6265.

b) Add a Cookie.DoubleQuoted boolen field. It is set to true if a double-quoted cookie is parsed from a Set-Cookie header, and if it is true the Cookie.String adds double-quotes to the cookie value.

c) Change the meaning of the Cookie.Value field to represent a cookie value as defined in RFC 6265.

[vdobler]

This issue is about a simple question: Does net/http.Cookie.Value represent the "semantic value" of a cookie or does it represent the raw data that RFC 6265 calls the "cookie-value".

RFC 6265 is not clear here (as it make much statements about how values should be interpreted) but common interpretation has been that the semantic value of a cookie can be optionally enclosed in double quotes or not enclosed. See e.g. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie stating "A cookie-value can optionally be wrapped in double quotes". This interpretation is shared by net/http which treats the double quotes as not being part of the value.

Unfortunately net/http.Cookie mentions RFC 6265 and maybe this lead to this confusion here. I still think that a= and a="" are semantically equivalent Cookie and Set-Cookie headers, both setting a cookie named "a" to an empty (semantic) value and anybody who needs to distinguish between them should process the raw HTTP headers himself and just cannot use net/http.Cookie which provides an interface to the "semantic" cookies, not the raw cookies as sent on the wire.

[neild]

a) Standard compliant cookies received with a Set-Cookie header with a double-quoted value are discarded, as allowed by RFC 6265.

This is obviously not something we can or should do.

b) Add a Cookie.DoubleQuoted boolen field. It is set to true if a double-quoted cookie is parsed from a Set-Cookie header, and if it is true the Cookie.String adds double-quotes to the cookie value.

This seems like the simplest way to preserve double-quoted cookie-values.

c) Change the meaning of the Cookie.Value field to represent a cookie value as defined in RFC 6265.

We could safely change (*Cookie).String to preserve surrounding double quotes in the value when present. However, we clearly cannot change (*Request).Cookie to return a doubled-quoted Cookie.Value without breaking existing users. We could provide a way to configure cookie parsing (e.g., via a configuration flag on Request), but this seems more complicated than option b above.

Preserving the ability to round-trip a cookie-value seems worth the cost of a boolean on Cookie to me. It would be nice to know what the practical negative impact of not preserving double quotes is, however.

[gazerro]

@neild I agree, the only viable option is b. Even if we could break existing users, this option does not force you to manage surrounding DQUOTE characters if you don't want to, and also allows you to adds surrounding DQUOTE characters if you need to.

It would be nice to know what the practical negative impact of not preserving double quotes is, however.

It's a good question. I honestly think no one knows. I also found this similar old issue #10195, closed but not solved.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

By analogy with url.URL.ForceQuery it seems like this should be Cookie.ForceQuote.
We do automatically quote when the value contains space or comma; I don't know if that's correct.
(The grammar seems to say commas are never valid.)

[neild]

We automatically quote when the value contains space or comma, which is weird because spaces and commas are never valid in cookie values. I don't know if automatic quoting is correct or incorrect, but it doesn't seem useful, and it is confusing.

ForceQuote seems fine to me.

[vdobler]

While spaces and commas are never valid in cookie values according to RFC 6265 all browsers (and even curl) support spaces and commas (if quoted) and people rely on this behaviour because a lot of systems, libraries and frameworks use them and they do work irl. (Firefox allows (almost) arbitrary UTF-8 strings as cookie values, at least when I tested it some years ago.)

I'm unsure about ForceQuote as this field is not only used to force quoting a value when sending but it also reports whether the received value was quoted or not. If you just read the documentation of net/url.URL` it's basically impossible to infer that ForceQuery is two-way (at least for me). But I'm fine with a consistent naming.

[rsc]

We can also just call it "Quoted". No need for Force.

[rsc]

Have all remaining concerns about this proposal been addressed?

The proposal is to add a new field ‘Quoted bool’ in http.Cookie.
When parsing a cookie, if double quotes are removed from the value, Quoted is set to true.
When printing a cookie, if Quoted is set to true, double quotes are added back to the value.
Double quotes are still also added implicitly for cookies that contain space or comma, for compatibility with older versions of Go.

[gazerro]

Have all remaining concerns about this proposal been addressed?

The cookiejar.Jar type should work with the new Quoted field. So, when we use the SetCookies method to add cookies, the Cookies method should return the correct value for Quoted. For everything else, it doesn't seem like there's anything missing

[rsc]

Thanks for pointing that out @gazerro. We should definitely update the cookiejar implementation to preserve Quoted. I'm surprised it doesn't just use the Cookie type directly. Maybe that would make more sense. But yes, please consider updating cookiejar part of this proposal.

[rsc]

Have all remaining concerns about this proposal been addressed?

The proposal is to add a new field ‘Quoted bool’ in http.Cookie.
When parsing a cookie, if double quotes are removed from the value, Quoted is set to true.
When printing a cookie, if Quoted is set to true, double quotes are added back to the value.
Double quotes are still also added implicitly for cookies that contain space or comma, for compatibility with older versions of Go.

The net/http/cookiejar implementation also has to be updated to preserve the Quoted field.

[nunogoncalves03]

Since all the concerns seem to have been addressed, can I take on this proposal?

[ianlancetaylor]

@nunogoncalves03 The proposal has not been formally accepted yet. But, sure, you can send a patch for it that can be submitted when and if it is accepted. Thanks.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

The proposal is to add a new field ‘Quoted bool’ in http.Cookie.
When parsing a cookie, if double quotes are removed from the value, Quoted is set to true.
When printing a cookie, if Quoted is set to true, double quotes are added back to the value.
Double quotes are still also added implicitly for cookies that contain space or comma, for compatibility with older versions of Go.

The net/http/cookiejar implementation also has to be updated to preserve the Quoted field.

[gopherbot]

Change https://go.dev/cl/577755 mentions this issue: net/http: add field Cookie.Quoted bool

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

The proposal is to add a new field ‘Quoted bool’ in http.Cookie.
When parsing a cookie, if double quotes are removed from the value, Quoted is set to true.
When printing a cookie, if Quoted is set to true, double quotes are added back to the value.
Double quotes are still also added implicitly for cookies that contain space or comma, for compatibility with older versions of Go.

The net/http/cookiejar implementation also has to be updated to preserve the Quoted field.