qs drops URL Encoded + and & signs

[TomiTiihonen]

Am I doing something wrong:

qs['q'] value for ?q=1%2B1 comes back '1 1' and not '1+1' as I was expecting?

Version: 0.24.3-dev
OS: Windows XP
Python version: 2.7.3

[pjz]

Can you try with 0.25.0 or later? There've been a lot of changes since 0.24.3.

[chadwhitacre]

Confirmed, still a problem in HEAD.

[pjz]

nevermind, repro'd in 0.25.0

[chadwhitacre]

:)

[chadwhitacre]

IRC

[chadwhitacre]

The Aspen request API calls for:

• request.line.uri.querystring—a Mapping of unicode to unicode
• request.line.uri.querystring.decoded—the entire querystring as a unicode
• request.line.uri.querystring.raw—the entire querystring as a bytestring

Here's how we get there from WSGI (this is all in http/request.py; qs in simplate contexts is an alias for the above):

1. WSGI environ goes into kick_against_goad
2. kick_against_goad does its best to rehydrate an HTTP message from a WSGI environment
3. make_franken_uri takes environ['PATH_INFO'] and environ['QUERY_STRING'] and returns a single bytestring
4. the bytestring returned by make_franken_uri goes out of kick_against_goad and in the top of Request as the uri argument
5. Request instantiates Line with uri, which instantiates URI with uri
6. URI splits uri with urlparse.urlsplit, and passes uri.query to Querystring
7. Querystring decodes uri and passes the result to cgi.parse_qs, the result of which is used to instantiate Querystring as a Mapping subclass.

The bug here is in the decoding of uri in Querystring.

[chadwhitacre]

It looks like cgi.parse_qs is an alias for urlparse.parse_qs, which delegates heavy lifting to urlparse.parse_qsl.

parse_qsl un-percent-encodes and also converts + to space for both keys and values.

[chadwhitacre]

parse_qsl is agnostic wrt str vs. unicode.

[chadwhitacre]

We have nine network engines. Do any of them unquote_plus environ['QUERY_STRING']?

[chadwhitacre]

WSGI is based on CGI, and CGI specifies that QUERY_STRING is percent-encoded.

[chadwhitacre]

gevent and eventlet both give us percent-encoded bytes.

[chadwhitacre]

But the twist on that end is that supposedly MSIE 6-10 gives us UTF-8 bytes for the querystring. I haven't actually seen that myself, and I don't know what (if anything) the various network engines do with those bytes before we get them in environ['QUERY_STRING']. Maybe some of them percent-encode them for us?

Understansing MSIE querystring behavior and Aspen's interaction with it reticketed as #197.

[chadwhitacre]

Okay, let's take it as given that make_franken_uri returns a properly percent-encoded path+qs.

[chadwhitacre]

@TomiTiihonen I pushed a fix to master. Can you confirm for us whether it works?

[TomiTiihonen]

Can confirm the fix works.

I tried the master but as mentioned earlier there has been a lot of changes and it broke my config. So ended up just patching the request.py. Thanks for quick response!