Only last octet is left in non-UTF-8 %-encoded sequence

[serhiy-storchaka]

>>> URL.build(path='/%f0%9f%90', encoded=True).path
'/%90'

[besfahbod]

Since URL is expected to be able to generate a Unicode version of all URL data fields, no exceptions, I believe the expectation is to deal with non-UTF8 encoding before URL instantiation.

With that, I think there are two possible expected outcomes here:

1. URL instantiation should raise an exception, for decoding error.
2. URL instantiation ignoring the erroneous runs, leaving us with only / as the path here.

We probably can make (2) an option to the parser, and have (1) be the default behavior.

What do you think?

[serhiy-storchaka]

I think that at first step for compatibility with the current code incorrect %-encoded sequence should be returned intact: '/%f0%9f%90' (preserving the case of letters). And correct %-sequences should be decoded. I'll try to fix this after fixing other issues in the unquoter.

But it would be nice to have a way to control the behavior:

• Raise an error (at instantiation time or at decoding time).
• Ignore incorrect escaping completely.
• Replace it with one or more U+FFFD characters.
• Use "surrogate escapes" as for non-decodable paths.
• Return it intact.

This is different and more complex issue.