webob chokes on %c5 in url when url_encoding is the default UTF-8

[tisdall]

  File "/sites/env/lib/python3.3/site-packages/WebOb-1.2.3-py3.3.egg/webob/request.py", line 167, in encget
    return bytes_(val, 'latin-1').decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc5 in position 18: invalid continuation byte

I was trying to pass short binary data via URL encoding and got this error.

So, I can do fun things like this: http://www.pylonsproject.org/abou%c5t/pylons

Server Error

An error occurred. See the error logs for more information. (Turn debug on to display exception reports here)

Essentially throw a server error on sites running Pyramid.

[mcdonc]

Pyramid requires that all URL segments be valid UTF-8., apologies if this is limiting Webob can be configured with a different default URL encoding, but Pyramid doesn't really anticipate it. FWIW, the 'latin-1' in the traceback you mention is a red herring. This is the insane "Python 3 WSGI dance" that has to happen to convert WSGI-encoded Unicode.

This is actually more a Pyramid issue than a Webob one, I believe.

[mcdonc]

FWIW, Pyramid does define a URLDecodeError which, if it's raised properly by the framework in all the right places, that you catch with an exception view for cases like this.

[mcdonc]

Note that this issue sort of turns into "webob should itself define URLDecodeError instead of making its consumers define it."

Task: wherever webob might raise a UnicodeDecodeError when attempting to decode a URL, it should raise a subclass of UnicodeDecodeError instead, e.g.:

class URLDecodeError(UnicodeDecodeError):
    pass

try:
    request.path
except UnicodeDecodeError as e:
    raise URLDecodeError(e.encoding, e.object, e.start, e.end, e.reason)

The tricky part is identifiying all of the places where WebOb might raise a UnicodeDecodeError related to a URL.

[mcdonc]

See https://github.com/pypa/warehouse/blob/master/warehouse/config.py#L65-L87 (junk_encoding_tween_factory) for the current state of where webob might raise a UnicodeDecodeError as mentioned in #114 (comment)